KEY POINTS

Question: What is the quality of available open access educational resources for anesthesia providers and trainees? Findings: We identified and evaluated the quality of 37 education websites that provide open access educational resources relevant to anesthesia providers and trainees worldwide. Meaning: The 37 open access websites may serve as a directory for anesthesia clinicians and educators globally seeking open access online learning or teaching resources, but the small number of available resources highlights the need for the development of additional open access resources for anesthesia providers worldwide.

The development of the Internet radically democratized the availability of information, affording access to anyone, including those who are underserved or living with limited resources. This is a result of many factors, including increased access to high-speed Internet, flexibility of content delivery, improved usability and decreased cost of content authoring and website development tools, enhanced ability to upload and share resources, and opportunities for inclusion of remote and/or global learners. Websites, such as MIT OpenCourseWare (https://ocw.mit.edu) and Khan Academy (https://www.khanacademy.org), were early influencers in the educational domain, providing open access to educational resources for university and primary and secondary school education, respectively.

In the health care field, online learning has rapidly expanded across the continuum of undergraduate to graduate to health professional continuing education including anesthesia.1 While open access to journal articles2 and free open access medical education (FOAM) resources are becoming increasingly prevalent,3 some online educational resources for health professional education exist behind paywalls and still require payment for access, such as UpToDate (https://www.uptodate.com/) and Anesthesia Toolbox (https://www.anesthesiatoolbox.com). Other online educational resources exist behind firewalls and do not allow external/open access as only students, faculty, and health care providers in those organizations have access to those websites. This lack of open access content may promote global inequities in the availability and sharing of information and may have the most significant impact on health care professionals working in lower resourced settings where the need is the greatest. In the Education of Health Professionals for the 21st Century Commission led by Frenk et al,4 several themes emerged as proposed strategies to assist with improving health care professional training globally. These include harnessing the power of information technology solutions to increase access to resources and to promote global sharing of online resources. The potential effect of e-learning was proposed to be revolutionary.

The extent of open access online educational websites that are freely accessible to clinicians and trainees in anesthesiology worldwide is unknown. In this study, we aimed to identify and describe available open access online educational websites for anesthesia providers and to evaluate the quality of these websites using a modification of a previously validated medical education website quality evaluation tool.

METHODS

We did not obtain institutional review board review for this research given it was limited to publicly available online resources and does not involve any subjects or contain any patient or protected information.

Modification of a Medical Education Website Assessment Tool

The Medical Education Website Quality Evaluation Tool (MEWQET) is a tool developed for assessment of undergraduate medical education websites in the field of pathology5 and has been previously modified for assessment of undergraduate and graduate medical education websites in the fields of otolaryngology (Modified Education in Otolaryngology Website [MEOW]) and critical care medicine (Critical Care Medical Education Website Quality Evaluation Tool [CCMEWQET]).6,7 To evaluate the quality of medical education websites in anesthesia, we adapted the CCMEWQET to be specific to anesthesia rather than critical care medicine by changing the term “critical care” to “anesthesia” or “anesthesiology.” The Anesthesia Medical Education Website Quality Evaluation Tool (AMEWQET, Supplemental Digital Content 1, Table 1, https://links.lww.com/AA/E34) assigns a score to websites based on 32 items assessing 10 domains: authorship, credibility, aim/scope, comprehensiveness, content quality/accuracy, currency of information, site navigability, access, interactivity, and graphic/media elements, with a possible total score of 73.

Literature Review and Web Search

There are many types of online educational resources available, including websites, mobile applications, social media posts on platforms such as Twitter, videos on platforms such as YouTube, or podcasts on platforms such as Soundcloud. Accurate evaluation of the quality of each of these types of educational resources requires its own individual evaluation tool.8–12 This study focuses on the evaluation of open access websites readily accessible through an Internet-based search. Thus, nonwebsite educational resources were excluded from our review.

We conducted a literature review and web search in May 2021 in collaboration with the Boston Children’s Hospital Library Services to identify websites relevant for anesthesia education. To identify relevant articles referencing education-focused anesthesia websites in the literature, we performed a PubMed search of articles published between 2009 and 2020 using the following search terms: (“anesthesia” OR “anesthesiology”) AND (“online resources,” “online education,” “online learning,” “educational website(s),” “educational video(s),” “video-based education,” “video-based learning,” “e-learning,” “Internet-based resources,” “Internet-based education,” “Internet-based learning,” “web-based resources,” “web-based education,” “web-based learning,” “web-based educational resources,” “podcast(s),” “blog(s),” “serious gaming,” “free open access medical education,” or “FOAMed”). We also conducted a web search using the search engine Startpage (www.startpage.com), which generates search results that are not influenced by geographic location or user browser history to reduce bias. We performed a search using the following terms: (“education,” “learn,” “resource,” “FOAM”) AND (“anesthesia,” “anaesthesia,” “anesthesiology,” OR “anaesthesiology”).

One author (A.A.K.) reviewed all articles that met inclusion criteria to identify potential website names and URLs, which were subsequently screened as well. Our inclusion criteria included websites that (1) potentially contained online educational content relevant for anesthesia providers or trainees, (2) offered content free of charge, and (3) were written in the English language. Websites not offering content free of charge or specific to anesthesia were excluded. Online journals were felt to be beyond the scope of this article; however, websites of online journals were included if they contained other types of educational materials (for example, podcasts) on their website. One author (A.A.K.) screened titles and abstracts of all resulting articles and all websites to determine whether they met our inclusion criteria. We identified additional websites that met inclusion criteria from references in the articles and links from the websites.

Evaluation of Identified Websites/Tool Application

All websites that met inclusion criteria were reviewed on a computer. They were not assessed for functionality in a tablet or mobile environment. To achieve consensus on the review process, all reviewers initially independently reviewed 5 websites, and scores were compared across reviewers.

Potential Conflict

F. M. Evans, A. A. Krotinger, G. A. Pereira, and T. A. Wolbrink are associated with OPENPediatrics, F. M. Evans and M. Lilaonitkul are associated with both the World Federation of Societies of Anaesthesiologists (WFSA) and ATOTW, and F. M. Evans is associated with SPA. No reviewer reviewed a website with which she is associated. Scoring discrepancies were discussed, and consensus was achieved. One medical education expert (T.A.W., A.A.K., or G.A.P.) and 1 anesthesiologist (F.M.E., H.F.K., or M.L.) reviewed each of the remaining websites and scored them using the AMEWQET. All scores were reviewed for agreement, and any disagreement was discussed with both reviewers until consensus was reached. Any remaining discrepancies were discussed among the entire team of reviewers (T.A.W., A.A.K., G.A.P., F.M.E., H.F.K., and M.L.) for further input. To evaluate interrater agreement, a kappa analysis was performed on the individual scores of the 2 reviewers for each item in each website. Absolute agreement percentage was calculated as well as kappa as a chance-corrected measure of agreement. Overall, there was excellent agreement and consistency between reviewers for all items across 37 websites (kappa = 0.81, absolute agreement = 85%). There was good to excellent agreement for each individual item or criteria (Supplemental Digital Content 2, Table 2, https://links.lww.com/AA/E35).

Statistical Analysis

Websites were divided into tertiles based on their total AMEWQET score. Frequencies and percentages were used to report overall scores and scores across tertiles for each criterion. The Fisher exact test was used to compare percentages for each AMEWQET criterion across the tertiles. Two-sided P values <.05 were considered statistically significant. Microsoft Excel 2016 (Microsoft Corporation) and Stata version 16.1 (StataCorp LLC) were used for data analysis.

RESULTS

Seventy-five articles and 108 websites were identified in the initial PubMed and Startpage search. Sixty-one articles and 41 websites did not meet inclusion criteria and were excluded (Figure 1). An additional 44 websites were excluded because they did not contain any open access content. A total of 37 websites were reviewed using the AMEWQET.

Figure 1.:

PRISMA flow diagram showing the PubMed and Startpage search strategy used to identify websites for scoring with AMEWQET. AMEWQET indicates Anesthesia Medical Education Website Quality Evaluation Tool; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Tables 1 and 2 list the 37 scored websites by tertile and include the website name, URL, affiliated organization(s) (if applicable), and description of learning activities/content. The most common types of educational content included videos (66%, 25/37), text-based resources (51%, 19/37), podcasts (35%, 13/37), and interactive learning resources (32%, 12/37). Most websites (73%, 27/37) included more than 1 type of content. Only 3 websites explicitly listed their aim to support education and training of anesthesia clinicians in low- and middle-income settings.

Table 1. - Thirty-Seven Scored Websites by Tertile, Including Website Name, URL, Affiliated Organization(s) (if Applicable), and Description of Learning Activities/Content (Upper Tertile) Website name (affiliated organization[s], if applicable) URL Learning activities/content Upper tertile The Anaesthesia Blog (Association of Anaesthetists) https://theanaesthesia.blog/ Blog for Anaesthesia, the official journal of the Association of Anaesthetists. Includes blog posts, open access journal articles, and podcasts on a variety of anesthesia-related topics. Anesthesia and Critical Care Reviews and Commentary http://accrac.com/ Bimonthly podcast series developed for anesthesia trainees reviewing for examinations and as a forum for interesting topics, debates, and interviews in anesthesia and critical care. Anesthesiology News https://www.anesthesiologynews.com/ Open access white papers and monographs on a wide variety of topics relevant to practicing anesthesiologists. Also includes videos, podcasts, open access journal articles, and CME opportunities. Anaesthesia UK https://www.frca.co.uk/ Interactive practice questions, journal abstracts, and reference articles aimed at trainees preparing for postgraduate anesthesia examination. eSafe (Royal College of Anaesthetists, Association of Anaesthetists of Great Britain Foundation, WFSA [see below], e-Learning for Health) https://www.e-safe-anaesthesia.org/ Interactive educational tool that includes e-learning self-paced courses, video tutorials, and links to open access articles on a variety of anesthesia topics. Developed to support the education, training, and CME of anesthesia providers working in resource-poor settings. New England Journal of Medicine https://www.nejm.org/multimedia/ Interactive case examples, daily questions, audio interviews, and illustration and video galleries. OpenAnesthesia (International Anesthesia Research Society) https://www.openanesthesia.org/ Hosts a variety of subspecialty areas and includes an assortment of educational resources such as “Question of the Day,” ABA key words, problem-based discussions, podcasts, video-based resources, and virtual grand rounds conducted in collaboration with various professional societies. OPENPediatrics (Boston Children’s Hospital) https://www.openpediatrics.org/ Digital learning platform for pediatric clinicians. Includes peer-reviewed videos in critical care, pediatrics, and anesthesia; podcasts; interactive mechanical ventilation and dialysis simulators; and courses with pre- and postcourse multiple-choice questions assessments. Society for Neuroscience in Anesthesiology and Critical Care https://www.snacc.org/ Focus on neuroanesthesia; resources include “articles of the month,” topic reviews, interactive clinical cases, quizzes, and podcasts. Society for Pediatric Anesthesia https://pedsanesthesia.org Hosts a Question of the Week that addresses keywords from the American Board of Anesthesia’s pediatric anesthesia in-training examination, 1-pagers that are concise summaries on a wide range of medical education, faculty development and wellness topics, SPA PowerPoint lecture series, and featured videos from previous meetings. Toronto General Hospital http://pie.med.utoronto.ca/ Focus on cardiac anesthesia; features a series of interactive teaching aids including notes, simulators, and videos to assist faculty in training residents and fellows. Ultrasound for Regional Anaesthesia http://www.usra.ca/ Ultrasound images and video galleries, background information, virtual anatomy resources, and procedural techniques for regional anesthesia and pain medicine. WFSA https://www.wfsahq.org/ Resource hub of open source journal articles, books, and videos in anesthesia and WFSA’s flagship online publications: ATOTW and UiA. Aims to support education and training especially in low- and middle-income settings.

Abbreviations: ABA, American Board of Anesthesiology; ATOTW, Anaesthesia Tutorial of the Week; CME, continuing medical education; SPA, Society for Pediatric Anesthesia; UiA, Update in Anaesthesia; WFSA, World Federation of Societies of Anaesthesiologists.

Table 2. - Thirty-Seven Scored Websites by Tertile, Including Website Name, URL, Affiliated Organization(s) (if Applicable), and Description of Learning Activities/Content (Middle and Lower Tertiles) Middle tertile URL Learning activities/content American Society of Anesthesiologists https://www.asahq.org/education-and-career Podcast series and a variety of CME courses available through paid subscription with select open source material. ASRA Pain Medicine https://www.asra.com Articles on regional and pain medicine, ASRA practice guidelines, “How I Do It” features that walk readers through different situations in clinical practice, an image gallery, videos, and access to ASRA apps. Anesthesia Hub https://www.anesthesiahub.com/ Curated site that includes links to open access “tools,” reference books, journals, blogs, videos on a variety of topics, professional societies, and more. Anesthesia Illustrated (Stanford University and Yale University) http://anesthesiaill.wpengine.com/ Obstetric anesthesia presentation recordings from the annual SOAP and Sol Shnider conferences, and videos and lectures on clinical anesthesia foundational practice. Anesthesiology Journal’s Podcast https://anesthesiology.libsyn.com/ Podcasts on featured articles, overview of monthly issue content published in the journal Anesthesiology, and audio interviews of authors and editorialists. BJA Education (joint venture between BJA and The Royal College of Anaesthetists in the United Kingdom) https://bjaed.org/ Open access peer-reviewed articles covering a variety of topics within anesthesia, question bank on various topics, CME opportunities, and a monthly podcast where authors discuss their work. ICE-TAP https://icetap.org/ Teaching modules on EEG monitoring and applications in the operating room, videos, case examples, lectures, and slideshows. IARS https://journals.lww.com/anesthesia-analgesia/ Website for Anesthesia and Analgesia, the official journal of IARS. Includes free articles, videos, and podcasts in a wide range of anesthesia topics. New York School of Regional Anesthesia https://www.nysora.com/ Comprehensive open access multimedia resource library on functional anatomy and ultrasound-guided regional anesthesia techniques. Paid subscription to mobile apps, LMS and CME courses in regional anesthesia. NIH Pain Consortium https://coepes.nih.gov/ Series of interactive pain modules that walk users through events in a simulated patient’s course of care and include videos, tips, and other resources. OpenAirway https://openairway.org/ Instructional videos, clinical guidelines, emergency algorithms, courses on airway management, and links to other anesthesia-focused FOAM web resources. Ramachandra Anesthesia Continuing Education (Institute of Higher Education and Research) https://www.race-elearn.com Anesthesia educational website for both anesthesia trainees and practicing anesthesiologists. Includes a variety of educational resources including video lectures, case discussions, PowerPoints, and e-Books on a variety of anesthesia topics. Lower tertile The Anesthesia Consultant https://theanesthesiaconsultant.com/ Anesthesia educational website for laypeople and medical specialists. Includes columns and opinion-editorials on topics in anesthesia from the viewpoint of an active physician anesthesiologist. Anesthesia Education Website http://www.anaesthesiamcq.com/ Open access examinations and interactive question banks, e-books, and links to external resources. Behind the Drape https://behindthedrape.wordpress.com/ Blog on topics in anesthesia. Aim of the site is to provide useful information for future student nurse anesthetist and/or current practicing CRNAs to promote best clinical practice. Case Reports in Anesthesia http://russellmd.blogspot.com/ Large library of case reports in anesthesia that include medical illustrations and graphics. CHOP- OPEN Medical Institute, American Austrian Foundation https://www.chop.edu/centers-programs/chop-open-access-medical-education/anesthesiology-and-critical-care-medicine-course Interactive courses, lectures, and medical illustrations in pediatric anesthesia and critical care. Cook County Regional https://cookcountyregional.com/index.html Book chapters and video collection on regional anesthesia. NeuroAxiom http://www.neuraxiom.com/ Illustrated techniques for nerve blocks, anatomy illustrations, background text, an open forum, and videos. Propofology.com https://www.propofology.com/resources.html “Infograms” and visual abstracts on anesthesia, critical care, and pain medicine topics. Also offers YouTube (free) and Vimeo channels (paid subscription) covering these topics. Radius Anesthesia https://radiusanesthesia.com/ Monthly blog on current anesthesia topics for practicing anesthesiologists. Rapid Sequence http://www.rapidsequence.org.uk/ Collection of open access blog posts, curated reviews of specific topics in anesthesia (airway, pain, renal, etc.), and links to external resources. Renaissance School of Medicine—Department of Anesthesiology https://renaissance.stonybrookmedicine.edu/anesthesiology/online-education Series of video collections on cardiac anesthesia, advanced cardiac life support, and procedural skills. Yale University https://medicine.yale.edu/anesthesiology/media/ Educational videos on room setup, procedures, and everyday tasks in anesthesia residency.

Abbreviations: ASRA, American Society of Regional Anesthesia; BJA, British Journal of Anaesthesia; CHOP, Children’s Hospital of Pennsylvania; CME, continuing medical education; CRNA, certified registered nurse anesthetist; EEG, electroencephalogram; FOAM, Free-Open Access Medical Education; IARS, International Anesthesia Research Society; ICE-TAP, International Consortium for EEG Training of Anesthesia Practitioners; LMS, learning management system; NIH, National Institutes of Health; OPEN, Online Pediatric Education Network; SOAP, Society for Obstetric Anesthesia and Perinatology.

Table 3. - AMEWQET Tool Criteria Compared Between Website Tertiles Category Criteria Overall, N = 37 (%) Top tertile, n = 13 (%) Middle tertile, n = 12 (%) Lowest tertile, n = 12 (%) P value 1. Authorship, credibility‚ and disclosure 1.1. Disclosure of author/webmaster? A. Author/webmaster(s)’ name(s), credentials, and contact information 25 (68) 11 (85) 8 (67) 6 (50) .58 B. Author/webmaster(s)’ name(s) and credentials 5 (14) 1 (8) 2 (17) 2 (17) C. Author/webmaster(s)’ name(s) 6 (16) 1 (8) 2 (17) 3 (25) D. No disclosure of authorship 1 (3) 0 (0) 0 (0) 1 (8) 1.2. If author/webmaster(s)’ credentials are given, author is (if multiple authors, the majority are): A. Anesthesiologist 26/36 (72) 11/13 (85) 9/12 (75) 6/11 (55) .49 B. Other health care professional/scientist 7/36 (19) 2/13 (15) 2/12 (17) 3/11 (27) C. Other 3/36 (8) 0/13 (0) 1/12 (8) 2/11 (18) 1.3. Disclosure of author/webmaster(s)‘ institution? A. Educational, nonprofit‚ or governmental domain 32 (86) 13 (100) 12 (100) 7 (58) .004 B. Other or no disclosure of institution 5 (14) 0 (0) 0 (0) 5 (42) 1.4. Disclosure of website’s institution? A. Educational, nonprofit or governmental domain 29 (78) 13 (100) 10 (83) 6 (50) .005 B. Other or no disclosure of institution 8 (22) 0 (0) 2 (17) 6 (50) 1.5. Is there an editorial review process stated on the website? Yes 9 (24) 6 (46) 3 (25) 0 (0) .02 No 28 (76) 7 (54) 9 (75) 12 (100) 1.6. Is there a disclosure of copyright, intellectual property issues, or a general disclosure? Yes 27 (73) 11 (85) 10 (83) 6 (50) .13 No 10 (27) 2 (15) 2 (17) 6 (50) 1.7. Are references provided? Yes 32 (86) 12 (92) 10 (83) 10 (83) .72 No 5 (14) 1 (8) 2 (17) 2 (17) 1.8. Is there a mechanism for learners to provide feedback to the author/webmaster(s)? Yes 33 (89) 13 (100) 10 (83) 10 (83) .36 No 4 (11) 0 (0) 2 (17) 2 (17) 1.9. Is advertising distinct from content? A. No advertisements 23 (62) 8 (62) 7 (58) 8 (67) .93 B. Yes 12 (32) 5 (38) 4 (33) 3 (25) C. No 2 (5) 0 (0) 1 (8) 1 (8) 2. Aim, scope, and audience 2.1. Is anesthesia the intended subject scope? A. Yes, exclusively anesthesia 31 (84) 11 (85) 10 (83) 10 (83) .99 B. Anesthesia, as part of another subject area (ie, critical care medicine) 4 (11) 1 (8) 2 (17) 1 (8) C. No 2 (5) 1 (8) 0 (0) 1 (8) 2.2. Is the educational material relevant to residents, fellows, and/or faculty in anesthesia? A. Highly relevant 24 (65) 11 (85) 10 (83) 3 (25) .004 B. Relevant 9 (24) 1 (8) 1 (8) 7 (58) C. Minimally relevant 4 (11) 1 (8) 1 (8) 2 (17) D. Not relevant or not stated 0 (0) 0 (0) 0 (0) 0 (0) 3. Content quality 3.1. Comprehensiveness: does it cover anesthesia education comprehensively? A. Yes 3 (8) 3 (23) 0 (0) 0 (0) <.001 B. No, but fairly comprehensive coverage of more than 1 specific area of interest 8 (22) 6 (46) 2 (17) 0 (0) C. No, but fairly comprehensive coverage of a specific area of interest 11 (30) 3 (23) 6 (50) 2 (17) D. No, even specific areas of interest are not covered comprehensively 15 (41) 1 (8) 4 (33) 10 (83) 3.2. Accuracy: is the information accurate? A. Accurate 33 (89) 13 (100) 12 (100) 8 (67) .02 B. Somewhat accurate 4 (11) 0 (0) 0 (0) 4 (33) C. Inaccurate 0 (0) 0 (0) 0 (0) 0 (0) 3.3. Does the website have summary statements/take-home points? Yes 20 (54) 9 (69) 7 (58) 4 (33) .21 No 17 (46) 4 (31) 5 (42) 8 (67) 4. Currency of information 4.1. Is the date of content creation/revision disclosed? Yes 32 (86) 13 (100) 11 (92) 8 (67) .03 No 5 (14) 0 (0) 1 (8) 4 (33) 4.2. When was the website (including references) last updated? A. <1 y ago 28 (77) 11 (85) 10 (83) 7 (58) .19 B. ≥1 y ago but <5 y ago 4 (11) 2 (15) 1 (8) 1 (8) C. ≥5 y ago or not disclosed 5 (14) 0 (0) 1 (8) 4 (33) 5. Navigability and speed 5.1. Can necessary information be found easily from the main homepage of the site? Yes 31 (84) 13 (100) 10 (83) 8 (67) .06 No 6 (16) 0 (0) 2 (17) 4 (33) 5.2. Does the site include a usable search engine or table of contents? Yes 34 (92) 13 (100) 10 (83) 11 (92) .3 No