Evaluation of Open Access Websites for Anesthesia Education : Anesthesia & Analgesia

Secondary Logo

Journal Logo

Original Research Articles: Original Clinical Research Report

Evaluation of Open Access Websites for Anesthesia Education

Evans, Faye M. MD*,†; Krotinger, Anna A. MBE; Lilaonitkul, Maytinee MBBS§; Khaled, Hiba F. MB BCh, BAO; Pereira, Gretchen A. MHA*,‡; Staffa, Steven J. MS*; Wolbrink, Traci A. MD, MPH*,†

Author Information
doi: 10.1213/ANE.0000000000006183

Abstract

KEY POINTS

  • Question: What is the quality of available open access educational resources for anesthesia providers and trainees?
  • Findings: We identified and evaluated the quality of 37 education websites that provide open access educational resources relevant to anesthesia providers and trainees worldwide.
  • Meaning: The 37 open access websites may serve as a directory for anesthesia clinicians and educators globally seeking open access online learning or teaching resources, but the small number of available resources highlights the need for the development of additional open access resources for anesthesia providers worldwide.

The development of the Internet radically democratized the availability of information, affording access to anyone, including those who are underserved or living with limited resources. This is a result of many factors, including increased access to high-speed Internet, flexibility of content delivery, improved usability and decreased cost of content authoring and website development tools, enhanced ability to upload and share resources, and opportunities for inclusion of remote and/or global learners. Websites, such as MIT OpenCourseWare (https://ocw.mit.edu) and Khan Academy (https://www.khanacademy.org), were early influencers in the educational domain, providing open access to educational resources for university and primary and secondary school education, respectively.

In the health care field, online learning has rapidly expanded across the continuum of undergraduate to graduate to health professional continuing education including anesthesia.1 While open access to journal articles2 and free open access medical education (FOAM) resources are becoming increasingly prevalent,3 some online educational resources for health professional education exist behind paywalls and still require payment for access, such as UpToDate (https://www.uptodate.com/) and Anesthesia Toolbox (https://www.anesthesiatoolbox.com). Other online educational resources exist behind firewalls and do not allow external/open access as only students, faculty, and health care providers in those organizations have access to those websites. This lack of open access content may promote global inequities in the availability and sharing of information and may have the most significant impact on health care professionals working in lower resourced settings where the need is the greatest. In the Education of Health Professionals for the 21st Century Commission led by Frenk et al,4 several themes emerged as proposed strategies to assist with improving health care professional training globally. These include harnessing the power of information technology solutions to increase access to resources and to promote global sharing of online resources. The potential effect of e-learning was proposed to be revolutionary.

The extent of open access online educational websites that are freely accessible to clinicians and trainees in anesthesiology worldwide is unknown. In this study, we aimed to identify and describe available open access online educational websites for anesthesia providers and to evaluate the quality of these websites using a modification of a previously validated medical education website quality evaluation tool.

METHODS

We did not obtain institutional review board review for this research given it was limited to publicly available online resources and does not involve any subjects or contain any patient or protected information.

Modification of a Medical Education Website Assessment Tool

The Medical Education Website Quality Evaluation Tool (MEWQET) is a tool developed for assessment of undergraduate medical education websites in the field of pathology5 and has been previously modified for assessment of undergraduate and graduate medical education websites in the fields of otolaryngology (Modified Education in Otolaryngology Website [MEOW]) and critical care medicine (Critical Care Medical Education Website Quality Evaluation Tool [CCMEWQET]).6,7 To evaluate the quality of medical education websites in anesthesia, we adapted the CCMEWQET to be specific to anesthesia rather than critical care medicine by changing the term “critical care” to “anesthesia” or “anesthesiology.” The Anesthesia Medical Education Website Quality Evaluation Tool (AMEWQET, Supplemental Digital Content 1, Table 1, https://links.lww.com/AA/E34) assigns a score to websites based on 32 items assessing 10 domains: authorship, credibility, aim/scope, comprehensiveness, content quality/accuracy, currency of information, site navigability, access, interactivity, and graphic/media elements, with a possible total score of 73.

Literature Review and Web Search

There are many types of online educational resources available, including websites, mobile applications, social media posts on platforms such as Twitter, videos on platforms such as YouTube, or podcasts on platforms such as Soundcloud. Accurate evaluation of the quality of each of these types of educational resources requires its own individual evaluation tool.8–12 This study focuses on the evaluation of open access websites readily accessible through an Internet-based search. Thus, nonwebsite educational resources were excluded from our review.

We conducted a literature review and web search in May 2021 in collaboration with the Boston Children’s Hospital Library Services to identify websites relevant for anesthesia education. To identify relevant articles referencing education-focused anesthesia websites in the literature, we performed a PubMed search of articles published between 2009 and 2020 using the following search terms: (“anesthesia” OR “anesthesiology”) AND (“online resources,” “online education,” “online learning,” “educational website(s),” “educational video(s),” “video-based education,” “video-based learning,” “e-learning,” “Internet-based resources,” “Internet-based education,” “Internet-based learning,” “web-based resources,” “web-based education,” “web-based learning,” “web-based educational resources,” “podcast(s),” “blog(s),” “serious gaming,” “free open access medical education,” or “FOAMed”). We also conducted a web search using the search engine Startpage (www.startpage.com), which generates search results that are not influenced by geographic location or user browser history to reduce bias. We performed a search using the following terms: (“education,” “learn,” “resource,” “FOAM”) AND (“anesthesia,” “anaesthesia,” “anesthesiology,” OR “anaesthesiology”).

One author (A.A.K.) reviewed all articles that met inclusion criteria to identify potential website names and URLs, which were subsequently screened as well. Our inclusion criteria included websites that (1) potentially contained online educational content relevant for anesthesia providers or trainees, (2) offered content free of charge, and (3) were written in the English language. Websites not offering content free of charge or specific to anesthesia were excluded. Online journals were felt to be beyond the scope of this article; however, websites of online journals were included if they contained other types of educational materials (for example, podcasts) on their website. One author (A.A.K.) screened titles and abstracts of all resulting articles and all websites to determine whether they met our inclusion criteria. We identified additional websites that met inclusion criteria from references in the articles and links from the websites.

Evaluation of Identified Websites/Tool Application

All websites that met inclusion criteria were reviewed on a computer. They were not assessed for functionality in a tablet or mobile environment. To achieve consensus on the review process, all reviewers initially independently reviewed 5 websites, and scores were compared across reviewers.

Potential Conflict

F. M. Evans, A. A. Krotinger, G. A. Pereira, and T. A. Wolbrink are associated with OPENPediatrics, F. M. Evans and M. Lilaonitkul are associated with both the World Federation of Societies of Anaesthesiologists (WFSA) and ATOTW, and F. M. Evans is associated with SPA. No reviewer reviewed a website with which she is associated. Scoring discrepancies were discussed, and consensus was achieved. One medical education expert (T.A.W., A.A.K., or G.A.P.) and 1 anesthesiologist (F.M.E., H.F.K., or M.L.) reviewed each of the remaining websites and scored them using the AMEWQET. All scores were reviewed for agreement, and any disagreement was discussed with both reviewers until consensus was reached. Any remaining discrepancies were discussed among the entire team of reviewers (T.A.W., A.A.K., G.A.P., F.M.E., H.F.K., and M.L.) for further input. To evaluate interrater agreement, a kappa analysis was performed on the individual scores of the 2 reviewers for each item in each website. Absolute agreement percentage was calculated as well as kappa as a chance-corrected measure of agreement. Overall, there was excellent agreement and consistency between reviewers for all items across 37 websites (kappa = 0.81, absolute agreement = 85%). There was good to excellent agreement for each individual item or criteria (Supplemental Digital Content 2, Table 2, https://links.lww.com/AA/E35).

Statistical Analysis

Websites were divided into tertiles based on their total AMEWQET score. Frequencies and percentages were used to report overall scores and scores across tertiles for each criterion. The Fisher exact test was used to compare percentages for each AMEWQET criterion across the tertiles. Two-sided P values <.05 were considered statistically significant. Microsoft Excel 2016 (Microsoft Corporation) and Stata version 16.1 (StataCorp LLC) were used for data analysis.

RESULTS

Seventy-five articles and 108 websites were identified in the initial PubMed and Startpage search. Sixty-one articles and 41 websites did not meet inclusion criteria and were excluded (Figure 1). An additional 44 websites were excluded because they did not contain any open access content. A total of 37 websites were reviewed using the AMEWQET.

F1
Figure 1.:
PRISMA flow diagram showing the PubMed and Startpage search strategy used to identify websites for scoring with AMEWQET. AMEWQET indicates Anesthesia Medical Education Website Quality Evaluation Tool; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Tables 1 and 2 list the 37 scored websites by tertile and include the website name, URL, affiliated organization(s) (if applicable), and description of learning activities/content. The most common types of educational content included videos (66%, 25/37), text-based resources (51%, 19/37), podcasts (35%, 13/37), and interactive learning resources (32%, 12/37). Most websites (73%, 27/37) included more than 1 type of content. Only 3 websites explicitly listed their aim to support education and training of anesthesia clinicians in low- and middle-income settings.

Table 1. - Thirty-Seven Scored Websites by Tertile, Including Website Name, URL, Affiliated Organization(s) (if Applicable), and Description of Learning Activities/Content (Upper Tertile)
Website name (affiliated organization[s], if applicable) URL Learning activities/content
Upper tertile
The Anaesthesia Blog (Association of Anaesthetists) https://theanaesthesia.blog/ Blog for Anaesthesia, the official journal of the Association of Anaesthetists. Includes blog posts, open access journal articles, and podcasts on a variety of anesthesia-related topics.
Anesthesia and Critical Care Reviews and Commentary http://accrac.com/ Bimonthly podcast series developed for anesthesia trainees reviewing for examinations and as a forum for interesting topics, debates, and interviews in anesthesia and critical care.
Anesthesiology News https://www.anesthesiologynews.com/ Open access white papers and monographs on a wide variety of topics relevant to practicing anesthesiologists. Also includes videos, podcasts, open access journal articles, and CME opportunities.
Anaesthesia UK https://www.frca.co.uk/ Interactive practice questions, journal abstracts, and reference articles aimed at trainees preparing for postgraduate anesthesia examination.
eSafe (Royal College of Anaesthetists, Association of Anaesthetists of Great Britain Foundation, WFSA [see below], e-Learning for Health) https://www.e-safe-anaesthesia.org/ Interactive educational tool that includes e-learning self-paced courses, video tutorials, and links to open access articles on a variety of anesthesia topics. Developed to support the education, training, and CME of anesthesia providers working in resource-poor settings.
New England Journal of Medicine https://www.nejm.org/multimedia/ Interactive case examples, daily questions, audio interviews, and illustration and video galleries.
OpenAnesthesia (International Anesthesia Research Society) https://www.openanesthesia.org/ Hosts a variety of subspecialty areas and includes an assortment of educational resources such as “Question of the Day,” ABA key words, problem-based discussions, podcasts, video-based resources, and virtual grand rounds conducted in collaboration with various professional societies.
OPENPediatrics (Boston Children’s Hospital) https://www.openpediatrics.org/ Digital learning platform for pediatric clinicians. Includes peer-reviewed videos in critical care, pediatrics, and anesthesia; podcasts; interactive mechanical ventilation and dialysis simulators; and courses with pre- and postcourse multiple-choice questions assessments.
Society for Neuroscience in Anesthesiology and Critical Care https://www.snacc.org/ Focus on neuroanesthesia; resources include “articles of the month,” topic reviews, interactive clinical cases, quizzes, and podcasts.
Society for Pediatric Anesthesia https://pedsanesthesia.org Hosts a Question of the Week that addresses keywords from the American Board of Anesthesia’s pediatric anesthesia in-training examination, 1-pagers that are concise summaries on a wide range of medical education, faculty development and wellness topics, SPA PowerPoint lecture series, and featured videos from previous meetings.
Toronto General Hospital http://pie.med.utoronto.ca/ Focus on cardiac anesthesia; features a series of interactive teaching aids including notes, simulators, and videos to assist faculty in training residents and fellows.
Ultrasound for Regional Anaesthesia http://www.usra.ca/ Ultrasound images and video galleries, background information, virtual anatomy resources, and procedural techniques for regional anesthesia and pain medicine.
WFSA https://www.wfsahq.org/ Resource hub of open source journal articles, books, and videos in anesthesia and WFSA’s flagship online publications: ATOTW and UiA. Aims to support education and training especially in low- and middle-income settings.
Abbreviations: ABA, American Board of Anesthesiology; ATOTW, Anaesthesia Tutorial of the Week; CME, continuing medical education; SPA, Society for Pediatric Anesthesia; UiA, Update in Anaesthesia; WFSA, World Federation of Societies of Anaesthesiologists.

Table 2. - Thirty-Seven Scored Websites by Tertile, Including Website Name, URL, Affiliated Organization(s) (if Applicable), and Description of Learning Activities/Content (Middle and Lower Tertiles)
Middle tertile URL Learning activities/content
American Society of Anesthesiologists https://www.asahq.org/education-and-career Podcast series and a variety of CME courses available through paid subscription with select open source material.
ASRA Pain Medicine https://www.asra.com Articles on regional and pain medicine, ASRA practice guidelines, “How I Do It” features that walk readers through different situations in clinical practice, an image gallery, videos, and access to ASRA apps.
Anesthesia Hub https://www.anesthesiahub.com/ Curated site that includes links to open access “tools,” reference books, journals, blogs, videos on a variety of topics, professional societies, and more.
Anesthesia Illustrated (Stanford University and Yale University) http://anesthesiaill.wpengine.com/ Obstetric anesthesia presentation recordings from the annual SOAP and Sol Shnider conferences, and videos and lectures on clinical anesthesia foundational practice.
Anesthesiology Journal’s Podcast https://anesthesiology.libsyn.com/ Podcasts on featured articles, overview of monthly issue content published in the journal Anesthesiology, and audio interviews of authors and editorialists.
BJA Education (joint venture between BJA and The Royal College of Anaesthetists in the United Kingdom) https://bjaed.org/ Open access peer-reviewed articles covering a variety of topics within anesthesia, question bank on various topics, CME opportunities, and a monthly podcast where authors discuss their work.
ICE-TAP https://icetap.org/ Teaching modules on EEG monitoring and applications in the operating room, videos, case examples, lectures, and slideshows.
IARS https://journals.lww.com/anesthesia-analgesia/ Website for Anesthesia and Analgesia, the official journal of IARS. Includes free articles, videos, and podcasts in a wide range of anesthesia topics.
New York School of Regional Anesthesia https://www.nysora.com/ Comprehensive open access multimedia resource library on functional anatomy and ultrasound-guided regional anesthesia techniques. Paid subscription to mobile apps, LMS and CME courses in regional anesthesia.
NIH Pain Consortium https://coepes.nih.gov/ Series of interactive pain modules that walk users through events in a simulated patient’s course of care and include videos, tips, and other resources.
OpenAirway https://openairway.org/ Instructional videos, clinical guidelines, emergency algorithms, courses on airway management, and links to other anesthesia-focused FOAM web resources.
Ramachandra Anesthesia Continuing Education (Institute of Higher Education and Research) https://www.race-elearn.com Anesthesia educational website for both anesthesia trainees and practicing anesthesiologists. Includes a variety of educational resources including video lectures, case discussions, PowerPoints, and e-Books on a variety of anesthesia topics.
Lower tertile
The Anesthesia Consultant https://theanesthesiaconsultant.com/ Anesthesia educational website for laypeople and medical specialists. Includes columns and opinion-editorials on topics in anesthesia from the viewpoint of an active physician anesthesiologist.
Anesthesia Education Website http://www.anaesthesiamcq.com/ Open access examinations and interactive question banks, e-books, and links to external resources.
Behind the Drape https://behindthedrape.wordpress.com/ Blog on topics in anesthesia. Aim of the site is to provide useful information for future student nurse anesthetist and/or current practicing CRNAs to promote best clinical practice.
Case Reports in Anesthesia http://russellmd.blogspot.com/ Large library of case reports in anesthesia that include medical illustrations and graphics.
CHOP- OPEN Medical Institute, American Austrian Foundation https://www.chop.edu/centers-programs/chop-open-access-medical-education/anesthesiology-and-critical-care-medicine-course Interactive courses, lectures, and medical illustrations in pediatric anesthesia and critical care.
Cook County Regional https://cookcountyregional.com/index.html Book chapters and video collection on regional anesthesia.
NeuroAxiom http://www.neuraxiom.com/ Illustrated techniques for nerve blocks, anatomy illustrations, background text, an open forum, and videos.
Propofology.com https://www.propofology.com/resources.html “Infograms” and visual abstracts on anesthesia, critical care, and pain medicine topics. Also offers YouTube (free) and Vimeo channels (paid subscription) covering these topics.
Radius Anesthesia https://radiusanesthesia.com/ Monthly blog on current anesthesia topics for practicing anesthesiologists.
Rapid Sequence http://www.rapidsequence.org.uk/ Collection of open access blog posts, curated reviews of specific topics in anesthesia (airway, pain, renal, etc.), and links to external resources.
Renaissance School of Medicine—Department of Anesthesiology https://renaissance.stonybrookmedicine.edu/anesthesiology/online-education Series of video collections on cardiac anesthesia, advanced cardiac life support, and procedural skills.
Yale University https://medicine.yale.edu/anesthesiology/media/ Educational videos on room setup, procedures, and everyday tasks in anesthesia residency.
Abbreviations: ASRA, American Society of Regional Anesthesia; BJA, British Journal of Anaesthesia; CHOP, Children’s Hospital of Pennsylvania; CME, continuing medical education; CRNA, certified registered nurse anesthetist; EEG, electroencephalogram; FOAM, Free-Open Access Medical Education; IARS, International Anesthesia Research Society; ICE-TAP, International Consortium for EEG Training of Anesthesia Practitioners; LMS, learning management system; NIH, National Institutes of Health; OPEN, Online Pediatric Education Network; SOAP, Society for Obstetric Anesthesia and Perinatology.

Table 3. - AMEWQET Tool Criteria Compared Between Website Tertiles
Category Criteria Overall, N = 37 (%) Top tertile, n = 13 (%) Middle tertile, n = 12 (%) Lowest tertile, n = 12 (%) P value
1. Authorship, credibility‚ and disclosure 1.1. Disclosure of author/webmaster? A. Author/webmaster(s)’ name(s), credentials, and contact information 25 (68) 11 (85) 8 (67) 6 (50) .58
B. Author/webmaster(s)’ name(s) and credentials 5 (14) 1 (8) 2 (17) 2 (17)
C. Author/webmaster(s)’ name(s) 6 (16) 1 (8) 2 (17) 3 (25)
D. No disclosure of authorship 1 (3) 0 (0) 0 (0) 1 (8)
1.2. If author/webmaster(s)’ credentials are given, author is (if multiple authors, the majority are): A. Anesthesiologist 26/36 (72) 11/13 (85) 9/12 (75) 6/11 (55) .49
B. Other health care professional/scientist 7/36 (19) 2/13 (15) 2/12 (17) 3/11 (27)
C. Other 3/36 (8) 0/13 (0) 1/12 (8) 2/11 (18)
1.3. Disclosure of author/webmaster(s)‘ institution? A. Educational, nonprofit‚ or governmental domain 32 (86) 13 (100) 12 (100) 7 (58) .004
B. Other or no disclosure of institution 5 (14) 0 (0) 0 (0) 5 (42)
1.4. Disclosure of website’s institution? A. Educational, nonprofit or governmental domain 29 (78) 13 (100) 10 (83) 6 (50) .005
B. Other or no disclosure of institution 8 (22) 0 (0) 2 (17) 6 (50)
1.5. Is there an editorial review process stated on the website? Yes 9 (24) 6 (46) 3 (25) 0 (0) .02
No 28 (76) 7 (54) 9 (75) 12 (100)
1.6. Is there a disclosure of copyright, intellectual property issues, or a general disclosure? Yes 27 (73) 11 (85) 10 (83) 6 (50) .13
No 10 (27) 2 (15) 2 (17) 6 (50)
1.7. Are references provided? Yes 32 (86) 12 (92) 10 (83) 10 (83) .72
No 5 (14) 1 (8) 2 (17) 2 (17)
1.8. Is there a mechanism for learners to provide feedback to the author/webmaster(s)? Yes 33 (89) 13 (100) 10 (83) 10 (83) .36
No 4 (11) 0 (0) 2 (17) 2 (17)
1.9. Is advertising distinct from content? A. No advertisements 23 (62) 8 (62) 7 (58) 8 (67) .93
B. Yes 12 (32) 5 (38) 4 (33) 3 (25)
C. No 2 (5) 0 (0) 1 (8) 1 (8)
2. Aim, scope, and audience 2.1. Is anesthesia the intended subject scope? A. Yes, exclusively anesthesia 31 (84) 11 (85) 10 (83) 10 (83) .99
B. Anesthesia, as part of another subject area (ie, critical care medicine) 4 (11) 1 (8) 2 (17) 1 (8)
C. No 2 (5) 1 (8) 0 (0) 1 (8)
2.2. Is the educational material relevant to residents, fellows, and/or faculty in anesthesia? A. Highly relevant 24 (65) 11 (85) 10 (83) 3 (25) .004
B. Relevant 9 (24) 1 (8) 1 (8) 7 (58)
C. Minimally relevant 4 (11) 1 (8) 1 (8) 2 (17)
D. Not relevant or not stated 0 (0) 0 (0) 0 (0) 0 (0)
3. Content quality 3.1. Comprehensiveness: does it cover anesthesia education comprehensively? A. Yes 3 (8) 3 (23) 0 (0) 0 (0) <.001
B. No, but fairly comprehensive coverage of more than 1 specific area of interest 8 (22) 6 (46) 2 (17) 0 (0)
C. No, but fairly comprehensive coverage of a specific area of interest 11 (30) 3 (23) 6 (50) 2 (17)
D. No, even specific areas of interest are not covered comprehensively 15 (41) 1 (8) 4 (33) 10 (83)
3.2. Accuracy: is the information accurate? A. Accurate 33 (89) 13 (100) 12 (100) 8 (67) .02
B. Somewhat accurate 4 (11) 0 (0) 0 (0) 4 (33)
C. Inaccurate 0 (0) 0 (0) 0 (0) 0 (0)
3.3. Does the website have summary statements/take-home points? Yes 20 (54) 9 (69) 7 (58) 4 (33) .21
No 17 (46) 4 (31) 5 (42) 8 (67)
4. Currency of information 4.1. Is the date of content creation/revision disclosed? Yes 32 (86) 13 (100) 11 (92) 8 (67) .03
No 5 (14) 0 (0) 1 (8) 4 (33)
4.2. When was the website (including references) last updated? A. <1 y ago 28 (77) 11 (85) 10 (83) 7 (58) .19
B. ≥1 y ago but <5 y ago 4 (11) 2 (15) 1 (8) 1 (8)
C. ≥5 y ago or not disclosed 5 (14) 0 (0) 1 (8) 4 (33)
5. Navigability and speed 5.1. Can necessary information be found easily from the main homepage of the site? Yes 31 (84) 13 (100) 10 (83) 8 (67) .06
No 6 (16) 0 (0) 2 (17) 4 (33)
5.2. Does the site include a usable search engine or table of contents? Yes 34 (92) 13 (100) 10 (83) 11 (92) .3
No 3 (8) 0 (0) 2 (17) 1 (8)
5.3. Does every page include a way to return to the homepage for the site? Yes 34 (92) 12 (92) 12 (100) 10 (83) .52
No 3 (8) 1 (8) 0 (0) 2 (17)
5.4. Was the website or server accessible in a timely manner? Yes 36 (97) 13 (100) 11 (92) 12 (100) .65
No 1 (3) 0 (0) 1 (8) 0 (0)
6. Access 6.1. Is the site accessible from the main search engines? Yes 37 (100) 13 (100) 12 (100) 12 (100) .99
No 0 (0) 0 (0) 0 (0) 0 (0)
6.2. How does access to the information through this website compare to other available sources? A. Easier to find/use 8 (22) 5 (38) 3 (25) 0 (0) .03
B. About the same effort to find/use 22 (59) 8 (62) 7 (58) 7 (58)
C. More difficult to find/use 7 (19) 0 (0) 2 (17) 5 (42)
6.3. Is special hardware or software required to access some or all of the resource? A. No 35 (95) 12 (92) 12 (100) 11 (92) .99
B. Some or all of the site requires special hardware or software 2 (5) 1 (8) 0 (0) 1 (8)
7. Interactivity 7.1. Are there any interfaces requiring relevant action on the part of the learner (eg, quizzes, self-assessments, interactive figures)? A. Definitely 6 (16) 5 (38) 0 (0) 1 (8) .009
B. Somewhat 5 (14) 3 (23) 2 (17) 0 (0)
C. No/does not apply 26 (70) 5 (38) 10 (83) 11 (92)
8. Graphics and media 8.1. Are graphic/media elements included to provide additional information to clarify existing content? A. Present and pertinent 24 (65) 10 (77) 9 (75) 5 (42) .09
B. Present 9 (24) 3 (23) 3 (25) 3 (25)
C. Other 4 (11) 0 (0) 0 (0) 4 (33)
8.2. Are graphic/media elements well integrated into the website? Yes 31 (84) 13 (100) 12 (100) 6 (50) .001
No 6 (16) 0 (0) 0 (0) 6 (50)
9. Layout and design 9.1. Is the display of information clear and professional? Yes 31 (84) 13 (100) 12 (100) 6 (50) .001
No 6 (16) 0 (0) 0 (0) 6 (50)
9.2. Is the website user-friendly and intuitive, with a logical layout? Yes 28 (76) 12 (92) 10 (83) 6 (50) .06
No 9 (24) 1 (8) 2 (17) 6 (50)
10. Hyperlinks 10.1. Are there any links to provide relevant additional information? Yes 30 (81) 13 (100) 10 (83) 7 (58) .02
No 7 (19) 0 (0) 2 (17) 5 (42)
10.2. If links are provided, are they active (≥90% of total links)? Yes 27/30 (90) 12/13 (92) 10/10 (100) 5/7 (71) .16
No 3/30 (10) 1/13 (8) 0/10(0) 2/7 (29)
Data are presented as n (%). P values were calculated using the Fisher exact test.
Denominators are shown for statements with branching logic.
Abbreviation: AMEWQET, Anesthesia Medical Education Website Quality Evaluation Tool.

F2
Figure 2.:
Histogram displaying the numeric distribution of scores across tertiles.

The median score across all websites was 55, with a range of 23–72. Figure 2 shows a histogram displaying the numeric distribution of scores across tertiles and Table 3 details the frequency and percentage of website scores for each criterion including overall scores and scores by tertile.

Authorship, Credibility, and Disclosure

Nearly all websites disclosed the author (97%, 36/37) with 26 (72%, 26/36) reporting an anesthesiologist author and 32 (86%, 32/37) disclosing the author/webmaster’s institution as educational, nonprofit, or governmental. Only 9 websites (24%, 9/37) described an editorial process. Fourteen websites (38%,14/37) contained some form of advertising. Many websites included a general disclosure/term of use (73%, 27/37), and most provided references (86%, 32/37) and an opportunity for feedback by users (89%, 33/37).

Aim, Scope, and Audience

Most websites (84%, 31/37) described anesthesiology as the exclusive intended scope, and the content was felt to be relevant or highly relevant for residents, fellows, or faculty in anesthesiology in most websites (89%, 33/37). In the domain of content quality, a few websites (8%, 3/37) were felt to provide comprehensive coverage of anesthesia, half of websites provided fairly comprehensive coverage of 1 or more subject areas (51%, 19/37), while many websites (41%, 15/37) did not provide fairly comprehensive coverage of even 1 subject area. Most websites (89%, 33/37) were felt to have accurate content, but only approximately half of websites (52%, 19/36) included references. In the domain of currency of information, most websites (86%, 32/37) disclosed a date of content creation/revision, with a majority of websites (76%, 28/37) describing content creation/revision within the last year.

Navigability and Speed

Most websites had information easily found from the home page (84%, 31/37), contained a usable search engine/table of contents (92%, 34/37), included a way to return to the homepage on every page (92%, 34/37), and was accessible in a timely manner (97%, 36/37). In the domain of access, all websites were accessible from a main search engine (100%, 37/37), and most websites did not require special software or hardware to use (95%, 35/37). Those that required special software contained content that required Adobe Flash (Adobe), which is no longer a supported software program. Slightly more than half the websites were felt to provide similar ease of access to information as compared to other websites (59%, 22/37), with 22% of websites providing greater ease of access (22%, 8/37) and 19% of websites offering information that was more difficult to use (19%, 7/37).

Interactivity

Only 11 websites (30%, 11/37) included opportunities for interaction. Interactive content was described as content that required some form of interaction by the user, ie, clicking on an answer to a multiple-choice question or working through an interactive case. In the domain of graphics and media, most websites (84%, 31/37) had graphics and media well-integrated into the site. Most websites were also felt to have a clear and professional layout (84%, 31/37) and a user-friendly and intuitive design (76%, 28/37). In the domain of hyperlinks, many websites included hyperlinks to other websites (81%, 30/37), and only 3 websites (10%, 3/30) had more than 10% of hyperlinks that were inactive.

Websites were grouped into tertiles based on AMEWQET scores. The tertiles differed significantly across multiple domains including: disclosure of author/webmaster/website institution (AMEWQET criteria 1.3, 1.4); editorial review process described (AMEWQET criteria 1.5); relevancy to residents, fellows, and faculty (AMEWQET criteria 2.2); comprehensiveness (AMEWQET criteria 3.1); accuracy (AMEWQET criteria 3.2); disclosure of date of content creation/revision (AMEWQET criteria 4.1); comparative ease of access to information (AMEWQET criteria 6.2); presence of interfaces requiring relevant action on the part of the learner (AMEWQET criteria 7.1); integration of graphic/media elements to clarify content (AMEWQET criteria 8.2); clear and professional presentation of information (AMEWQET criteria 9.1); links to external information are present (AMEWQET criteria 10.1).

DISCUSSION

We identified and evaluated the quality of 37 education websites that provide open access educational resources relevant to anesthesia providers and trainees worldwide. The majority of websites were associated with an academic institution or national or international organizations. Only 3 websites explicitly listed their aim to support education and training of anesthesia clinicians in low- and middle-income settings. Video was the most common type of educational content included in websites, but text-based resources, podcasts, and interactive learning resources were also found. Few websites included opportunities for active engagement or interaction by learners, and few described an editorial review process or were found to be comprehensive. There were significant differences found in the quality between websites in the upper, middle, and lower tertiles in several areas including disclosure of author/webmaster/website institution; description of editorial review process; relevancy to residents, fellows, and faculty; comprehensiveness; accuracy; disclosure of content creation or revision; ease of access to information; interactivity; integration of graphic/media elements to clarify content; clear and professional presentation of information; and links to external information.

The benefits and importance of free open access medical educational materials across a continuum of learners have been previously demonstrated.3,13,14 The impact is further amplified in low- and middle-income countries (LMICs), where the number of anesthesia providers and anesthesia educators is very low.15 Identification of high-quality open access anesthesia websites can help to relieve the time and cost burden of seeking and creating educational content and be a useful resource for both anesthetic trainees and trainers in these settings. Before the COVID-19 pandemic, technology was identified as one of the major barriers to online learning, especially in LMICs.16 The acceleration in the use of online learning during the COVID-19 pandemic has shown us that this is no longer a major limitation as there has been much-needed investment into technology-based pedagogy and upsurge of incorporating online learning within institutions worldwide.17 This change along with the availability and access to high-quality online educational resources including anesthesia websites can help to support the few anesthesia educators in many LMICs in delivering high-quality teaching and help to promote global health equity. However, we found that only 3 websites explicitly listed their aim to support education and training of anesthesia clinicians in low- and middle-income settings. Thus, more websites with content explicitly directed at these settings could be useful to support the needs of clinicians practicing in these environments.

Our study found that 37 open access websites targeted toward providing education for anesthesia trainees and providers. One possible explanation for this absence might be the high costs and individual time investment associated with development and maintenance. The majority of the open access websites that we evaluated were supported by academic institutions, national or international organizations, or journals that have adequate funding to support these types of endeavors. Several possibilities exist for recovering costs associated with development and maintenance of educational websites such as allowing advertising on their websites, as we found in more than one-third of websites we evaluated. Additionally, an explicit tiered approach where those who cannot afford to pay the full subscription price are provided free access or a very nominal fee might provide a plausible solution such as the model used by the Hinari programme18 or UpToDate.19 Despite these potential challenges, other specialties have published reports citing larger numbers of open access websites for their specialties than we found for anesthesia, including 97 websites found in critical care medicine, 43 websites for otolaryngology, and 278 websites found in the field of pathology using a similar search strategy.5–7 More work is needed to identify why these specialties have a larger representation of open access websites that might provide insight into strategies that can help the anesthesia community, including organizations, academic departments, and journals to expand the availability of high-quality open access educational resources to support the needs of anesthesia providers worldwide.

One significant opportunity for the improvement of website quality relates to the relative lack of opportunities for active engagement. Despite evidence from adult and active learning theories that active engagement promotes more and deeper learning,20–24 a few evaluated websites included opportunities for interaction by the learners. One of the challenges for website designers who want to create this interactivity is cost. As authoring tools have made it easier for many clinicians to develop text, audio, and video resources, as well the website to host content, costs to develop a basic website are relatively inexpensive. However, the addition of interactivity often involves professional services of a programmer, instructional designer, or web developer, which can be outside of the scope of funding for most clinicians, unless there is substantial grant support or a partnership with a professional organization that can provide funding for these services. With some creativity and familiarity with technology, clinician educators can apply educational techniques built on established learning theories, such as the flipped classroom model, to enhance interactivity and engagement by learners.25,26 Multiple-choice questions could be added to websites to actively engage the learner and improve knowledge gains. Active learning experiences could also be designed by incorporating existing passive online content into activities that engage the learners through interactivity (ie, active engagement with a group discussion) or construction (ie, application of content through the creation of a collaborative document).22

Simpson et al27 described several responsibilities of educators that should be considered when teaching using technology. One key responsibility is that most educators should focus on curating, not creating, content. This would allow educators to use their valuable time to develop teaching sessions that promote active engagement for learners to create, reflect, and apply knowledge. As an example, a learner may be asked to watch a video on regional anesthesia technique before a teaching session. The learner would then perform the technique on a task trainer in a simulated environment. Learners could also be paired and provide feedback on each other’s technique, enhancing the ability and efficiency of faculty to teach hands-on sessions with multiple learners simultaneously. Our list of currently available websites and their associated content description should facilitate awareness of resources that could be curated and integrated into such meaningful and engaging interactive learning activities. Thoughtful and sustained integration into local educational environment, especially in the low- and middle-income settings, with alignment with the local institutional and national strategic initiatives will be essential for the successful utilization of online learning resources to meaningfully support educational endeavors.28

Very few websites we evaluated described an explicit editorial review process. The presence of an editorial process has been identified as one of the key quality indicators,9,10 and disclosure of the editorial process has been shown to help provide credibility.29 Compared to traditional peer-reviewed journals, this variable editorial process can lead to skepticism when it comes to quality. Thus, to improve transparency about quality, educators creating and hosting educational websites content should consider disclosing the editorial process including who is involved in creating and revising the resources included on the website, their qualifications, and whether the content undergoes an internal or external review.

There are several limitations to our study. We acknowledge that there are fee-based education websites or those contained in password-protected learning management systems or behind hospital firewalls that may be high quality but were inaccessible for our review. As the aim of this study was to identify websites available to anesthesia providers, trainees, and educators free of charge through an Internet-based search, fee-based or inaccessible websites were not included in our review. While other types of educational modalities exist (such as Twitter posts, videos on YouTube, and mobile applications) for anesthesia education, they each require individual and specific evaluation tools designed for those purposes, which was beyond the scope of this study and remains an opportunity for future evaluation. We also appreciate that our study provides an appraisal of the current existing landscape of open access websites in online anesthesia education at the time of our search and is likely to change as websites are constantly being developed and removed from the Internet. However, as many of the websites were affiliated with an institution or organization, they are likely to have funding for continued support. Finally, we did not evaluate usage data of these websites as this was publicly unavailable, nor did we assess how training programs or clinicians were actually using these websites, and these remain an opportunity for further study.

CONCLUSIONS

We found 37 education websites that provide open access educational resources relevant to anesthesia providers and trainees. While our findings may serve as a valuable resource for anesthesia educators who are looking to supplement synchronous learning activities with existing online learning content, more efforts are needed to expand the number, including those targeted toward clinicians working in LMICs, and improve existing open access websites, including interactivity. The ideal websites would be targeted toward the education of anesthesia clinicians, be open access and updated frequently, have a clear editorial process, and incorporate active learning strategies such as interactivity. Considering the global anesthesia workforce crisis and lack of anesthesia educators in many parts of the world, open access educational websites can help promote education equity by crossing institutional and geographical borders and provide valuable resources for our colleagues working in these areas.27 However, to transform information into education, these resources will need to be actively and sustainably integrated into formal training and accreditation programs aligned with local and national strategic priorities.

ACKNOWLEDGMENTS

The authors thank Meaghan Muir, MLIS, Manager of Library Services at Boston Children’s Hospital, for help with developing the literature and web search strategy.

DISCLOSURES

Name: Faye M. Evans, MD.

Contribution: This author helped with study design; literature search; data collections, analysis‚ and interpretation; creation of tables; writing of the manuscript; and approval of the final version of the manuscript.

Name: Anna A. Krotinger, MBE.

Contribution: This author helped with study design; literature search; data collection, analysis‚ and interpretation; creation of tables/figures; writing of the manuscript; and approval of the final version of the manuscript.

Name: Maytinee Lilaonitkul, MBBS.

Contribution: This author helped with data collection, analysis‚ and interpretation; manuscript review and editing; and approval of the final version of the manuscript.

Name: Hiba F. Khaled, MB BCh, BAO.

Contribution: This author helped with data collection, analysis‚ and interpretation; manuscript review and editing; and approval of the final version of the manuscript.

Name: Gretchen A. Pereira, MHA.

Contribution: This author helped with data collection, analysis‚ and interpretation; manuscript review and editing; and approval of the final version of the manuscript.

Name: Steven J. Staffa, MS.

Contribution: This author helped with data interpretation, manuscript review and editing, and approval of the final version of the manuscript.

Name: Traci A. Wolbrink, MD, MPH.

Contribution: This author helped in study design; literature search; data collections, analysis‚ and interpretation; creation of tables; writing of the manuscript; and approval of the final version of the manuscript.

This manuscript was handled by: Angela Enright, MB, FRCPC.

    REFERENCES

    1. Martinelli SM, Isaak RS, Schell RM, Mitchell JD, McEvoy MD, Chen F. Learners and luddites in the twenty-first century: bringing evidence-based education to anesthesiology. Anesthesiology. 2019;131:908–928.
    2. BMJ. Health Internetwork Access to Research Initiative (HINARI). Accessed March 5, 2022. https://www.bmj.com/company/research4life-free-access-to-journals/.
    3. Chan TM, Stehman C, Gottlieb M, Thoma B. A short history of free open access medical education. The past, present, and future. ATS Sch. 2020;1:87–100.
    4. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–1958.
    5. Alyusuf RH, Prasad K, Abdel Satir AM, Abalkhail AA, Arora RK. Development and validation of a tool to evaluate the quality of medical education websites in pathology. J Pathol Inform. 2013;4:29.
    6. Yang N, Hosseini S, Mascarella MA, et al. Identifying high quality medical education websites in Otolaryngology: a guide for medical students and residents. J Otolaryngol Head Neck Surg. 2017;46:42.
    7. Wolbrink TA, Rubin L, Burns JP, Markovitz B. The top ten websites in critical care medicine education today. J Intensive Care Med. 2019;131:908–928.
    8. Singh D, Alam F, Matava C. A Critical analysis of anesthesiology podcasts: identifying determinants of success. JMIR Med Educ. 2016;2:e14.
    9. Thoma B, Chan TM, Paterson QS, Milne WK, Sanders JL, Lin M. Emergency medicine and critical care blogs and podcasts: establishing an international consensus on quality. Ann Emerg Med. 2015;66:396–402.e4.
    10. Lin M, Thoma B, Trueger NS, Ankel F, Sherbino J, Chan T. Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators. Postgrad Med J. 2015;91:546–550.
    11. Chan TM, Grock A, Paddock M, Kulasegaram K, Yarris LM, Lin M. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68:729–735.
    12. Colmers-Gray IN, Krishnan K, Chan TM, et al. The revised METRIQ score: a quality evaluation tool for online educational resources. AEM Educ Train. 2019;3:387–392.
    13. O’Malley D, Barry DS, Rae MG. How much do preclinical medical students utilize the internet to study physiology? Adv Physiol Educ. 2019;43:383–391.
    14. Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019;19:36.
    15. Kempthorne P, Morriss WW, Mellin-Olsen J, Gore-Booth J. The WFSA global anesthesia workforce survey. Anesth Analg. 2017;125:981–990.
    16. Ali S, Uppal M, Gulliver S. A conceptual framework highlighting e-learning implementation barriers. Inf Technol People. 2018;31:56–180.
    17. Adefuye AO, Adeola HA, Busari J. The COVID-19 pandemic: the benefits and challenges it presents for medical education in Africa. Pan Afr Med J. 2021;40:42.
    18. Hinari. Hinari Access to Research or Health programme. Accessed March 20, 2022. https://www.who.int/hinari/en/.
    19. Kluwer W. UpToDate Donations Program. Accessed March 5, 2022. https://www.wolterskluwer.com/en/solutions/uptodate/about/donations-program.
    20. Prince M. Does active learning work? A review of the research. J Eng Educ. 2004;93: 223–231.
    21. Kolb AY, Kolb D. Experiential Learning Theory: A Dynamic, Holistic Approach to Management Learning, Education and Development. 2009.
    22. Dennick R. Constructivism: reflections on twenty five years teaching the constructivist approach in medical education. Int J Med Educ. 2016;7:200–205.
    23. Roediger HL, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006;17:249–255.
    24. Ellman MS, Schwartz ML. Online learning tools as supplements for basic and clinical science education. J Med Educ Curric Dev. 2016;3:JMECD.S18933.
    25. Kay D, Kibble J. Learning theories 101: application to everyday teaching and scholarship. Adv Physiol Educ. 2016;40:17–25.
    26. Hurtubise L, Hall E, Sheridan L, Han H. The flipped classroom in medical education: engaging students to build competency. J Med Educ Curric Dev. 2015;2:JMECD.S23895.
    27. Simpson D, Marcdante K, Souza KH, Anderson A, Holmboe E. Job roles of the 2025 medical educator. J Grad Med Educ. 2018;10:243–246.
    28. Barteit S, Jahn A, Banda SS, et al. E-learning for medical education in sub-Saharan Africa and low-resource settings: viewpoint. J Med Internet Res. 2019;21:e12449.
    29. Azim A, Beck-Esmay J, Chan TM. Editorial processes in free open access medical educational (FOAM) Resources. AEM Educ Train. 2018;2:204–212.

    Supplemental Digital Content

    Copyright © 2022 International Anesthesia Research Society