These 20 studies presented a global sample of interventions and included 7 (35%) from the United States, 2 (10%) each from the United Kingdom and Thailand, and 1 (5%) each from Australia, China, Czech Republic, Iran, Japan, Jordan, Malaysia, Nigeria, and Pakistan. All described an EBM educational intervention at a single institution, with the exception of one30 that included two institutions in the United Kingdom.
EBM instruction was delivered in classroom, clinical, and online environments, or a combination of settings. Most interventions (n = 15; 75%) included classroom instruction, whereas 8 (40%) involved clinical settings (e.g., bedside rounds). Three interventions (15%) included online environments; one of these was completely online.30
Interventions targeted medical students at all years of study. Six interventions (30%) were aimed at preclinical students, whereas 12 (60%) focused specifically on clinical students. Two (10%) of the interventions16,35 included both preclinical and clinical students, and 1 (5%) included clinical, nursing, and pharmacy students.25
Seven (58%) of the 12 interventions focused on clinical students were integrated into clerkships. Five studies named specific clerkships—pediatrics,7,27,28 family medicine,32 and internal medicine31—whereas 2 studies8,33 indicated that EBM interventions were implemented across all clerkships. The other 5 clinical-level interventions were stand-alone courses or workshops.
We were not able to determine the instructor’s profession from the descriptions provided in 8 (40%) of the included studies, including the online-only intervention.30 All 12 (60%) of the articles that provided this information identified physicians as instructors, and more than half of these interventions (n = 7) also included collaborating librarians,21,26,28,32 medical educators,35 business school faculty,29 or nurses and pharmacists.25 Residents served as instructors in 1 intervention.7 Six (30%) of the included studies described faculty development initiatives related to the EBM intervention, ranging from faculty being provided the workshop materials that would be distributed to the students34 to a half-day EBM workshop.33
EBM skills addressed
Each study described an intervention that addressed a combination of EBM skills: recognizing a knowledge gap (n = 4; 20%), asking a clinical question (n = 18; 90%), searching for literature (n = 18; 90%), appraising evidence (n = 17; 85%), applying evidence to patient care (n = 13; 65%), and evaluating the change in practice (n = 1; 5%).
Although none of the interventions addressed all six skills, 5 (25%) covered five skills. Four of these—such as the intervention for preclinical students described by Nieman et al26 that included a two-hour workshop and the completion of EBM assignments during preclinical preceptorship activities—did not address the skill of evaluating changes in practice. Although Liabsuetrakul et al10 did address evaluating changes in practice in the longitudinal classroom-based EBM course they implemented across two years of their institution’s curriculum, they did not include activities related to recognizing knowledge gaps. Eleven (55%) interventions addressed the four traditional EBM steps introduced by Strauss13: ask, acquire, appraise, and apply.
Five (25%) of the articles described a one-time educational intervention, such as a three-hour workshop designed to improve clerkship students’ clinical question formulation and literature search skills.31 Twelve (60%) included a series of interventions occurring over a single year. For example, Aronoff et al8 described a yearlong EBM course that included several sessions delivered concurrently with the first clinical year. None of the studies included longitudinal EBM curricula taught across the student’s entire tenure, but 3 (15%) described interventions implemented across more than one year.10,16,35 For example, West et al16 reported on a formal EBM curriculum that students begin in their second preclinical year and conclude in their third clinical year. We identified only one other intervention35 that trained students at both the preclinical and clinical levels.
Using Khan and Coomarasamy’s20 three-level hierarchy of EBM teaching and learning methods, we determined that 8 (40%) of the 20 interventions used level 1 (interactive, clinically integrated), 8 (40%) used level 2 (interactive, classroom-based or didactic, but clinically integrated), and 4 (20%) used level 3 (didactic, classroom-based, or stand-alone) methods. Aronoff and colleagues8 provided an example of level 1 methods in their description of a multipart EBM course that included online instructional modules and formalized activities generated from students’ formulating their own clinical questions based on their clerkship experiences.
Physicians’ failure to engage in EBM has repercussions for the health of individuals and populations.4,5 It is therefore essential that medical students receive foundational training in EBM. To provide medical educators with an overview of current EBM training, which has become a common topic covered in medical education,36 we reviewed 20 recent studies describing a range of educational interventions taught in a variety of settings and aimed at students at all levels of UGME. Among these interventions, we identified an emphasis on the skills of forming clinical questions, searching the literature, and appraising evidence critically. Our findings suggest there is a need for broader, more comprehensive EBM training, especially in the domains of recognizing a knowledge gap and evaluating changes in practice.
As the reviewed studies lacked robust outcome measures, it was not possible to determine the efficacy of interventions. Therefore, we consider our findings within the broader contexts of UGME and health care. Then, on the basis of evidence from the reviewed literature and our professional experience—as a physician, as medical educators, and as medical librarians—we offer recommendations for modifying current EBM training.
Implications and recommendations for UGME
Timing of EBM instruction.
There is little evidence as to the most efficacious timing for EBM instruction.37 Yet, our review confirms earlier findings that most EBM educational interventions take place in the clinical years of medical school.38,39 This timing is generally based on the rationale that the clerkship setting enhances the clinical relevance of the training.40 However, the trend toward providing students with early clinical experiences41 may provide opportunities for introducing EBM earlier, in the context of patient care in the preclinical years. For example, one study demonstrated that first-year medical students who received EBM training prior to a primary care preclinical preceptorship reported increases in self-efficacy and a likelihood that they will continue to use EBM.26 We suggest that medical educators consider integrating EBM instruction into early clinical experiences, as doing so may increase students’ self-efficacy and provide a framework that helps students deal with the uncertainty of being new to the clinical setting.
Longitudinal EBM training.
Although multiple exposures to EBM training were common in the studies reviewed, longitudinal curricula were lacking. In 75% of the included interventions, medical students received EBM training on more than one occasion, a practice that has been linked generally with increased learning.42 Yet most of the interventions were delivered over short, intensive time periods. The compressed nature of these learning opportunities limits medical educators’ ability to successively build EBM skills across levels of student development. In a spiral curriculum43—a format that has been adopted to teach some components of medical education—learners are provided multiple, successive exposures to a concept at different levels of their development so that each encounter builds on the previous encounter.44 We suggest that integrating EBM training as a spiral curriculum across all levels of UGME may be an effective model.
We identified only one intervention that included both medical students and learners from other health professions.25 Given the trend in medical education toward recommending the use of interprofessional education (IPE)45 and the adoption of evidence-based practice by a spectrum of allied health professions, we encourage medical educators to consider taking an IPE approach to EBM instruction. Teaching EBM in an IPE environment could expand students’ exposure to a variety of discipline-specific, evidence-based resources and provide them with insight into the ways in which learners and professionals in other fields cope with uncertainty.
Related to IPE, we found several interventions that were taught by both physicians and instructors from other disciplines. Some included medical librarians, whose participation has been shown to add authenticity to EBM training.28 Of note, one intervention29 reached beyond the medical school’s resources to integrate business school faculty, who shared their expertise in change management techniques and organizational behavior. We recommend that medical educators consider including instructors from a variety of disciplines and think beyond the classroom setting when integrating multidisciplinary teachers. For example, all of the interventions that involved medical librarians as instructors were set in classrooms. A classroom-based approach, however, does not enable clinical librarians, who have been working at the point of care for over 30 years,46 to demonstrate real-time information retrieval and interprofessional collaboration.
Active and online learning environments.
Recently, Prober and Heath47 declared, “It’s time to change the way we educate doctors,” and advocated a shift from a lecture-based format to an active learning approach that blends online learning with more-interactive classroom activities, such as case studies.47 We found that EBM interventions for medical students are already using interactive teaching methods and online learning. Three interventions28,30,32 employed an online format to decompress classroom time, extend the reach of strained faculty resources, mitigate scheduling difficulties, and/or reach students at diffuse clinical sites. As online instruction—which has been shown to be as effective as in-person instruction for EBM32,48—continues to evolve, we suggest that educators look for opportunities to increase the use of online components in EBM curricula.
Gaps in knowledge.
Finally, medical educators have dedicated much attention to the physician’s ability to self-monitor, that is, to recognize the limitations of one’s skill and knowledge to act in a specific situation.49,50 In the context of EBM, this self-monitoring connects with the practitioner’s ability to recognize gaps in his or her knowledge, which has been dubbed “step zero” in evidence-based practice.51 Being able to identify awareness of a knowledge gap is critical, as doing so acts as the fuel that ignites the EBM process and prompts the physician to ask clinical questions and proceed through the subsequent steps. Yet, whereas 90% of reviewed interventions addressed the mechanics of asking a clinical question, only 20% addressed the necessary precursor of identifying knowledge gaps. For example, one intervention26 challenged students to identify and record at least four knowledge gaps generated by encounters with patients. On the basis of our review findings, and research showing that physicians tend to be weak in recognizing when they have clinical questions,52 we suggest that all EBM training should cover the essential skill of recognizing a knowledge gap.
Implications for health care and related recommendations
In the Patient Protection and Affordable Care Act of 2010, the United States committed funding to support patients and clinicians in making shared, evidence-based decisions.53 Although a handful of included interventions7,8,16,28,29,33 required students to generate clinical questions based on their patient encounters and contemplate how they might apply evidence to patients’ care, none included training on discussing evidence with patients. We suggest that medical educators consider how to provide opportunities for students to engage in sharing evidence with patients to facilitate decision-making activities.
The related scenario of the patient as the bearer of evidence was also absent from the reviewed interventions. Recent research has demonstrated that physicians “experience considerable anxiety in response to patients bringing information from the internet to a consultation,”54 which suggests that this is an area ripe for improvement. As it is probable that patients will increasingly bring information to appointments, we recommend that medical educators train students to evaluate the evidence retrieved by patients and to engage in productive conversations in which the student and patient can reflect on the information and, if appropriate, integrate it into the patient’s care.
Electronic health records.
Although all of the reviewed interventions included instruction on how to search the literature using traditional resources (e.g., PubMed), none addressed how to access information via electronic health records (EHRs). EHRs are becoming increasingly prevalent in medical practice, and their capabilities are improving with regard to delivering information and prompting clinical questions within the workflow via alerts and “infobuttons.”55 Educators therefore need to ensure that EBM training accommodates these evolving resources. For example, they may need to shift emphasis from PubMed search skills to information management skills, such that students learn to manage or triage point-of-care information presented within EHRs. Additionally, educators should seize the opportunity to use EHRs to facilitate EBM teaching. For example, students could be asked to summarize any information they consulted in caring for a patient and attach the summary to the patient’s EHR. This would provide students with authentic EBM opportunities and also broaden the evidence base of all health professionals who interact with the patient’s EHR.
The increasing use of EHRs may also expand the types of available evidence. Recently, a physician team at Stanford was temporarily stymied by the lack of published evidence related to the treatment of a complicated pediatric case. However, by querying the EHR system, they identified a cohort of similar patients and analyzed outcomes data to make an informed treatment decision.56 Although this is currently considered a “novel” process, such use of EHR data is likely to become increasingly prevalent. We therefore recommend that medical educators expand the coverage of searching the biomedical literature to include querying EHRs.
This review must be considered in the context of its limitations. We only captured descriptions of UGME interventions that targeted multiple EBM skills and were published during 2006–2011. It is also possible that there are institutions that have robust EBM education programs but have not published studies on these programs. Further, we were not able to judge the strength of the interventions and make recommendations accordingly.
Given the importance of EBM to providing high-quality patient care and widespread application of this approach across health professions, it is surprising that UGME training programs for EBM skill development have not been well described in the literature. Better descriptions of interventions and more rigorous research methods are needed so that educators and researchers can draw conclusions about the efficacy of interventions and, if so desired, replicate them. We encourage educators to consider designing EBM interventions that are in alignment with major trends in medical education and health care, including IPE, EHRs, and patient-centered care, and publishing their findings.
Acknowledgments: The authors wish to thank Dr. David Irby for his feedback on an earlier version of the manuscript and Ms. Olya Gary for her assistance with the search strategy figure.
Other disclosures: None.
Ethical approval: Not applicable.
1. Bloch RM, Swanson MS, Hannis MD. An extended evidence-based medicine curriculum for medical students. Acad Med. 1997;72:431–432
2. van Dijk N, Hooft L, Wieringa-de Waard M. What are the barriers to residents’ practicing evidence-based medicine? A systematic review. Acad Med. 2010;85:1163–1170
3. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What it is and what it isn’t. BMJ. 1996;312:71–72
4. Banzi R, Cinquini M, Liberati A, et al. Speed of updating online evidence based point of care summaries: Prospective cohort analysis. BMJ. 2011;343:d5856
5. Dawes M, Summerskill W, Glasziou P, et al.Second International Conference of Evidence-Based Health Care Teachers and Developers. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5:1
6. Tilson JK, Kaplan SL, Harris JL, et al. Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ. 2011;11:78
7. Lai NM, Teng CL. Self-perceived competence correlates poorly with objectively measured competence in evidence based medicine among medical students. BMC Med Educ. 2011;11:25
8. Aronoff SC, Evans B, Fleece D, Lyons P, Kaplan L, Rojas R. Integrating evidence based medicine into undergraduate medical education: Combining online instruction with clinical clerkships. Teach Learn Med. 2010;22:219–223
9. Johnston JM, Schooling CM, Leung GM. A randomised-controlled trial of two educational modes for undergraduate evidence-based medicine learning in Asia. BMC Med Educ. 2009;9:63
10. Liabsuetrakul T, Suntharasaj T, Tangtrakulwanich B, Uakritdathikarn T, Pornsawat P. Longitudinal analysis of integrating evidence-based medicine into a medical student curriculum. Fam Med. 2009;41:585–588
11. Chen HC, Tan JP, O’Sullivan P, Boscardin C, Li A, Muller J. Impact of an information retrieval and management curriculum on medical student citations. Acad Med. 2009;84(10 suppl):S38–S41
12. Lai NM, Nalliah S. Information-seeking practices of senior medical students: The impact of an evidence-based medicine training programme. Educ Health (Abingdon). 2010;23:151
13. Strauss SE Evidence-Based Medicine: How to Practice and Teach EBM. 20053rd ed New York, NY Elsevier/Churchill Livingstone
14. Wyer PC, Naqvi Z, Dayan PS, Celentano JJ, Eskin B, Graham MJ. Do workshops in evidence-based practice equip participants to identify and answer questions requiring consideration of clinical research? A diagnostic skill assessment. Adv Health Sci Educ Theory Pract. 2009;14:515–533
15. Maggio LA, Tannery NH, Kanter SL. AM Last Page: How to perform an effective database search. Acad Med. 2011;86:1057
16. West CP, Jaeger TM, McDonald FS. Extended evaluation of a longitudinal medical school evidence-based medicine curriculum. J Gen Intern Med. 2011;26:611–615
18. . Qualtrics [computer program]. Version 12,018. 2009 Provo, Utah Qualtrics
19. Kirkpatrick DCraig RL, Bittel LR. Evaluation of training. In: Training and Development Handbook. 1967 New York, NY McGraw-Hill:87–112
20. Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006;6:59
21. Chakraborti C. Teaching evidence-based medicine using team-based learning in journal clubs. Med Educ. 2011;45:516–517
22. Iqbal M, Zaidi Z. The “ripple effect” of introducing evidence-based medicine into a curriculum. Med Educ. 2009;43:475
23. Khader YS, Batayha W, Al-Omari M. The effect of evidence-based medicine (EBM) training seminars on the knowledge and attitudes of medical students towards EBM. J Eval Clin Pract. 2011;17:640–643
24. McMahon GT, Dluhy RG. Impact of a novel evidence-based medicine curriculum. Med Educ. 2006;40:464–465
25. Nango E, Tanaka Y. Problem-based learning in a multidisciplinary group enhances clinical decision making by medical students: A randomized controlled trial. J Med Dent Sci. 2010;57:109–118
26. Nieman LZ, Cheng L, Foxhall LE. Teaching first-year medical students to apply evidence-based practices to patient care. Fam Med. 2009;41:332–336
27. Okoromah CA, Adenuga AO, Lesi FE. Evidence-based medicine curriculum: Impact on medical students. Med Educ. 2006;40:465–466
28. Potomkova J, Mihal V, Zapletalova J, Subova D. Integration of evidence-based practice in bedside teaching paediatrics supported by e-learning. Biomed Pap Med Fac Univ Palacky Olomouc Czech Repub. 2010;154:83–87
29. Rhodes M, Ashcroft R, Atun RA, Freeman GK, Jamrozik K. Teaching evidence-based medicine to undergraduate medical students: A course integrating ethics, audit, management and clinical epidemiology. Med Teach. 2006;28:313–317
30. Sandars J, Siddiqi K, Walsh K, Richardson J, Ibison J, Maxted M. An undergraduate education package on evidence-based medicine: Some NICE lessons. Med Educ. 2010;44:511–512
31. Sastre EA, Denny JC, McCoy JA, McCoy AB, Spickard A 3rd. Teaching evidence-based medicine: Impact on students’ literature use and inpatient clinical documentation. Med Teach. 2011;33:e306–e312
32. Schilling K, Wiecha J, Polineni D, Khalil S. An interactive Web-based curriculum on evidence-based medicine: Design and effectiveness. Fam Med. 2006;38:126–132
33. Stockler MR, March L, Lindley RI, Mellis C. Students’ PEARLS: Successfully incorporating evidence-based medicine in medical students’ clinical attachments. Evid Based Med. 2009;14:98–99
34. Taheri H, Mirmohamadsadeghi M, Adibi I, Ashorion V, Sadeghizade A, Adibi P. Evidence-based medicine (EBM) for undergraduate medical students. Ann Acad Med Singapore. 2008;37:764–768
35. Wanvarie S, Sathapatayavongs B, Sirinavin S, Ingsathit A, Ungkanont A, Sirinan C. Evidence-based medicine in clinical curriculum. Ann Acad Med Singapore. 2006;35:615–618
36. West CP, McDonald FS. Evaluation of a longitudinal medical school evidence-based medicine curriculum: A pilot study. J Gen Intern Med. 2008;23:1057–1059
37. Holloway R, Nesbit K, Bordley D, Noyes K. Teaching and evaluating first and second year medical students’ practice of evidence-based medicine. Med Educ. 2004;38:868–878
38. Finkel ML, Brown HA, Gerber LM, Supino PG. Teaching evidence-based medicine to medical students. Med Teach. 2003;25:202–204
39. Keim SM, Howse D, Bracke P, Mendoza K. Promoting evidence based medicine in preclinical medical students via a federated literature search tool. Med Teach. 2008;30:880–884
40. Ilic D. Teaching evidence-based practice: Perspectives from the undergraduate and post-graduate viewpoint. Ann Acad Med Singapore. 2009;38:559–555
41. Dornan T, Littlewood S, Margolis SA, Scherpbier A, Spencer J, Ypinazar V. How can experience in clinical and community settings contribute to early medical education? A BEME systematic review. Med Teach. 2006;28:3–18
42. Norman G. The American College of Chest Physicians evidence-based educational guidelines for continuing medical education interventions: A critical review of evidence-based educational guidelines. Chest. 2009;135:834–837
43. Harden RM. What is a spiral curriculum? Med Teach. 1999;21:141–143
44. Bruner JS The Process of Education. 1963 New York, NY Vintage Books
45. Thompson C. Do interprofessional education and problem-based learning work together? Clin Teach. 2010;7:197–201
46. Marshall JG, Hamilton JD. The clinical librarian and the patient: Report of a project at McMaster University Medical Centre. Bull Med Libr Assoc. 1978;66:420–425
47. Prober CG, Heath C. Lecture halls without lectures—A proposal for medical education. N Engl J Med. 2012;366:1657–1659
48. Davis J, Crabb S, Rogers E, Zamora J, Khan K. Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: A randomized controlled trial. Med Teach. 2008;30:302–307
49. Eva KW, Regehr G. Knowing when to look it up: A new conception of self-assessment ability. Acad Med. 2007;82(10 suppl):S81–S84
50. McConnell MM, Regehr G, Wood TJ, Eva KW. Self-monitoring and its relationship to medical knowledge. Adv Health Sci Educ Theory Pract. 2012;17:311–323
51. Johnston L, Fineout-Overholt E. Teaching EBP: “Getting from zero to one.” Moving from recognizing and admitting uncertainties to asking searchable, answerable questions. Worldviews Evid Based Nurs. 2005;2:98–102
52. Ely JW, Burch RJ, Vinson DC. The information needs of family physicians: Case-specific clinical questions. J Fam Pract. 1992;35:265–269
53. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307:1583–1584
54. Ahluwalia S, Murray E, Stevenson F, Kerr C, Burns J. “A heartbeat moment”: Qualitative study of GP views of patients bringing health information from the Internet to a consultation. Br J Gen Pract. 2010;60:88–94
55. Cimino JJ. Infobuttons: Anticipatory passive decision support. AMIA Annu Symp Proc. November 6, 2008:1203–1204
© 2013 by the Association of American Medical Colleges
56. Frankovich J, Longhurst CA, Sutherland SM. Evidence-based medicine in the EMR era. N Engl J Med. 2011;365:1758–1759