Rao, Goutham MD; Kanter, Steven L. MD
Teaching medical students and residents to interpret original research accurately and to apply results to patient care is important. Evidence-based medicine (EBM) is a useful framework for this purpose. Unfortunately, the statistical aspects of EBM, which are important for accurate interpretation of research, are neglected during training. In this article, we provide a detailed description and critical analysis of an approach to teaching the accurate interpretation and application of study results, in which the quantitative or numerical aspects of studies are the focus.
EBM is the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.1” It includes five essential skills: (1) recognition of a problem and construction of a structured clinical question, (2) efficient and effective searching for information, (3) critical appraisal of the evidence, (4) gaining a full understanding of the results (e.g., understanding the impact of a new therapy when expressed as an odds ratio for development of a negative outcome), and (5) integration of the evidence into patient care.2 EBM is included in the Practice-Based Learning and Improvement core competency described by the Accreditation Council for Graduate Medical Education. The Association of American Medical Colleges (AAMC) supports teaching EBM to medical students as a general objective and as part of training in understanding and improving quality of care.3,4 National efforts to reform health care also emphasize evidence-based care.5
Though the need to teach EBM to medical students is accepted and many medical schools include EBM curricula, the emphasis in such instruction is on constructing questions, searching, and critical appraisal, and to a lesser extent on integration of evidence into patient care, with little attention paid to accurate interpretation of results. Accurate interpretation of results requires understanding of biostatistics and research design.6 Application of those results to patient care also requires quantitative skills, such as calculating and conveying risk to patients in a meaningful way. The importance of these skills is widely recognized. Among 301 students, residents, and teaching faculty surveyed at the Mayo Clinic, for example, 92.7% believed that biostatistics is an important part of EBM, and 88.0% believed that knowledge of biostatistics is necessary when evaluating medical literature. In general, attitudes toward biostatistics among clinicians are positive.7,8 Despite these positive attitudes, physicians and physicians-in-training have been shown repeatedly to have poor understanding of the statistical aspects of studies and to lack confidence in their ability to interpret study results.8–12
Teaching students and residents to interpret research accurately is therefore an important curricular need. Medical educators seldom have training in the statistical aspects of EBM and could benefit from training as well. Patients are also important stakeholders in improving EBM teaching, since they are directly affected by the decisions made by clinicians when they interpret research evidence. Moreover, through the Internet and other media, patients are increasingly aware of emerging and alternative therapies and tests, and they require accurate information from their physicians about these tests and therapies to make rational decisions.
With respect to EBM, “physician numeracy” can be defined as understanding the statistical aspects of and terminology associated with the design, analysis, and results of original research. This includes, for example, having a general understanding of how the power of a study is calculated, or understanding the meaning of a confidence interval. EBM has become an integral and valuable part of undergraduate and postgraduate medical curricula, but current EBM teaching overemphasizes critical appraisal and pays too little attention to physician numeracy, which is essential for accurate interpretation of research. We posit that physician numeracy, rather than other EBM skills, can form the basis for teaching EBM.
Current Approaches to EBM Teaching and Numeracy
How EBM is taught in residency programs
A number of reports describe EBM teaching in residencies. Journal club is the predominant format.13,14 Alguire15 points out that journal clubs have become popular because they are easy to implement, require little preparation, are comfortable for faculty, and emphasize resident-centered learning. Despite widespread incorporation of journal clubs into residency programs, self-perceived EBM skills among residents are poor.16 A recent survey of 26 internal medicine programs revealed that all included journal clubs, 18 included formal sessions on searching the literature, 14 included lectures on critiquing the literature, and 8 included EBM assignments.17 No program devoted time specifically for understanding research design or interpreting quantitative results. A 1999 study concluded that residency EBM curricula emphasized critical appraisal to the exclusion of other EBM skills and that such curricula were ineffective.18
Only one published article describes a resident-level EBM curriculum which emphasizes numeracy. Pediatrics residents at the Mayo Clinic meet in small groups roughly every two weeks to discuss study designs and key statistics used to summarize results including numbers needed to treat (NNTs), likelihood ratios, and confidence intervals. The Mayo curriculum also includes the more commonly taught EBM skills of formulating questions and searching for studies.19
It can be concluded that critical appraisal and searching the literature are emphasized in residency EBM teaching, often as part or in conjunction with journal clubs. Teaching in this way in residency programs has not been shown to improve residents' EBM skills.
How EBM is taught to medical students
A 1999 survey of internal medicine clerkships revealed that 38.5% incorporated EBM, but to what extent numeracy was included was not specified.20 A survey of U.S. medical schools in 2002–2003 revealed that 106 schools included a mean of 20 hours of instruction in EBM. One hundred seven schools included a mean of 13 hours of training in biostatistics. One hundred ten schools included a mean of 16 hours of training in epidemiology.21 The survey shows, therefore, that medical schools make a significant commitment to these areas, but the specific topics and the extent to which they were covered were not specified.
To obtain a clearer picture of how EBM is taught to medical students, we conducted a literature search to identify relevant articles and reports that describe complete, freestanding EBM curricula. Curricula for epidemiology or biostatistics outside the context of EBM were not included. We searched the Medline database from 1999 to 2009 using the MeSH terms “evidence-based medicine” and “medical students,” and subheading, “education.” We also searched the AAMC's online MedEdPORTAL using the term “evidence-based medicine.” Finally, we used the search term “evidence-based medicine AND curriculum AND medical students OR medical school” in the Google search engine to identify additional curricula that were not published in journals or listed in the MedEdPORTAL. Only reports published in English that described established EBM curricula (as opposed to evaluation tools, etc.) were reviewed closely to determine the EBM content included. We specifically set out to identify any presence of physician numeracy in the curricula described. To qualify as including physician numeracy, reports had to explicitly list one or more quantitative terms, skills, or statistics associated with research design or analysis (e.g., likelihood ratios, confidence intervals). Reports describing critical appraisal without mention of specific quantitative aspects of design and analysis were not considered to have included physician numeracy. The results of the search are summarized in Table 1.
We retrieved 28 reports of specific EBM curricula. Nine included numeracy. Three of these nine did not specify which topics were covered.22–24 The six articles detailing physician numeracy topics are summarized below.
* Schilling et al25 describe an interactive, Web-based EBM curriculum for family medicine clerkship students. Numeracy was limited to calculation of the NNT statistic.
* Cayley26 describes a six-hour EBM curriculum within a primary care rotation, of which one hour was devoted to “basic EBM statistics” including test characteristics (sensitivity, etc.), likelihood ratios, and NNT/number needed to harm (NNH).
* Hunt et al27 describe a seven-week (15-hour) EBM course for second-year medical students. The course was taught in small groups, using a team learning approach that involved both independent and group study. Three of the seven sessions collectively included the following numeracy topics: process of randomization, study design, P values, confidence intervals, relative and absolute risk reduction (RRR and ARR), NNT, test characteristics, and influence of prevalence on test performance. Students were expected to understand the conceptual definitions of these terms and/or perform calculations.
* Forjuoh et al28 describe a five-hour curriculum that incorporates both complementary and alternative medicine and EBM within a family medicine clerkship. Three hours consisted of lectures which included basic principles of epidemiology and biostatistics (specific topics were not described), study design, screening, statistical tests of significance, and meta-analysis.
* Weaver29 has assembled a four-lecture online EBM curriculum, including one lecture devoted to statistics that features an overview of descriptive statistics, z scores and the normal distribution, the central limit theorem, confidence intervals, P values, type 1 and type 2 errors, power, and a brief description of common statistical tests (e.g., chi-square, Fisher exact test, Kruskal–Wallis test) and procedures. Given the short duration of the curriculum, these topics are not covered in any significant depth.
* Srinivasan et al30 describe an EBM course for first-year medical students that comprises two 1-hour lectures and three 2-hour small-group sessions in which students were expected to apply concepts learned in lectures using clinical vignettes. They were also encouraged to use online and written EBM resources. The authors describe a large number of numeracy topics for such a short curriculum, the majority of which were covered in lectures. These included understanding or interpreting the following: types of data, descriptive statistics, rules of probability, randomization/blinding/bias, event rates, relative risk/RRR/ARR/NNT/NNH, confidence intervals, power, type 1/type 2 error, sample size calculations, study design, test characteristics, and odds ratios. Evaluation of the course revealed high student satisfaction. Students performed well on a final examination that used an open-book format.
In general, our literature review revealed that numeracy is conspicuously absent from the majority of EBM curricula. The skills of formulating a sound clinical question, literature searching, and critical appraisal dominate medical school teaching. A few curricula describe a large number of numeracy topics, but their short duration likely inhibits in-depth coverage.
An Innovative Approach to Teaching EBM Based on Numeracy
We developed an innovative approach to EBM education at the University of Pittsburgh using numeracy as a foundation. We implemented this approach in a course called Introduction to Medical Decision Making, taught to first-year medical students. This course was developed gradually between 2001 and 2003 by the course director (G.R.) and a planning committee consisting of the course director and five course faculty, with the support and guidance of the vice dean (S.L.K.). It replaced a course in clinical epidemiology and biostatistics, which covered epidemiological principles, elements of research design, and basic descriptive and comparative biostatistics. Such courses were once common in medical schools but have largely been replaced by learning experiences in which EBM is the main focus. At the outset, the design of Introduction to Medical Decision Making made use of five guiding principles:
1. Journal clubs have important limitations. They overemphasize critical appraisal—used to determine whether an article is useful or not—by using predetermined criteria for relevance and validity. Journal club sessions typically involve the same tasks repeated each time. Individual sessions lack specific objectives. Overall, journal clubs are not a useful way to improve EBM knowledge or skills.31
2. Understanding the quantitative aspects of research design and data analysis promotes an in-depth understanding of original research (which is critically important to begin in the early stages of medical education).
3. Physician numeracy, rather than the more commonly taught skills of literature searching and critical appraisal, can form the basis of an EBM course.
4. As they read a research article, instead of asking themselves whether the article is useful or not, students ought to ask, “What is useful about this article?”
5. The content of an EBM course should be based only on those numeracy concepts and skills that are needed to accurately interpret evidence and apply it to individual patients.
The course makes use of the existing EBM framework in which articles are divided into categories including diagnosis, therapy/prevention, etiology/prognosis, survival, etc.32 Content, however, emphasizes concepts and quantitative aspects of research design and data analysis. Critical appraisal is not explicitly discussed, under the assumption that students who understand the methodology of articles in general will independently be able to provide critical reviews.
The course is allotted 32 hours in the first-year curriculum (including a two-hour final examination). Like the majority of first-year medical school courses at the University of Pittsburgh, the Introduction to Medical Decision Making course uses a mix of lectures and small-group sessions that is tailored to the needs of this course. The current version of the course comprises nine 2-hour lecture sessions and six 2-hour small-group sessions. Lectures and small groups are divided into modules, each pertaining to a category of articles or research. Lecture topics are introduced throughout the course using clinical vignettes.
Small groups consist of 8 to 10 students and are led by a clinical faculty member with an interest and expertise in research or EBM. The content of five of the small-group sessions is based on lectures immediately preceding them. In advance of each small-group session, students complete an “analytic exercise” during which they read an original research article and answer specific questions about the methodology or results. Questions pertaining to one analytic exercise are shown in the supplemental digital appendix (http://links.lww.com/ACADMED/A24). During the first hour of each session, students discuss their responses to the questions. Faculty leaders have access to correct or recommended responses and lead these discussions. The second hour of each session is flexible and is decided on by the students and faculty leaders. Some groups elect to discuss concepts covered in class. Some faculty leaders discuss research methodology and analysis in the context of their own research.
During the final small-group session, students are required to present an original research article in a structured format, designed to evaluate their understanding of research design and statistical analysis. The final examination consists of 65 multiple-choice questions. A complete list of modules, lectures, topics, and small-group sessions is shown in List 1.
Rationale for an EBM curriculum based on physician numeracy
An EBM curriculum based on physician numeracy offers distinct advantages. First, such a curriculum addresses the important need physicians and physicians-in-training themselves have identified as necessary for accurate interpretation of studies. The goal of physician numeracy is to provide skills, not only for interpretation of research, but also for application to patient care. This is in marked contrast to journal clubs, which trainees often perceive as an EBM activity distinct from patient care.15 For example, our curriculum incorporates an introduction to Bayes' theorem and its application to diagnostic testing. Students are taught to combine diagnostic test results with what is already known about a patient's condition (which can be expressed mathematically as the pretest probability or odds of an outcome or disease). They are also taught the important principle that diagnostic tests are most useful in patients with intermediate (rather than high or low) pretest probabilities of an outcome or disease. This principle can be directly applied to patient care when the question of ordering a diagnostic test comes up. By contrast, a purely critical appraisal discussion of a research article about a diagnostic test is likely to omit this principle. Finally, because physician numeracy emphasizes a broad range of quantitative skills and includes a review of the design and analysis of different types of studies, it is more likely than current EBM approaches to stimulate trainees' interest in pursuing research and other types of scholarly activity. This is extremely important, as the number of physician–scientists is rapidly declining.33
During our nearly 10 years of experience in teaching EBM to medical students using a physician numeracy framework, we have encountered several challenges. The experience and comfort of medical students with statistics and mathematics varies widely. The problem is not convincing students that numeracy is important but, rather, convincing them that they can learn sometimes difficult and abstract concepts if these concepts are presented simply and systematically.
Our experience has revealed two additional important challenges. The Introduction to Medical Decision Making course takes place during the first year of the curriculum, but physician numeracy is easily forgotten if it is not reinforced. Some reinforcement of content takes place in two follow-up courses and through completion of a scholarly project required of all medical students. The difficulty, as with many other aspects of the curriculum, has been to reinforce important concepts throughout the four years of training in both clinical and nonclinical experiences. This is closely related to what is perhaps the greatest challenge.
Students completing Introduction to Medical Decision Making emerge with considerable knowledge about and confidence in interpreting original research. By contrast, our clinical faculty members have not had the same opportunity to acquire this knowledge. In the early years of the course, it was a challenge to recruit faculty members with either the knowledge of or the willingness to learn what was needed to facilitate small groups. This challenge has been met gradually as faculty members have gained experience. Most faculty members in clinical settings, however, are generally unfamiliar with basic statistical and related concepts. The end result is that there are few faculty members who are able to reinforce the course content. Like physicians in general, our faculty have expressed a sincere interest in physician numeracy. We are trying to meet this faculty development need by providing an online faculty development curriculum based on the Introduction to Medical Decision Making course.34
How best to teach EBM and which content to emphasize will continue to evolve. We do not claim that physician numeracy is the definitive, final answer. We have pursued this strategy as an alternative to formal courses in statistics or epidemiology or to journal clubs which emphasize critical appraisal. Our approach does not abandon other EBM skills. Rather, it shifts the emphasis to a neglected, but crucial, area. The practice of medicine has changed dramatically over the past 15 years. The way medicine is taught in general has also changed significantly, especially with new teaching technologies. The way EBM is taught today, however, remains virtually unchanged since the important “Users Guide to the Medical Literature” articles first appeared in the 1990s.2 We believe that the way EBM is taught deserves the same scrutiny as the research articles EBM was designed to evaluate.
1Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence-based medicine: What it is and what it isn't. BMJ. 1996;312:71–72.
2Guyatt G, Rennie D, eds. Users' Guides to the Medical Literature: A Manual for Evidence-Based Practice. Chicago, Ill: AMA Press; 2002.
6Monahan J. Statistical literacy: A prerequisite for evidence-based medicine. Psychol Sci Public Interest. 2008;8(2):1–2.
7Swift L, Miles S, Price GM, Shepstone L, Leinster SJ. Do doctors need statistics? Doctors' use of and attitudes to probability and statistics. Stat Med. 2009;28:1969–1981.
8Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. JAMA. 2007;298:1010–1022.
9Berwick DM, Fineberg HV, Weinstein MC. When doctors meet numbers. Am J Med. 1981;71:991–998.
10Weiss ST, Samet JM. An assessment of physician knowledge of epidemiology and biostatistics. J Med Educ. 1980;55:692–697.
11Wuff HR, Andersen B, Brandenhoff P, Guttler F. What do doctors know about statistics? Stat Med. 1987;6:3–10.
12Gramling R, Irvin JE, Nash J, Sciamanna C, Culpepper L. Numeracy and medicine: Key family physician attitudes about communicating probability with patients. J Am Board Fam Med. 2004;17:473.
13Kersten HB, Randis TM, Giardino AP. Evidence-based medicine in pediatric residency programs: Where are we now? Ambul Pediatr. 2005;5:302–305.
14Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal clubs. Moving toward an integrated evidence-based medicine curriculum. J Gen Intern Med. 2006;21:538–541.
15Alguire P. A review of journal clubs in post-graduate medical education. J Gen Intern Med. 1998;13:347–353.
16Watkins R, Wilson WC, Richardson S. Chief residents' skills with evidence-based medicine: Still hazy after all these years. J Gen Intern Med. 2004;19(suppl):130.
18Green ML. Graduate medical education training in clinical epidemiology, critical appraisal, and evidence-based medicine: A critical review of curricula. Acad Med. 1999;74:686–694.
20Aiyer M, Hemmer P, Meyer L, Albritton A, Levine S, Reddy S. Evidence-based medicine in internal medicine clerkships: A national survey. South Med J. 2002;95:1389–1395.
21Barzansky B, Etzel SI. Educational programs in US medical schools, 2002–2003. JAMA. 2003;290:1190–1196.
22Wanvarie S, Sathapatayavongs B, Sirinavin S, Ingsathit A, Ungkanont A, Sirinan C. Evidence-based medicine in clinical curriculum. Ann Acad Med Singapore. 2006;35:615–618.
23Kuhnigk O, Bothern AM, Schoder V, Bullinger M, Harendza S. Evidence-based medicine at medical school: A three-week course on the basis of problem-based learning. GMS Z Med Ausbild. 2007;24(4):Doc187.
24Elnicki MD, Halperin AK, Schockcor WT, Aronoff SC. Multidisciplinary evidence-based journal clubs: Curriculum design and participants' reactions. Am J Med Sci. 1999;317:243–246.
25Schilling K, Wiecha J, Polineni D, Khalil S. An interactive Web-based curriculum on evidence-based medicine: Design and effectiveness. Fam Med. 2006;38:126–132.
26Cayley WE. Evidence-based medicine for medical students: Introducing EBM in a primary care rotation. Wisc Med J. 2005;104:34–37.
27Hunt DP, Haidet P, Coverdale JH, Richards B. The effect of using team learning in an evidence-based medicine course for medical students. Teach Learn Med. 2003;15:131–139.
28Forjuoh SN, Rascoe TG, Symm B, Edwards JC. Teaching medical students complementary and alternative medicine using evidence-based principles. J Altern Complement Med. 2003;9:429–439.
30Srinivasan M, Weiner M, Breitfeld PP, Brahmi F, Dickerson KL, Weiner G. Early introduction of an evidence-based medicine course to preclinical medical students. J Gen Intern Med. 2002;17:58–65.
31Stockler MR, March L, Lindley RI, Mellis C. ACP Journal Club. Students PEARLS: Successfully incorporating evidence-based medicine in medical students' clinical attachments. Ann Intern Med. 2009;150:JC4-2, JC4-3.
32Keim SM, Howse D, Bracke P, Mendoza K. Promoting evidence-based medicine in preclinical medical students via a federated literature search tool. Med Teach. 2008;30:880–884.
33Taheri H, Mirmohamadsadeghi M, Adibi I, Ahorion V, Sadeghizade A, Adibi P. Evidence-based medicine (EBM) for undergraduate medical students. Ann Acad Med Singapore. 2008;37:764–768.
Reference Cited Only in Table 1
35West CP, McDonald FS. Evaluation of a longitudinal medical school evidence-based medicine curriculum: A pilot study. J Gen Intern Med. 2008;23:1057–1059.
36Nieman LZ, Cheng L, Foxhall LE. Teaching first-year medical students to apply evidence-based practices to patient care. Fam Med. 2009;41:332–336.
37Krueger PM. Teaching critical appraisal: A pilot randomized controlled outcomes trial in undergraduate osteopathic medical education. J Am Osteopath Assoc. 2006;106:658–662.
38Rhodes M, Ashcroft R, Atun RA, Freeman GK, Jamrozik K. Teaching evidence-based medicine to undergraduate medical students: A course integrating ethics, audit, management, and clinical epidemiology. Med Teach. 2006;28:313–317.
39Gruppen LD, Rana GK, Arndt TS. A controlled comparison study of the efficacy of training medical students in evidence-based medicine literature searching skills. Acad Med. 2005;80:940–944.
40Bradley P, Oterholt C, Herrin J, Nordheim L, Bjorndal A. Comparison of directed and self-directed learning in evidence-based medicine: A randomized controlled trial. Med Educ. 2005;39:1027–1035.
41Lloyd FJ, Reyna VF. A Web exercise in evidence-based medicine using cognitive theory. J Gen Intern Med. 2001;16:94–99.
42Sanchez-Mendiola M. Evidence-based teaching in the Mexican Army Medical School. Med Teach. 2004;26:661–663.
43Dorsch JL, Aiyer MK, Meyer LE. Impact of an evidence-based medicine curriculum on medical students' attitudes and skills. J Med Libr Assoc. 2004;92:397–406.
44Holloway R, Nesbit K, Bordley D, Noryes K. Teaching and evaluating first and second year medical students' practice of evidence based medicine. Med Educ. 2004;38:868–878.
45Badgett RG, Paukert JL, Levy L. The evolution of SUMsearch for teaching clinical informatics to third-year medical students. Acad Med. 2001;76:541.
46Wadland WC, Barry HC, Farquhar L, Holzman C, White A. Training medical students in evidence-based medicine: A community campus approach. Fam Med. 1999;31:703–708.