Review of previous curriculum
Once during the first year, each resident searched the literature and selected an article he or she felt best addressed one of their clinical questions. Each resident then critically appraised the article and presented the findings to the faculty and residents at his or her teaching site. This critical appraisal project was eliminated because it was time consuming, not relevant to daily practice, and offered little in gaining the skills to regularly answer clinical questions.
The librarian/informatics session, a yearly, hourlong, large-group didactic session including an assigned literature searching project, needed modification. The EBMC committee determined that, in this form, the session was minimally effective in teaching residents the skills necessary to formulate clinical questions and identify resources applicable to family medicine.
The journal club, a monthly, one-hour, small-group session during academic half-day (AHD), was unpopular. Articles were selected by faculty and provided to a second-year resident to review and present in a didactic format. Because topics did not stem from resident interest, there was little motivation for resident involvement.
The practice and quality-improvement project, a one-time, second-year project, required each resident to review clinic charts to determine whether practice patterns followed established practice guidelines for specific clinical concerns. The EBMC committee thought the practice and quality-improvement project offered good insight into practice patterns and factors influencing best practice.
On considering the curriculum overall, many residents felt they lacked adequate training in EBM, and a survey of resident opinion supported this sentiment.14 In addition, some informally expressed concern that some preceptors may lack evidence-based practice skills.
New Curriculum Components
The EBMC committee developed a number of suggestions that were based on discussion, identified goals and objectives, and literature-based evidence available about EBM practices. From the suggestions, four main curricular components (workshop, Family Medicine Desktop, brief evidence-based assessment of the research [BEAR], and journal club) were developed and implemented in 2005. We describe these in detail below. After the primary components were drafted, the EBMC committee developed a comprehensive evaluation strategy to monitor and research each component. Faculty members of the EBMC committee had multiple meetings to secure funding for the various components, arrange and develop the Family Medicine Desktop component (which required more initial time and resources than the other components), and seek input from stakeholders (particularly those whose educational activities were changing, such as the librarians). Once these aspects were finalized, the committee began the process of promoting change in the department.
Family medicine EBM workshop
Because studies have demonstrated improved knowledge with workshops and courses in EBM,8,9,15,16 the committee called for a formal EBM workshop to take place early in the residency. The two-day workshop for first-year residents is broken up into four half-days, each consisting of a one-hour lecture followed by a two-hour, small-group session. Three of the lectures are devoted to appraising articles (one each of therapy, diagnosis, and meta-analysis), and one is an overview of family practice research. All lectures are given by University of Alberta faculty, but the first three topics are taught from the principles of the Users’ Guides to the Medical Literature.17 In three small-group sessions, 8 to 10 residents, led by a faculty member and a second-year resident facilitator, appraise previously unseen papers on therapy, diagnosis, and meta-analysis. In the other small-group session, faculty librarians lead residents in the computer lab through question development, efficient Medline search techniques, and identification of summary resources (available on the Family Medicine Desktop, which is detailed below). Librarian faculty participation has proven to be an important part of EBM education.18,19
The workshop also includes a brief lecture on why EBM is important, a tutorial on basic statistical concepts, and a short, “game show”-type quiz with prizes during the last session. The statistical session includes working through control event rates, experimental event rates, absolute risk, number needed to treat, and likelihood ratios.
The yearly cost for this session is $6,000 (Canadian dollars), including facilities, catering (two breakfasts and two lunches), materials, and gifts. At the beginning of the workshop, before any teaching, residents complete (1) a survey to determine their self-assessed skills, past training, and objectives in EBM, and 2) a preworkshop quiz of 15 questions, derived in part from two previous EBM tests.9,20 At the conclusion of the conference, at the end of their last small-group session, residents complete (1) a postworkshop evaluation of the conference, and (2) a postworkshop quiz (same as the preworkshop quiz). Of note, the residents were not given answers for the preworkshop quiz, and small-group instructors were not aware of the content of the quiz. Analysis (in process) of the multiyear results will allow validation of the tool for assessing EBM knowledge.
Family Medicine Desktop
The Family Medicine Desktop is an Internet resource for each resident to reduce search time, identify information resources, and develop skills as EBM users. It uses a graphically displayed front page with links to relevant resources as well as tabs to pages of various themes. The front page (browser) includes rapid links to Cochrane, PubMed, Google, ACP Journal Club, Evidence-Based Medicine, Clinical Evidence, Bandolier, UpToDate, InfoPOEMS, Therapeutics Initiative, and BestBETs. There are tabs to Guides (Users’ Guides to the Medical Literature), Profession (links to the Web pages of differing professional bodies), Activities (residency program-specific features like BEAR presentations, journal club, and previous faculty presentations), Clinical (guidelines and local practice tools) and e-library (an extensive list of Internet medical resources). The Family Medicine Desktop is created and managed by the Canadian Centre for Health Evidence (CCHE), but it is amended and updated by members of the faculty. The Family Medicine Desktop can be accessed from any computer with Internet access. Because all residents have access to a desktop computer during their three- or four-month first-year family medicine rotation, the Family Medicine Desktop component is available to all residents at some point, regardless of whether they have access to the Internet outside work.
The yearly cost for the Family Medicine Desktop is $5,000 (Canadian dollars) for all residents and full-time faculty in the department. The site is monitored for frequency and duration of use by CCHE. Faculty committee members review these data annually. These data will be used to determine whether residents’ use of Family Medicine Desktop-based EBM resources changes throughout residency or during particular periods (such as immediately after the workshop or while completing BEARs).
The BEAR project helps residents identify daily clinical questions, focus their search for answers, get answers rapidly, and reflect on the utility of those answers. During their family medicine rotation in the first year, residents record questions arising from their clinical practice and complete one BEAR assignment monthly. The BEAR form (Appendix 1) guides residents through the exercise with a target time of less than one hour per question. Suggested resources include guidelines, filtered resources (e.g., EBM and ACP Journal Club), summary sites (e.g., Bandolier and UpToDate), review sites (e.g., Cochrane), and research studies (including RCT, Cohort, Qualitative). Faculty challenge residents to look at different questions (diagnostic, therapeutic, harm, practice management, etc.) and experiment with a variety of resources. Each BEAR is presented in approximately 15 minutes or less during rounds and is scored simply as pass for completion. To help generate discussion in rounds, the presentations include the case, the clinical question, resources examined, and practice reflection. Family medicine rotations in first year at the University of Alberta are generally 16 weeks (12 in the rural program), meaning four (or three in rural) BEARs per resident.
Integrating EBM exercises into clinical work improves residents’ adaptation to evidence-based practice.4,21,22 Furthermore, the use of preappraised and summarized evidence is required if clinicians are to achieve evidence-based practice6,7,23,24 and become “evidence users.”25 Lastly, the element of reflection, in both completing the BEAR form and during presentation with group discussion, enhances the likelihood of residents integrating EBM into practice and changing behavior for the long term.26
There is no yearly cost for the BEAR curricular component. The residents’ faculty advisor reviews each BEAR. Resident evaluation of the BEAR project is collected as part of the annual survey of the EBM curriculum. Full review of the BEAR projects is in process to examine the type of questions asked in clinical practice, resources used (summary sites, original research, guidelines, etc.), answers obtained, and practice impact. Activity records from the Family Medicine Desktop and direct observation of residents will be correlated with resource use reported in the BEARs to determine the reliability of these data. The results of this work will be reported in a subsequent publication.
Resident journal club (academic half-day)
Each month, the small-group AHD has an established theme (e.g., neurology). In preparation for the upcoming AHD, a second-year resident generates a case and clinical question related to the monthly topic and completes a literature search. He or she then reviews the abstracts of the resulting articles and selects two to three papers to best address the question, without critically appraising the papers. Whereas the preparation of the case and question should take no more than 30 minutes, the search and selection of two to three papers could take up to two hours.
At the AHD, the designated resident presents the case, question, and summarized search in approximately five minutes. Papers are then distributed, and the residents break into two to three subgroups, based on the number of papers to be reviewed. Each subgroup reviews one of the papers as a team. Each subgroup has 30 to 40 minutes to browse and appraise the article as a team to develop consensus. After that, each subgroup shares the information with the entire group. The total time is approximately 60 minutes. Peer group learning and teaching has been described in previous EBM education models.27,28
After the first year of the new curriculum, residents indicated that the articles chosen by colleagues were frequently low in quality and not relevant. As a result, articles are now selected from those abstracted or listed in the other articles noted section of ACP Journal Club or in the Evidence-Based Medicine journal. Whereas the article is picked from these sources, the original, complete article is brought to AHD for review.
There is no yearly cost for the journal club component. Resident evaluation of the journal club occurs at each AHD and as part of the annual survey of the EBM curriculum.
Other components of the curriculum include teaching on industry–physician interaction, presentation skills, and a quarterly review of the literature. For the quarterly review, a faculty member presents a five-minute EBM teaching point, a summary of five to seven family-medicine-relevant articles from the last year, and an introduction to an evidence-based practice tool (e.g., links to calculators for stroke risk in atrial fibrillation). In addition, the previously mentioned practice audit and quality-improvement project continues unchanged.
Although organization differed from past curriculum elements, the overall time use for faculty and residents was cost neutral, apart from two clinical days to attend the family medicine EBM workshop.
There is no yearly cost for the additional components. Residents complete evaluations after each teaching session. A year-end summative evaluation/survey obtains residents’ opinions regarding the four primary components and the curriculum overall. It also includes attitude questions regarding evidence-based practice in general. This information is used to assess the curriculum and to modify areas found to be inadequate.
A number of steps were taken to ensure that faculty, staff, and residents had the opportunity to learn and discuss concerns as well as generate feedback for the EBMC committee to improve the new curriculum. Urban and rural residents’ opinions were surveyed regarding EBM in general and the previous curriculum specifically.14 The draft proposal was distributed to all members of the residency program committee approximately one month before a planned review. Faculty, staff, and resident representatives of the residency program committee spent one hour reviewing and discussing the suggested changes. The planned changes were also presented to all urban residents at a program-wide AHD. The proposal was presented and discussed at the departmental geographic full-time faculty meeting.
After feedback from both residents and faculty, the EBMC committee decided to offer a two-day faculty development workshop in evidence-based practice for faculty members. The workshop was a small-group seminar with didactic, interactive, and computer learning time. To allow for maximum attendance, it ran on two separate occasions before the start of the new curriculum. The importance of faculty development and role modeling to promote resident learning in EBM is well known.29,30
One month before the start of the new curriculum, the final description was distributed to all staff and faculty. A faculty member from the EBMC committee visited each teaching site and met with faculty and staff to review any last questions. Finally, two months after the start of the curriculum, e-mail was distributed to each site director and site coordinating staff, and another follow-up visit was offered. Although a few brief concerns were reviewed over e-mail, none of the site directors or the site coordinating staff felt a follow-up visit was necessary. Initial conception to implementation took nine months.
The annual EBM curriculum evaluation has now been completed by two groups of residents (2005–2006 and 2006–2007) since implementation, and the results were compared with those from the year before implementation (2004–2005). The evaluation was distributed to residents attending an AHD in the last quarter of the academic year and was emailed once to all those reporting as absent from the AHD. The survey lacked any personal identifiable features. This research was approved by the health research ethics board of the University of Alberta.
The combined response rate for the three years was 60% (181 returned out of a possible 304). There was significant improvement in all areas of comfort with evidence-based practice and opinions regarding the curriculum (Table 1). Three of four questions of general opinion (e.g., The practice of EBM results in better patient care) showed improvement as well, albeit to a lesser degree.
In addition to the formal evaluation, resident and faculty feedback was sought at meetings and during presentations. Faculty have been strong supporters of the new curriculum and were particularly satisfied with the BEARs. Informal comments from faculty exemplified a view that the BEARs were more applicable to daily practice as compared with the previous critical appraisal projects. Residents reported satisfaction with the workshop and the didactic literature reviews of the AHD. Some residents struggled with the simplicity of the BEAR project and, at times, put much more effort than would be possible for questions in practice. They requested a tutorial on the BEAR, and this will be integrated into the EBM workshop. As mentioned earlier, although residents thought the journal club had improved, they felt that many papers chosen by their colleagues were poor in quality and relevance. The EBMC committee decided that residents would select journal club articles from those abstracted or noted in ACP Journal Club or the Evidence-Based Medicine journal. This change effectively addressed this concern for the residents.
Evaluations of other specific components (such as the workshop and the BEARs) are too large and complex to present here, but they will be published in subsequent articles.
This new integrated EBM curriculum is based on goals and objectives felt to be important to residents and faculty, and it incorporates the best available evidence in teaching EBM. Comparison of the goals and objectives with components of the curriculum reveals significant overlap. Surveys of other EBM curricula find that components such as faculty development, accessible electronic information resources, and evaluation mechanism are frequently missing,31 and some authors have provided suggestions for future EBM curricula.21,22 Our curriculum targeted these areas and more.
Time is perhaps the most significant barrier to evidence-based practice,7,32 and most clinicians will dedicate only a small amount of time (1–10 minutes) to answering clinical questions.6,33,34 Our curriculum focused on this issue. The Family Medicine Desktop introduced residents to the variety of EBM resources, with an emphasis on preappraised, summarized sites (i.e., BestBETs, ACP Journal Club) that will serve as the tools for their future as evidence users.6,23,25 By collecting these in one location and accessing any with a single click, residents can search different sites rapidly.
The workshop and journal club do not include prereading papers, because it is not realistic to read papers entirely and then spend another hour discussing them. Additionally, journal clubs often have trouble with some participants not reading articles.10 We focus on helping residents gain practical skills to review and appraise papers in a rapid, abbreviated fashion so that they may efficiently obtain answers to questions. Finally, the BEAR is the key feature to help residents become time-efficient evidence users because it promotes focusing their clinical questions, accessing information from summarized and preappraised resources, and reflecting on the level of influence the results have on daily practice.
Knowledge is often cited as a barrier to evidence-based practice,35,36 and, clearly, there is a place for learning the basic principles of EBM, even for evidence users.25 Interpreting the quality of preappraised, summarized evidence requires a basic understanding of EBM terms and statistics. Furthermore, occasionally preappraised sources will not have the required information.25 The workshop provides these foundations, and the journal club builds the knowledge and skills.
Access to resources is also frequently cited as a barrier to evidence-based practice.7,31,36 The Family Medicine Desktop provides easily accessible evidence-based resources, and the BEAR promotes the use of the resources and helps users gain familiarity with these preappraised, filtered, and summarized sites.
Evaluation is recommended as essential to EBM-teaching programs,31 and the EBMC committee felt it integral to each component and the curriculum in general. Evaluation allows faculty to adapt the curriculum according to changing needs or outcomes (such as Family Medicine Desktop use). Although rigorous research of EBM teaching is ideal,37 we needed to introduce a new curriculum within a limited time. Also, funding prohibited a trial with a comparator group, the examination of patient outcomes, and accounting for confounders. Previous authors used or encouraged testing knowledge, assessing attitudes/opinions, and evaluating learners around specific curriculum components.20,22 They provide excellent assessment of the curriculum while offering potentially useful research information. Additionally, we are monitoring Family Medicine Desktop use and directly observing residents in clinics (before and after the workshop) to determine the types of questions asked in clinical practice and the utility of evidence-based resources in answering their questions. We will compare these results with the questions and resources used in BEAR assignments.
In this article, we report a statistically significant improvement in resident attitude and self-reported abilities. Self-reported improvements after EBM educational intervention can correlate with improvement in skills and knowledge.38,39 Unfortunately, linking self-reported improvements after EBM educational interventions with persistent behavioral changes does not seem to have been studied. Therefore, although it might be reasonable to assume that residents’ skills and knowledge have improved with this curriculum, actual long-term behavioral changes remain uncertain.
In summary, this integrated EBM curriculum provides a foundation of knowledge, supplies easy access to resources, promotes point-of-care and team learning, and develops applicable skills for lifelong learning while achieving faculty goals of being evidence based, low cost, and time sensitive.
The authors would like to thank Olga Szafran with her assistance in the process of securing ethical approval for this project and the evaluation work, as well as Victoria Ma for her assistance in the compilation of the evaluation data. The authors would also like to thank Dr. Paul Humphries and the residency program committee for their support in moving this project forward.
1 Whitcomb ME. Why we must teach evidence-based medicine. Acad Med. 2005;80:1–2.
2 Green ML. Graduate medical education training in clinical epidemiology, critical appraisal, and evidence-based medicine: A critical review of curricula. Acad Med. 1999;74:686–694.
3 Norman GR, Shannon SI. Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: A critical appraisal. CMAJ. 1998;158:177–181.
4 Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004;329:1017–1019.
5 Korenstein D, Dunn A, McGinn T. Mixing it up: Integrating evidence-based medicine and patient care. Acad Med. 2002;77:741–742.
6 Ely JW, Osheroff JA, Ebell MH, et al. Analysis of questions asked by family doctors regarding patient care. BMJ. 1999;319:358–361.
7 McColl A, Smith H, White P, Field J. General practitioners’ perceptions of the route to evidence based medicine: A questionnaire survey. BMJ. 1998;316:361–365.
8 Schoenfeld P, Cruess D, Peterson W. Effect of an evidence-based medicine seminar on participants’ interpretations of clinical trials: A pilot study. Acad Med. 2000;75:1212–1214.
9 Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002;325:1338–1341.
10 Elnicki DM, Halperin AK, Shockcor WT, Aronoff SC. Multidisciplinary evidence-based medicine journal clubs: Curriculum design and participants’ reactions. Am J Med Sci. 1999;317:243–246.
11 Ebbert JO, Victor MM, Schultz HJ. The journal club in postgraduate medical education: A systematic review. Med Teach. 2001;23:455–461.
12 Ozuah PO, Orbe J, Sharif I. Ambulatory rounds: A venue for evidence-based medicine. Acad Med. 2002;77:740–741.
13 Crowley SD, Owens TA, Schardt CM, et al. A Web-based compendium of clinical questions and medical evidence to educate internal medicine residents. Acad Med. 2003;78:270–274.
14 Allan GM, Manca DP, Szafran O, Korownyk C. EBM a challenge for international medical graduates. Fam Med. 2007;39:612.
15 Ghali WA, Saitz R, Eskew AH, Gupta M, Quan H, Hershman WY. Successful teaching in evidence-based medicine. Med Educ. 2000;34:18–22.
16 Smith CA, Ganschow PS, Reilly BM, et al. Teaching residents evidence-based medicine skills: A controlled trial of effectiveness and assessment of durability. J Gen Intern Med. 2000;15:710–715.
17 Guyatt G, Rennie D, eds. Users’ Guides to the Medical Literature. Chicago, Ill: American Medical Association Press; 2002.
18 Finkel ML, Brown HA, Gerber LM, Supino PG. Teaching evidence-based medicine to medical students. Med Teach. 2003;25:202–204.
19 Dorsch JL, Jacobson S, Scherrer CS. Teaching EBM teachers: A team approach. Med Ref Serv Q. 2003;22:107–114.
20 Ross R, Verdieck A. Introducing an evidence-based medicine curriculum into a family practice residency—Is it effective? Acad Med. 2003;78:412–417.
21 Del Mar C, Glasziou P, Mayer D. Teaching evidence based medicine. BMJ. 2004;329:989–990.
22 Dobbie AE, Schneider FD, Anderson AD, Littlefield J. What evidence supports teaching evidence-based medicine? Acad Med. 2000;75:1184–1185.
23 Smith R. What clinical information do doctors need? BMJ. 1996;313:1062–1068.
24 Shaughnessy AF, Slawson DC. Are we providing doctors with the training and tools for life-long learning? BMJ. 1999;319:1280.
25 Guyatt GH, O Meade M, Jaeschke RZ, Cook DJ, Haynes RB. Practitioners of evidence based care. BMJ. 2000;320:954–955.
26 Robertson K. Reflection in professional practice and education. Aust Fam Phys. 2005;34:781–783.
27 Hunt DP, Haidet P, Coverdale JH, Richards B. The effect of using team learning in an evidence-based medicine course for medical students. Teach Learn Med. 2003;15:131–139.
28 Edwards K, Wolf PK, Hetlzer T. Pediatric residents as learners and teachers of evidence-based medicine. Acad Med. 2002;77:748.
29 Bhandari M, Montori V, Devereaux PJ, Dosanjh S, Sprague S, Guyatt GH. Challenges to the practice of evidence-based medicine during residents’ surgical training: A qualitative study using grounded theory. Acad Med. 2003;78:1183–1190.
30 Lam WW, Fielding R, Johnston JM, Tin KY, Leung GM. Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: A longitudinal focus group study. Med Educ. 2004;38:987–997.
31 Green ML. Evidence-based medicine training in internal medicine residency programs. J Gen Intern Med. 2000;15:129–133.
32 Toulkidis V, Donnelly NJ, Ward JE. Engaging Australian physicians in evidence-based medicine: A representative national survey. Intern Med J. 2005;35:9–17.
33 Ramos K, Linscheid R, Schafer S. Real-time information-seeking behavior of residency physicians. Fam Med. 2003;35:257–260.
34 Schilling LM, Steimer JF, Lundahl K, Anderson RJ. Residents’ patient-specific clinical questions: Opportunities for evidence-based learning. Acad Med. 2005;80:51–56.
35 McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-based medicine and the practicing clinician. J Gen Intern Med. 1999;14:236–242.
36 Al-Almaie SM, Al-Baghli N. Barriers facing physicians practicing evidence-based medicine in Saudi Arabia. J Cont Educ Health. 2004;24:163–170.
37 Hatala R, Guyatt G. Evaluating the teaching of evidence-based medicine. JAMA. 2002;288:1110–1112.
38 Green ML, Ellis PJ. Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997;12:742–750.
© 2008 Association of American Medical Colleges