According to the Accreditation Council for Graduate Medical Education (ACGME), graduate medical education must take place in “an environment of inquiry and scholarship in which residents participate in the development of new knowledge, learn to evaluate research findings, and develop habits of inquiry as a continuing professional responsibility.” More specifically, the Residency Review Committees (RRCs) of the ACGME mandate that house officers demonstrate evidence of scholarly activity such as original research, case reports, or reviews of clinical and research topics prior to the completion of their training.1 Inadequate demonstration of residents' scholarship is one of the most frequent reasons medicine residency programs are cited by the RRC.2
The potential benefits of residents' research are numerous. Residents' research may lead to better clinical care by fostering critical appraisal skills, clinical reasoning, and lifelong learning.3,4 Because publication during postgraduate training is positively correlated with the pursuit of academic careers,5 residents' research may also help reverse the trend of decreasing numbers of clinician investigators.6 Presentation and publication of sound research by house officers may increase the recognition of residency programs, enabling them to attract more desirable applicants.7,8 Finally, research training is an asset to those applying for jobs or fellowships.9
Although many authors have suggested methods for involving house officers in research, the recommendations consist primarily of overviews of curricular components and expert opinion.10,11,12,13,14 These suggestions may be insufficient given the logistic challenges residents face, such as limited time and funding, lack of expertise, and difficulty in finding mentors.15 A comprehensive curriculum that includes support for research, protected time, and opportunities for presentation is needed to address these obstacles.16 Although educators may want to incorporate such a curriculum, no single curriculum will work for all residency programs. Successful curricula should involve carefully performed needs assessments, well-defined learning objectives, educational strategies, and evaluation mechanisms developed or adapted with the goals and resources of individual training programs in mind.17
This article describes the results of a systematic review that educators might use for locating, modifying, and developing research curricula. Because articles about curriculum development often present scanty information,18 articles were examined for whether the authors reported essential elements of the curriculum development process such as needs assessments, goals or objectives, and evaluation mechanisms.19
For inclusion, the research curricula described must have been implemented in a residency program. Articles were excluded, however, if they described curricula (1) offered exclusively to faculty or medical students, (2) tailored to residents outside the typical residency period (e.g., the clinical investigator pathway, research sabbaticals), (3) offered to residents from nonclinical specialties (e.g., laboratory medicine), (4) designed to teach single components of the research process (e.g., critical appraisal, literature searches, evidence-based medicine, writing for publication), (5) designed for national samples of residents, or (6) in basic science.
Two investigators (RSH, RBL) independently performed the search and reached consensus on whether articles met the inclusion criteria. First, they searched Medline (1966—present) and National Library of Medicine Gateway (1958—present) databases for English-language articles using the medical subject headings (MeSH) “internship and residency” and “research/education” and subheading (SH) “education.” Of 966 articles, 31 met the inclusion criteria.20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50 Second, the same investigators searched the Educational Resources Information Center (ERIC, 1966—present) database for “graduate medical education” and “student research” or “graduate medical education” and “research skills” or “graduate medical education” and “curricula or curriculum.” Of 181 articles, two usable articles were identified.51,52 Third, 11 medical education Web sites were searched,53,54,55,56,57,58,59,60,61,62,63 and two articles describing curricula (two articles by Summers referring to a single curriculum) were retrieved.64,65,66 Fourth, bibliographies of all included articles were reviewed, capturing four additional papers.67,68,69,70 Fifth, papers meeting inclusion criteria were entered into the ISI Science Citation Index cited reference search, and two articles were retrieved.71,72 Finally, we contacted 15 authors of published resident research curricula, but found no further articles.
The same two investigators reached consensus about whether articles described (1) needs assessments (literature review of existing curricula, surveys, focus groups, interviews, or other formal assessments of learners/teachers), (2) goals and objectives, (3) educational strategies, (4) evaluation methods (anecdotal report, learner self-assessment, or testing), and (5) outcomes evaluation (learners' satisfaction and curricular effectiveness). An evaluation of curricular effectiveness was required to have a description of a curriculum's impact on learners' knowledge, skills, and attitudes towards research, or a report of residents' presentations and publications. As described by Green, articles were reviewed for descriptions of process: a report of sustainability (the length of time the curriculum had been in place and the total number of learners that had participated) and feasibility (obstacles encountered and modifications made in the curriculum).73,74 Protected time, including elective time, was defined as dedicated time outside research seminars or didactics during which residents' primary responsibility was working on research projects.
The search identified 41 articles describing curricula: nine from family medicine, seven from psychiatry, six from emergency medicine, three from otolaryngology, two each from community medicine, internal medicine, obstetrics and gynecology, pediatrics/medicine—pediatrics, and one each from child psychiatry, dermatology, general surgery, and radiology. Three curricula were developed for residents from multiple specialties, and one did not specify a specialty (see Table 1). Four described curricula implemented outside the United States.30,44,50,71
Many articles failed to report steps essential to curriculum development. Twenty-seven articles (66%) reported curricular objectives (see Table 2). The most common objectives were to increase house officers' research productivity and improve their critical appraisal skills. Two residency programs developed curricula to meet the ACGME scholarly activity requirements. Only one curriculum was designed with the goal of producing academic physicians.64 Eleven articles (27%) documented needs assessments. The methods employed included holding planning sessions attended by educators and researchers,31,32,35,49 literature reviews,37,52,66,68 and surveying residents about their interests and skills.34,43,44
Most curricula used multiple educational strategies (see Table 1). Common methods used were conducting research projects, exposing learners to research mentors or role models, and providing house officers with multiple opportunities to present their work, often at seminars or “research in progress” sessions. In many programs, residents were expected to present their final work at departmental (e.g., research day, grand rounds), regional, or national forums,21,24,26,27,31,32,33,34,36,39,40,41,42,43,47,48,50,51,52,67,69, 70,72 or submit a manuscript for publication.29,39,40,41,43,45,69,72
Whereas most research curricula were integrated as longitudinal components of the residency curriculum, a few were scheduled in dedicated blocks of time (range: two days to four months).23,24,27,35,41,49,50 Twenty-two programs had a required curriculum20,21,24,26,27,29,32,34,35,36,39,41,42,43,47,49,50,51,66,67,70,72 and 16 offered protected time to work on research projects21,22,24,26,27,29,31,32,36,43,49,50,51,52,68,69 (range: one to nine months).
Many unique educational strategies were used to teach house officers. In four programs, the research curriculum was mandatory but a research project was not.26,34,35,66 In contrast, Lambert describes an informal, mentor-based research program without lectures or seminars for only those residents interested in research. Learners were paired with mentors who guided the house officers in all phases of a research project.68 The Internal Medicine Clinic Research Consortium Faculty reported that 66 second- and third-year residents worked together on a single project.40 At another program, upper-level residents acted as principal investigators on projects. Portions of a research project were assigned to junior residents at the discretion of the principal investigator.46 Other distinct strategies included offering academic coursework,20 master's degree credit,39 and requiring residents to do multiple research projects.36 Neale described a consortium composed of community- and university-affiliated hospitals as well as a medical school whose goal was to assemble educational research programs with broader depth than might be possible at individual sites.48
Evaluation Methods and Outcomes
The evaluation of learners was often sparse, frequently limited to authors' anecdotal reports or learners' self-assessments of their knowledge, skills, or satisfaction (learners were generally described as having increased their knowledge or skills and being satisfied with the research experience). Only five articles (12%) described objective pre- and post-intervention testing of knowledge.23,35,41,52,66 All post-intervention testing was done immediately after the curriculum; none of the reports included long-term follow up. No curriculum was evaluated as a prospective pretest—posttest controlled trial.
Twenty programs described the success of their house officers in having research accepted for presentation at regional or national meetings and in publishing their research.21,24,25,28,31,32,33,35,36, 40,42,43,46,48,49,51,52,67,68,72 Nine articles compared house officers who had participated in the curriculum with those who had not. Five reported increases in their quantity and/or quality of residents' research.32,35,37,68,71 Gaspar and Ely and Fraker et al. reported that, prior to implementing their curricula, zero and 19% of their house officers, respectively, had participated in research. After implementation of the curricula, the percentages increased to 75% and 100%.31,41 Kirchner et al. described an increase in residents' presentations at national conferences.72 Rydman et al. reported that within four months of completing the program, 76% of house officers had submitted National Institutes of Health grant proposals, versus 23% the preceding year.39 Five articles stated that faculty research productivity had increased as a result of the residents' research curricula.28,31,32,35,41
Sustainability and Feasibility
In the articles, curricula had been in place a median of five years (range one to 18 years) and had attracted relatively few participants since implementation (median 20, range four to 800; see Table 1). The articles revealed that faculty effort needed to increase residents' research may be substantial. The seven faculty members of the Internal Medicine Clinic Research Consortium described an “intense time commitment.”40 Lambert and Garver state that mentors must “extensively” involve themselves in residents' projects.68 More specifically, Alguire et al. delineated the time commitment of a research director as “10–20% of a full-time position.”52 Costs were described as minimal or modest.32,47,71 Wilson and Redman reported that each resident was given $200 to complete his or her project.70 Alguire and Neale stated that their programs had annual budgets of $3,000 and $7,000, respectively.48,52 In articles where this information was available, the numbers of faculty involved ranged from two to seven.37,39,40,66
Rare mention was made of the barriers encountered in implementing curricula. Obstacles were often described in general terms (see Table 3), and the few curricular modifications reported were nonspecific. General statements described modifications or periodic revisions of curricula,29,48,52,70 such as “the course is modified based on feedback from students and instructors,”23 and “the program was piloted and improved.”50 Clayton and Sheldon-Keller were more specific, stating that residents' dissatisfaction with the structure of the research project had led to changes in the research requirement.66
The published curricula represent a diverse group of educational interventions designed largely to increase house officers' scholarly productivity, develop critical appraisal skills, and give residents an appreciation for research. Common educational strategies include lectures and seminars, having residents perform a research project, learning from role models and mentors, and providing house officers with multiple opportunities to present their work. Information about the curriculum development process, evaluation methods, and outcomes was often limited.
Completing a research project during an already busy experience is challenging. Factors consistently described in the literature as being necessary for successful research training include (1) exposure to and guidance from mentors, (2) training in basic research methods, (3) protected time, and (4) an environment supportive of research.75,76,77 We found many of these elements incorporated in the reviewed curricula.
Despite this, most curricula lacked key information for critical review by educators. Many reports provided incomplete descriptions of program and learner characteristics, instructional strategies, sustainability, and feasibility. Needs assessments, articulation of clearly defined learning objectives, and evaluations were also frequently missing. These are important to perform and document for many reasons. A curriculum developed without a needs assessment risks inefficiency or ineffectiveness. Unnecessary resources may be devoted to areas learners have already mastered and insufficient resources devoted to areas of particular need.17 Measurable objectives further refine curricular content and help guide the selection of educational and evaluation methods.78 Evaluations provide information useful for improving the performances of individuals and programs.79. In addition, evaluation results can be used to maintain support for a curriculum, document the accomplishments of learners and developers, and serve as a basis for garnering funding. To the extent that a curriculum addresses an important need or introduces novel educational strategies, evaluation results may be of interest to educators from other institutions.17 Finally, reports lacking curricular elements can hinder educators interested in adopting curricula.80
Several limitations of our study should be considered. First, we limited our search to published curricula. Clinician educators may not rely on a national reputation for promotion and may be less inclined to publish their curricula. Other curricula likely exist but are not readily available to educators interested in modifying them for their own needs. Second, we may have missed some published curricula. To minimize this, two investigators independently performed a literature search using a strategy developed with the aid of a medical librarian.
Residents' research curricula teach house officers how to articulate clinical questions, appraise the literature, apply research skills, and work with mentors. As such, these educational interventions have great potential to impact learners and increase scholarly activity. Curriculum developers have an obligation to use established principles and report reliable and meaningful outcome evaluations so learners and educators can maximally benefit from their efforts.
1. Residency Review Committees. 〈http://www.acgme.org
〉. Accessed 3/15/02. Accreditation Council for Graduate Medical Education, Chicago, IL, 2002.
2. Beasley BW, Scrase DR, Schultz HJ. Determining the predictors of internal medicine residency accreditation: what they do (not what they say). Acad Med. 2002;77:238–46.
3. Abramson M. Improving resident education: what does resident research really have to offer? Trans Am Acad Ophthalmol Otolaryngol. 1977;84:984–5.
4. Does research make for better doctors? Lancet. 1993;342:1063–4.
5. Hillman BJ, Fajardo LL, Witzke DB, Cardenas D, Irion M, Fulginiti JV. Factors influencing radiologists to choose research careers. Invest Radiol. 1989;24:842–8.
6. Rosenberg LE. Young physician—scientists: internal medicine's challenge. Ann Intern Med. 2000;133:831–2.
7. Heinrich G, Nori D, Tome J, Parikh S. Developing a research program in a community teaching hospital. Teach Learn Med. 1999;11:89–93.
8. Schultz HJ. Research during internal medicine residency training: meeting the challenge of the residency review committee. Ann Intern Med. 1996;124:340–2.
9. Souba WW, Tanabe KK, Gadd MA, Smith BL, Bushman MS. Attitudes and opinions toward surgical research. A survey of surgical residents and their chairpersons. Ann Surg. 1996;223:377–83.
10. Collins J, Meyer R, Dawson JR. Getting residents involved in research: a challenge in the era of managed care. Acad Med. 1999;74:1155–7.
11. Karlik SJ. How to develop and critique a research protocol. Am J Roentgenol. 2001;176:1375–80.
12. Carlson G, Irving M. How to get the most out of a period of research. Br J Hosp Med. 1996;56:275–7.
13. Boninger ML, Chan L, Harvey R, et al. Resident research education in physical medicine and rehabilitation: a practical approach. Am J Phys Med Rehabil. 2001;80:706–12.
14. Pollack CV Jr, Wadbrook PS. Residents' perspective. Ann Emerg Med. 1999;33:117–20.
15. Alguire PC, Anderson WA, Albrecht RR, Poland GA. Resident research in internal medicine training programs. Ann Intern Med. 1996;124:321–8.
16. DeHaven MJ, Wilson GR, O'Connor-Kettlestrings P. Creating a research culture: what we can learn from residencies that are successful in research. Fam Med. 1998;30:501–7.
17. Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education. A Six-Step Approach. Baltimore, MD: Johns Hopkins University Press, 1998.
18. Sheets KJ, Anderson WA. The reporting of curriculum development activities in the health professions. Teach Learn Med. 1991;3:221–6.
19. Reznich CB, Anderson WA. A suggested outline for writing curriculum development journal articles: the ICDRD format. Teach Learn Med. 2001;13:4–8.
20. Gottschalk LA. Training in psychosomatic research. Adv Psychosom Med. 1967;5:25–45.
21. Baker SL. On the importance of a research experience during the clinical residency. Med Ann District of Columbia. 1968;37:273–4.
22. Portelli FR. Status of resident research in military programs in otolaryngology (ENT). Trans Am Acad Ophthalmol Otolaryngol. 1970;74:1093–8.
23. Conrad ME. An experiment in postgraduate medical education: an orientation program for residents in clinical research. Mil Med. 1974;139:30–2.
24. Tucker HM. Resident research program at upstate medical center. In: Alberti PW, Bryce DP., (eds). Centennial Conference on Laryngeal Cancer. New York: Appleton—Century—Crofts, 1976:819–26.
25. Anwar RA, Wagner DK. The research component in the development of emergency medicine as a specialty. J Med Educ. 1977;52:55–8.
26. Demers RY. Integrating community based research into residency training. J Fam Pract. 1981;12:675–9.
27. Cummings CW. Research training for residents. Arch Otolaryngol. 1982;108:630–2.
28. Brautigan MW. A systematic approach to research curricula for emergency medicine residencies. J Emerg Med. 1984;1:459–64.
29. Holloway RL, Altemeier TM. A comprehensive research program for family medicine residents. Fam Med. 1986;18:133–5.
30. Ben Tovim DI, Battersby MW, Allan JA, Papay PS. A group method for learning about psychiatric research. Aust NZ J Psychiatry. 1987;21:392–5.
31. Gaspar MJ, Ely TL. Research in a busy family practice training program. Fam Med. 1987;19:463–5.
32. Jones J, Dougherty J, Cannon L, Schelble D. Teaching research in the emergency medicine residency curriculum. Ann Emerg Med. 1987;16:347–53.
33. Liese BS, Johnson CA, Govaker DA, O'Dell ML. Increasing research productivity in a university-based residency program: a case study. Kans Med. 1988;89:143–7.
34. Konen JC, Fromm BS. A family practice residency curriculum in critical appraisal of the medical literature. Fam Med. 1990;22:284–7.
35. Breitenbach RA, Zanecchia MD. Enhancing military primary care research. Mil Med. 1992;157:364–9.
36. Sulak PJ, Croop JA, Hillis A, Kuehl TJ. Resident research in obstetrics and gynecology: development of a program with comparison to a national survey of residency programs. Am J Obstet Gynecol. 1992;167:498–502.
37. Paniagua FA, Puariega AJ, O'Boyle M, Meyer WJ. The role of a research seminar for child psychiatry residents. J Am Acad Child Adolesc Psychiatry. 1993;32:446–52.
38. DeHaven MJ, Wilson GR, Murphree DD. Developing a research program in a community-based department of family medicine: one department's experience. Fam Med. 1994;26:303–8.
39. Rydman RJ, Zalenski RJ, Fagan JK. An evaluation of research training in a large residency program. Acad Emerg Med. 1994;1:448–53.
40. Internal Medicine Clinic Research Consortium Faculty. Housestaff team research in the ambulatory setting: it can be done. J Gen Intern Med. 1995;10:219–22.
41. Fraker LD, Orsay EM, Sloan EP, Bunney EB, Holden JA, Hart RG. A novel curriculum for teaching research methodology. J Emerg Med. 1996;14:503–8.
42. Brouhard BH, Doyle W, Aceves J, McHugh MJ. Research in pediatric residency programs. Pediatrics. 1996;97:71–3.
43. Kirsner RS, Kerdel FA, Falanga V, Trent J, Eaglstein WH. The role of mandated research during dermatology residency training. J Invest Dermatol. 1999;112:400–1.
44. Bansal RK. Research stimulating programme for interns. Indian J Med Sci. 1996;50:185–9.
45. Crooks KA, Pato MT, Berger RC. A curriculum for producing resident researchers. Acad Med. 1999;74:614.
46. Chung R, Diaz J, Li P. A method of teaching clinical research in a community hospital residency program. Am J Surg. 1999;177:83–5.
47. Miller JL, Smith M. Research training for residents. Fam Med. 2000;32:305–6.
48. Neale AV, Pieper D, Hammel E. A consortium-based research education program for residents. Acad Med. 2000;75:298–301.
49. Gay SB, Hillman BJ. Evaluation of a mandatory radiology resident research rotation. Acad Radiol. 2000;7:172–5.
50. Chaturvedi S, Aggarwal OP. Training interns in population-based research: learners' feedback from 13 consecutive batches from a medical school in India. Med Educ. 2001;35:585–9.
51. Winter RJ, Unti SM, Collins JW. Research during residency training. Teach Learn Med. 1993;5:96–101.
52. Alguire PC, Anderson WA, Henry RC. Teaching research skills: development and evaluation of a new research program for residents. Teach Learn Med. 1993;5:37–43.
53. Association of Program Directors in Internal Medicine. 〈http://www.apdim.med.edu
〉. Accessed 2/15/02. Association of Program Directors of Internal Medicine, Washington, DC, 2001.
54. American College of Physicians—American Society of Internal Medicine. 〈http://www.acponline.org
〉. Accessed 2/15/02. American College of Physicians—American Society of Internal Medicine, Philadelphia, PA, 1996–2002.
55. Society of General Internal Medicine. 〈http://www.sgim.org
〉. Accessed 2/15/02. Society of General Internal Medicine, Washington, DC, 2001.
56. Association of American Medical Colleges. www.aamc.org
. Accessed 2/15/02. Association of American Medical Colleges, Washington, DC, 2002.
57. Center for Instructional Support. 〈http://www.uchsc.edu/CIS
〉. Accessed 2/15/02. Center for Instructional Support, Denver, CO, 1996.
58. Association for Medical Education in Europe. 〈http://www.amee.org
〉. Accessed 2/15/02. Association for Medical Education in Europe, Dundee, UK, 2002.
59. Association for the Study of Medical Education. 〈http://www.asme.org.uk
〉. Accessed 2/15/02. Association for the Study of Medical Education, Edinburgh, UK, 2002.
60. The Network: Community Partnerships for Health Through Innovative Education, Service and Research. 〈http://www.network.unimaas.nl/home.htm
〉. Accessed 2/15/02. The Network: Community Partnerships for Health Through Innovative Education, Service and Research, Maastricht, NL, 2002.
61. TIMELit: Topics in Medical Education. 〈http://www.timelit.org
〉. Accessed 2/15/02. Centre for Medical Education, Dundee, UK, 2001.
62. BEME Collaboration: Best Evidence Medical Education. 〈http://www.bemecollaboration.org
〉. Accessed 2/15/02. Scottish Council for Postgraduate Medical and Dental Education, Dundee, UK, 2002.
63. Cochrane Effective Practice and Organization of Care Group. 〈http://www.abdn.ac.uk/hsru
〉. Accessed 9/27/02. Health Services Research Unit, University of Aberdeen, UK, 2002.
64. Summers RL, Woodward LH, Sanders DY, Galli RL. Research curriculum for residents based on the structure of the scientific method. Med Teach. 1998;20:35–7.
65. Summers RL, Woodward LH. Learning the process of research by using the scientific method. Med Teach. 1999;21:591–3.
66. Clayton AH, Sheldon-Keller AE. The design and evaluation of a group research experience during psychiatric residency training. Acad Psychiatry. 2001;25:68–76.
67. Thompson RJ, Benrubi GI, Kaunitz AM. Resident research in obstetrics and gynecology. J Reprod Med. 1994;39:635–8.
68. Lambert MT, Garver DL. Mentoring psychiatric trainees' first paper for publication. Acad Psychiatry. 1998;22:47–55.
69. Coleridge ST. Teaching residents to write a research paper. J Am Osteopath Assoc. 1993;93:936–40.
70. Wilson JL, Redman RW. A program for teaching research in a family practice residency. J Fam Pract. 1980;10:729–30.
71. Raphael B, Dunne M, Byrne G. A research seminar programme for doctoral candidates in psychiatry. Aust NZ J Psychiatry. 1990;24:207–13.
72. Kirchner JE, Owen RR, Nordquist CR, Clardy JA. Developing clinician scientists through integrated research training in psychiatry. Teach Learn Med. 1998;10:183–7.
73. Green ML. Graduate medical education training in clinical epidemiology, critical appraisal, and evidence-based medicine: a critical review of curricula. Acad Med. 1999;74:686–94.
74. Green ML. Identifying, appraising, and implementing medical education curricula: a guide for medical educators. Ann Intern Med. 2001;135:889–96.
75. Bland CJ, Schmitz CC. Characteristics of the successful researcher and implications for faculty development. J Med Educ. 1986;61:22–31.
76. Bland CJ, Ruffin MT. Characteristics of a productive research environment: literature review. Acad Med. 1992;67:385–97.
77. Temte JL, Hunter PH, Beasley JW. Factors associated with research interest and activity during family practice residency. Fam Med. 1994;26:93–7.
78. Harden RM. Ten questions to ask when planning a course or curriculum. Med Educ. 1986;20:356–65.
79. Wilkes M., Bligh J. Evaluating educational interventions. BMJ. 1999;318:1269–72.
80. Ende J, Atkins E. Conceptualizing curriculum for graduate medical education. Acad Med. 1992;67:528–34.