Secondary Logo

Journal Logo

Reviews

Increasing Scholarly Activity Productivity During Residency: A Systematic Review

Stevenson, Michelle D. MD, MS; Smigielski, Elizabeth M. MA, MSLS; Naifeh, Monique M. MD, MPH; Abramson, Erika L. MD, MSc; Todd, Christopher MD, MS; Li, Su-Ting T. MD, MPH

Author Information
doi: 10.1097/ACM.0000000000001169

Abstract

In the United States, there is growing concern over the shortage of physician researchers.1,2 Participation in research during residency may help address this shortage by increasing interest in a research career. At the individual level, resident scholarly activity productivity (RSAP) is associated with choosing an academic career.3–6 At the programmatic level, residency programs with higher levels of RSAP yield residents who are more likely to pursue academic careers.7 Even for those residents with no interest in a research career, participation in research is valuable in that it may improve the patient care they provide by increasing their comfort with evidence-based medicine8 and enhancing their analytical and critical thinking skills.2,9

The Accreditation Council for Graduate Medical Education (ACGME) requires that residency programs provide curricula that advance residents’ knowledge of the basic principles of research, ensure resident participation in scholarly activity, and allocate resources to facilitate this participation.10 The ACGME has recently added required reporting of resident scholarly activity, including publications and regional, national, or international presentations.11 Although steadfast in its requirement for scholarship in residency, the ACGME has never given specific recommendations about how this should be achieved. As a result, individual programs have developed diverse curricula to meet the requirement, resulting in variable productivity and satisfaction.12–20 However, multiple barriers to successful resident scholarly activity have been described, including lack of time, mentorship, and resources,20 along with variability in resident attitudes toward scholarly activity.21

A better understanding of interventions that promote RSAP may help program directors meet ACGME scholarly project requirements and foster the development of a productive workforce in academic medicine. Few studies have attempted to determine which program qualities lead to increased scholarly productivity.20,22,23 A prior systematic review published in 2003 focused on describing resident research curricula but did not address RSAP.9 In this systematic review, we seek to answer the following question: Among residency training programs in the United States and Canada, what interventions at the programmatic level are effective in increasing RSAP, as measured by participation in scholarly activity, publications, or presentations? Knowledge gained through this research can provide program directors across specialties with best practices to increase RSAP within their own programs.

Method

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines,24 we systematically reviewed articles that described outcomes of interventions designed to increase RSAP. A review protocol is available from the authors upon request.

Search strategy

A librarian (E.M.S.) conducted all literature searches in consultation with the research team. Electronic searches of the PubMed and MEDLINE (peer reviewed), Cochrane Library of Systematic Reviews, CINAHL, PsycINFO, and ERIC databases were conducted from database inception through October 9–10, 2013. Search terms included the domains of residency, graduate medical education, research, research training, and scholarly activity. Keyword and medical subject headings (MeSH) search strategies were used for PubMed. The MeSH search strategy included the terms “Education, Medical, Graduate,” “Internship and Residency,” “Medicine/education,” and “Research.” A keyword search was conducted in the Cochrane Library database. The remaining databases were searched using controlled vocabularies. For full search strategies, see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A338.

We performed a hand search of the bibliographies of included articles. We did not search conference proceedings.

Study selection and definitions

We limited articles to studies on residency programs in the United States or Canada (because of similarities in training requirements) and published in English. We included randomized control studies, case–control studies, cohort studies, survey studies, and analyses of programs that evaluated factors or interventions to increase RSAP in any medical specialty. No date limits were applied. Exclusion criteria were lack of relevant outcome, lack of control group, editorials and commentaries, and articles focused exclusively on fellowships or combined MD/PhD programs.

Our primary outcome was RSAP, defined as research project participation, presentations, or publications by residents. Our secondary outcomes included attainment of grant funding (because reporting of grant funding is not required in the Next Accreditation System11) and fellowship or faculty positions.

Data extraction

Two standardized data collection forms were developed, pilot tested, and approved by our research group. The first form was used to select potentially applicable articles based on pertinent titles and abstracts. It was piloted and finalized by two reviewers (S.T.L. and M.D.S.). The second form served as our main data extraction tool, and it documented reasons for article inclusion or exclusion after full-text review. All reviewers (S.T.L., M.D.S., C.T., E.A., M.N.) pilot tested the data collection form on two articles each (n = 10). The form was then revised and used to collect data on the articles selected for full-text review.

After the initial search, one reviewer (E.M.S.) screened all records to exclude editorials and commentaries. Two reviewers (S.T.L. and M.D.S.) screened titles and abstracts of all remaining records to identify articles that appeared relevant for full-text review and randomly distributed them equally among all reviewers (S.T.L., M.D.S., C.T., E.A., M.N.). Two reviewers independently reviewed each article, extracted data, and determined whether the article met study inclusion criteria. We resolved discordance by discussion and consensus.

Data elements recorded for fully reviewed articles included citation information, study design, program specialty, study population (residents, program directors, or research directors), sample size, intervention or type of observation, and outcomes. Risk of bias was assessed among the articles that reported statistics using the Medical Education Research Study Quality Instrument (MERSQI), a validated scoring tool for medical education research.25 A maximum possible score of 15 (indicating high quality) on the MERSQI was used because one score component (regarding validity of an evaluation instrument) was not applicable to the articles in our review.

Analysis

Because of expected heterogeneity in the reporting of primary outcomes of RSAP, we did not plan or perform a meta-analysis. For articles that reported statistics, we categorized results by type of RSAP outcome, and we summarized the results separately for surgical and medical specialties because of differences in training program structures. Because we were concerned that our results might have been influenced by self-selection of residents interested in an academic career into training programs with better infrastructure for scholarly activity, we evaluated studies that controlled for potential confounding variables (e.g., resident gender, prior research experience, or level of training) separately.26 We discussed and, by group consensus, summarized overall trends in associations between interventions and RSAP outcomes.

Results

Characteristics of all included studies

We identified and screened the titles of 6,248 records after removal of duplicates (see Figure 1). Of the 848 abstracts screened for inclusion, 192 were chosen for full-text review. Of the 80 articles that underwent data abstraction, 26 studies13,16,27–50 described outcomes without a comparison group (for details of these 26 studies, see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/A339). Fifty-four studies4,8,12,14,15,17–19,22,23,51–94 describing outcomes with comparison groups were included in the final analysis.

Figure 1
Figure 1:
Study selection and review process for systematic review on interventions to increase resident scholarly activity productivity.

Thirteen specialties were represented: 9 medical specialties (69%) and 4 surgical specialties (31%). Associations between studied interventions and RSAP outcomes reported in the 54 included studies are summarized in Appendix 1, categorized by medical (n = 38; 70%) or surgical (n = 16; 30%) specialty. The 54 studies evaluated outcomes using a historical cohort design (n = 36; 67%), case–control design (n = 2; 4%), or cross-sectional survey of residents or program/research directors (n = 16; 30%). Interventions included a requirement for participation in scholarly activity, protected research time (of varying duration), a research curriculum, a dedicated research day, a research track, and a research director. Among medical specialty studies, 15 of the articles17–19,23,51,53,56,60,64,65,71,75,79,92,93 reported trends toward improvement in outcomes with the studied interventions but did not use or report statistical analysis; the surgical specialty literature included 4 such articles.66,72,76,83 To focus on literature of higher quality, in the remainder of our review, we summarize results from the 35 articles that reported statistical analysis.

Characteristics of articles reporting statistical analysis

Table 1 summarizes the effect of studied interventions on RSAP outcomes, restricted to the 35 articles that reported statistical analysis. Appendixes 2 and 3 report detailed findings of these articles for medical specialties (n = 23; 66%) and surgical specialties (n = 12; 34%), respectively. Fifteen of these 35 publications involved data from multiple programs, obtained either through surveys of residents or program/research directors (n = 14) or, in 1 article,90 through searching publicly available information. The sample size for the 14 studies with a survey design ranged from 60 to 1,474 participants, with a median response rate of 63% (range 24%–92%).22,52,55,57,59,67,68,70,74,85–87,89,94 The sample size for the 2 case–control studies14,88 and the 19 cohort studies4,8,12,15,54,58,61–63,69,73,77,78,80–82,84,90,91 varied from 24 to 626 residents. Study quality scores, using the modified MERSQI, ranged from 8 to 13 across the 35 articles.

Table 1
Table 1:
Summary of Effect of Interventions on Resident Scholarly Activity Productivity Outcomes as Reported in the 35 Studies With Comparison Groups and Reporting Statistical Analysis

Interventions associated with participation

All 5 publications (4 medical and 1 surgical) that evaluated resident participation in scholarly activity as an outcome reported increased participation as a result of the intervention(s) studied. Multiple studies bundled interventions, including protected research time, a research day, research curricula, a research track, and/or a research requirement.8,22,58,77,85

Interventions associated with presentations

Results of the 10 studies8,15,57,61,69,80,82,87,89,94 addressing presentation of scholarly work by residents were mixed in both medical (5/8) and surgical (1/2) specialties. In the medical specialties, a research director (3/3)* resulted in increased regional and national presentations.61,69,94 In 2 of these studies,69,94 a research director was combined with a research curriculum and protected time. Of the remaining 5 medical specialty studies without a research director (but using a combination of protected time, research curriculum, research track, and/or research requirement), only 2 showed a positive effect on presentations.82,89 In the surgical specialties, 1 study found a positive association between a research requirement and presentations.80 A multicenter study of orthopedic residents found no association between presentations and programs with a research director, research curriculum, and/or protected time.57

Interventions associated with publications

Most studies in both the medical (n = 17) and surgical (n = 8) specialties studied the effect of multiple interventions on publications.

In the medical specialties, a research track (4/4)67,70,82,86 and a research director (3/4)61,69,88 increased publications. Two studies combined a research director with a research curriculum and protected time.69,88 The majority of studies found that providing protected research time (7/12),22,67,69,78,82,86,88 a research curriculum (9/11),14,22,52,69,73,78,82,86,88 and/or a research day (5/6)12,22,73,78,82 was associated with increased publications. Protected time most often ranged from 1 to 3 months8,15,52,69,82,87,88; however, 1 program allowed up to 6 months,78 and in 2 multicenter studies the majority of psychiatry residents86 or radiation oncology residents67 had ≥ 6 months of dedicated research time. In the medical specialties, an increase in publications associated with protected time was always found in association with another intervention: a research curriculum (n = 6), a research day (n = 3), a research track (n = 3), a research director (n = 2), and/or faculty support (n = 1). Only 3 studies found a positive effect on publications from a single intervention: a research day,12 a research track,70 or a research director.61

In the surgical specialties, protected research time (6/7)4,57,59,68,90,91—either as a single intervention (3/4)57,68,90 or in combination with a research requirement (1/1)59 or track (2/2)4,91—was the most common intervention associated with increased publications. Three articles reported an average of 1 year of protected research time,4,59,91 while others reported mean protected research time of 4 to 5 months,57,68 and 1 multicenter study reported a range of 0 to 12 months.90 The duration of protected research time was significantly associated with the quantity of resident publications in the majority of these studies.57,59,68,90 A research director did not improve publication rates.57

Interventions associated with grants, fellowship positions, and faculty positions

Grants.

Acquisition of research grants was the least commonly studied outcome, examined in just 3 medical and 2 surgical studies. In 1 multicenter study from radiology,70 participation in an introduction to research program was associated with attainment of intramural research funding. Similarly, residents from multiple radiation oncology programs who participated in a specialized research track were more likely than nonparticipants to obtain peer-reviewed grants based on work initiated during residency.67 In contrast, a small pediatric training program did not find a significant increase in grant funding after implementation of a research curriculum, protected research time, and a research day.15

In the surgical specialties, 1 multicenter study found that orthopedic residents who had at least one month of protected time for research were more likely to obtain grant funding during training than those with less time. Having a research director or a research curriculum did not have an impact on grant funding.57 Another study from a single general surgery training program found that the number of publications as a resident was associated with faculty grant support.81

Fellowship and faculty positions.

One study (1/4) in the medical specialties showed a positive association between an intervention and either fellowship or academic faculty positions: A large multi-institutional study of radiology residents demonstrated that participants in a research track were more likely than nonparticipants to pursue faculty positions.70 In 5 studies (5/6) in the surgical specialties, protected research time during residency was associated with obtaining a fellowship and/or academic faculty position after completion of training.54,59,63,68,90

Studies that addressed potential confounding variables

Eight articles employed a matched case–control design,14 multivariate logistic regression analysis,12,22,55,67,74,94 or Poisson regression analysis68 to account for potential confounding variables. Only 1 of these articles was in a surgical specialty.68

Three articles evaluated the outcome of participation in scholarly activity22,55 or residency review committee (RRC) citation for lack thereof74 while controlling for confounding variables. Dedicated research time, a research day, program director involvement, and resident recognition for scholarship were associated with resident participation in research in family medicine programs.22 The presence of funding specifically for resident scholarly activity was the only factor protective against RRC citation for lack of such activity in internal medicine residency programs.74 In contrast, only mentorship outside of the department was significantly associated with fewer completed projects among physical medicine and rehabilitation training programs.55

Six articles controlled for confounding variables when examining the outcome of publications.12,14,22,67,68,94 One of these articles also evaluated grant funding,67 and another also evaluated presentations.94 Publications by program directors and by multiple other faculty (six or more faculty publications within two years) were associated with resident publications among family medicine programs.22 After adjustment for gender and prior publications, an association between participation in a research day and resident publications was demonstrated in a pediatric training program.12 In another study, conditional logistic regression confirmed that pediatric residents with a research requirement and research curriculum were significantly more likely than those without a research requirement to publish both during and after residency.14 Accounting for prior publications and postgraduate year, the number of months of protected time for research had a significant impact on the number of manuscripts submitted across urology residency programs.68 After adjustment for gender and prior publications, radiation oncology residents who participated in a specialized research pathway were more likely than nonparticipants to be successful in obtaining grant funding.67 While the quantity of protected time for research was not a significant predictor of grant funding in that study, it was the sole factor associated with resident first-author publications in multivariate analysis. In contrast, no interventions were found to predict resident publications in a multicenter study of emergency medicine residency programs, yet departmental financial support and protected time were associated with presentations.94

Discussion

We found that interventions to increase resident participation in scholarly activity were uniformly positive in their effects. Scholarly activity participation in residency is the minimum ACGME requirement. As resident participation in scholarly activity, without necessarily having a tangible product, is relatively easy to achieve, it is not surprising that multiple different interventions resulted in increased participation. For example, mandating participation of residents in scholarly activity was as effective in increasing participation as more resource-intensive approaches such as protected research time, a research curriculum, and having a research track or research director.

Increasing tangible products of scholarship, such as presentations or publications, was more difficult to achieve, and the results were mixed. In many studies, interventions were bundled, so it was difficult to determine the relative importance of a single intervention. For example, having a research track or research director was associated with increased presentations and publications, but each of these interventions was often paired with protected research time and a research curriculum. Solely giving residents protected research time or a research curriculum was not consistently sufficient to increase presentations and publications, particularly in the medical specialties. Programs may need to provide increased structure and rigor through multiple pathways. In the medical specialties, a research director was associated with increased presentations and publications, emphasizing the importance of having an identified champion of scholarship within a residency program. In surgical specialties, program directors who wish to increase resident publications are likely to achieve success by providing increased protected time for research. Extensive time for research may not be feasible to offer in programs in all specialties without lengthening the time of training, however.

The results from the studies that accounted for potential confounders support the key concept that overall program research culture influences RSAP. When study authors adjusted for prior research experience or publication (which could serve as a proxy for interest in an academic career), all studied interventions (a research day, specialized research track, or protected time) were associated with RSAP.12,67,68 Mentorship, particularly within the department specialty, also played an important role in RSAP.22,55 Finally, funding for resident scholarly activity was associated with program compliance with ACGME guidelines.74

Our review illustrates the diversity of approaches to support RSAP at the programmatic level across graduate medical education specialties, reinforcing the need for clarity in the definition of resident scholarship and expected outcomes. Programs are now required to provide evidence of resident products of scholarly activity through the Next Accreditation System,11 yet how the ACGME will interpret the reported outcomes remains unclear. In response, some specialties such as orthopedic surgery have at the RRC level defined the expected outcomes of participation in sponsored research, manuscript preparation/publication, participation in a structured literature review, or presentations.95 Simultaneously, the ACGME’s 10-year self-study requires programs to define their own aims—which could apply to scholarly activity—“as a way to differentiate programs,” and is expected to “ultimately evaluate program effectiveness in meeting these aims.”96 Through a recent editorial, Simpson and colleagues97 actively solicited the opinions of medical educators about whether all graduating residents should produce scholarship that adds to the body of knowledge and whether requirements for resident scholarship should be redefined. The published findings were mixed, with some respondents advocating a universal requirement for scholarship, while others disagreed, citing logistical barriers and resource challenges.98

We propose that individual residency programs should clearly define their own goals for RSAP, meeting or exceeding the minimum requirement outlined under the RRC for their specialty. While some programs may have resources, infrastructure, and the desire to support scholarly activity during residency to train future physician scientists and academic leaders, others may excel in training future community practitioners. The latter types of programs should focus on ensuring that their graduates understand the rigors of a scholarly approach to clinical questions, appraisal of the literature, quality improvement, and community health and place less emphasis on publications and presentations as outcomes. A third type of residency program may seek to prepare graduates with career interests in either area. Each of these program categories will vary in their RSAP outcomes at the programmatic level, as well as in their choice of interventions to achieve their program goals. Expected RSAP outcomes by program type should be incorporated quantitatively into accreditation review, aligning with the aforementioned goal of the ACGME’s 10-year self-study.96 Further, programs should work with their residents to tailor scholarship opportunities to the residents’ individual career goals within the program’s overall expectations of scholarship.

We support future efforts of programs and the ACGME to define expectations for resident scholarly activity at the program and individual resident levels using a universal language with clearly outlined outcomes in order to reach the goal of developing a cadre of physicians with differing skill sets yet encompassing all domains of scholarship across the profession.99 With expectations defined, programs may then more carefully choose, and rigorously study, interventions that align with their aims to support RSAP.

Limitations

Our review has several limitations. Although our inclusion criteria were broad to maximize our assessment of the available literature on scholarly activity during residency, we did not search conference proceedings, and we limited our review to studies performed in the United States and Canada and published in English. Publication bias may have led to inclusion of studies with more positive outcomes. Not all studies may have accounted for the possibility that residents interested in scholarship self-selected into training programs that supported scholarly activity. Because of the heterogeneity of the studies, it was not possible to combine results quantitatively in a meta-analysis. The cost of the interventions is an important consideration for programs100 but was not well described in the included studies. Finally, multiple barriers to RSAP encountered by residents (e.g., motivation, navigating institutional review boards, time to complete projects, support for study design and analysis), which are beyond the scope of this review, must be considered as programs develop goals and support for resident scholarship.

Conclusions

Interventions such as protected time, research training, a research director, or a specialized research track generally resulted in increased resident participation in scholarly activity in U.S. and Canadian training programs across medical and surgical specialties, but had mixed effects on presentations or publications by residents. Our findings highlight the need for a clear definition of resident scholarly activity success, aligned specifically to individual program and resident goals.

Acknowledgments: The authors wish to thank Kendra Sikes, E.I.T., and the Kornhauser Health Sciences Library interlibrary loan staff for their assistance.

References

1. Chesney RW, Dungy CI, Gillman MW, et al.; Committee on Pediatric Research. Promoting education, mentorship, and support for pediatric research. Pediatrics. 2001;107:14471450.
2. Rosenberg LE. Young physician–scientists: Internal medicine’s challenge. Ann Intern Med. 2000;133:831832.
3. Crowley RW, Asthagiri AR, Starke RM, Zusman EE, Chiocca EA, Lonser RR; Research Committee of the Congress of Neurological Surgeons. In-training factors predictive of choosing and sustaining a productive academic career path in neurological surgery. Neurosurgery. 2012;70:10241032.
4. Macknin JB, Brown A, Marcus RE. Does research participation make a difference in residency training? Clin Orthop Relat Res. 2014;472:370376.
5. Namdari S, Jani S, Baldwin K, Mehta S. What is the relationship between number of publications during orthopaedic residency and selection of an academic career? J Bone Joint Surg Am. 2013;95:e45.
6. Cull WL, Yudkowsky BK, Schonfeld DJ, Berkowitz CD, Pan RJ. Research exposure during pediatric residency: Influence on career expectations. J Pediatr. 2003;143:564569.
7. Burkhardt J, Kowalenko T, Meurer W. Academic career selection in American emergency medicine residents. Acad Emerg Med. 2011;18(suppl 2):S48S53.
8. Kanna B, Deng C, Erickson SN, Valerio JA, Dimitrov V, Soni A. The research rotation: Competency-based structured and novel approach to research training of internal medicine residents. BMC Med Educ. 2006;6:52.
9. Hebert RS, Levine RB, Smith CG, Wright SM. A systematic review of resident research curricula. Acad Med. 2003;78:6168.
10. Accreditaton Council for Graduate Medical Education. ACGME common program requirements. Effective July 1, 2015. https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs_07012015.pdf. Accessed February 10, 2016.
11. Philibert I, Lieh-Lai M, Miller R, Potts JR 3rd, Brigham T, Nasca TJ. Scholarly activity in the next accreditation system: Moving from structure and process to outcomes. J Grad Med Educ. 2013;5:714717.
12. Mills LS, Steiner AZ, Rodman AM, Donnell CL, Steiner MJ. Trainee participation in an annual research day is associated with future publications. Teach Learn Med. 2011;23:6267.
13. Vinci RJ, Bauchner H, Finkelstein J, Newby PK, Muret-Wagstaff S, Lovejoy FH Jr. Research during pediatric residency training: Outcome of a senior resident block rotation. Pediatrics. 2009;124:11261134.
14. Ozuah PO. Residency research requirement as a predictor of future publication productivity. J Pediatr. 2009;155:12, 2.e1.
15. Roth DE, Chan MK, Vohra S. Initial successes and challenges in the development of a pediatric resident research curriculum. J Pediatr. 2006;149:149150.
16. Shin AY, Almond CS, Mannix RC, et al.; Boston Combined Residency Program. The Boston Marathon study: A novel approach to research during residency. Pediatrics. 2006;117:18181822.
17. Seehusen DA, Asplund CA, Friedman M. A point system for resident scholarly activity. Fam Med. 2009;41:467469.
18. Desbiens NA. A departmental experience in promoting oral and poster presentations. Teach Learn Med. 2008;20:254260.
19. Crownover B, Crawford PF. Areas of concentration increase scholarly activity: A 15-month experience. Fam Med. 2008;40:8790.
20. Abramson EL, Naifeh MM, Stevenson MD, et al. Research training among pediatric residency programs: A national assessment. Acad Med. 2014;89:16741680.
21. Ledford CJ, Seehusen DA, Villagran MM, Cafferty LA, Childress MA. Resident scholarship expectations and experiences: Sources of uncertainty as barriers to success. J Grad Med Educ. 2013;5:564569.
22. Crawford P, Seehusen D. Scholarly activity in family medicine residency programs: A national survey. Fam Med. 2011;43:311317.
23. DeHaven MJ, Wilson GR, O’Connor-Kettlestrings P. Creating a research culture: What we can learn from residencies that are successful in research. Fam Med. 1998;30:501507.
24. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009;6:e1000097.
25. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:10021009.
26. Gill S, Levin A, Djurdjev O, Yoshida EM. Obstacles to residents’ conducting research and predictors of publication. Acad Med. 2001;76:477.
27. Fisher S, Bender SK. A program of research training in psychiatry: Ten-year evaluation and follow-up. Am J Psychiatry. 1975;132:821824.
    28. McCabe BF. Resident research in otolaryngology at the University of Iowa. Can J Otolaryngol. 1975;4:846848.
      29. Goldman L, Shea S, Wolf M, Braunwald E. Clinical and research training in parallel: The internal medicine research residency track at the Brigham and Women’s Hospital. Clin Res. 1986;34:15.
        30. Sulak PJ, Croop JA, Hillis A, Kuehl TJ. Resident research in obstetrics and gynecology: Development of a program with comparison to a national survey of residency programs. Am J Obstet Gynecol. 1992;167:498502.
          31. Housestaff team research in the ambulatory setting: it can be done. Internal Medicine Clinic Research Consortium Faculty. J Gen Intern Med. 1995;10:219222.
            32. Chung R, Diaz J, Li P. A method of teaching clinical research in a community hospital residency program. Am J Surg. 1999;177:8385.
              33. Gay SB, Hillman BJ. Evaluation of a mandatory radiology resident research rotation. Acad Radiol. 2000;7:172175.
                34. El-Mallakh RS. A novel residents’ research seminar. Acad Psychiatry. 2001;25:5760.
                  35. Lampman RM, Wolk SW, Fowler J, et al. Resident research training conducted in a community hospital general surgery residency program. Curr Surg. 2003;60:304309.
                    36. Bernstein J, Ahn J, Iannotti JP, Brighton CT. The required research rotation in residency: The University of Pennsylvania experience, 1978–1993. Clin Orthop Relat Res. 2006;449:9599.
                      37. Holmes JF, Sokolove PE, Panacek EA. Ten-year experience with an emergency medicine resident research project requirement. Acad Emerg Med. 2006;13:575579.
                        38. Rahal A, Head HW, Jung AJ, et al. Combined radiology residency/PhD program for education of academic radiologists: A response to revitalizing the radiology research enterprise. Radiology. 2007;245:1420.
                          39. Kunik ME, Hudson S, Schubert B, Nasrallah H, Kirchner JE, Sullivan G. Growing our own: A regional approach to encourage psychiatric residents to enter research. Acad Psychiatry. 2008;32:236240.
                            40. Lovejoy FH Jr, Zuckerman BS, Fleisher GR, Vinci RJ; Association of Medical School Pediatric Department Chairs, Inc. Creating an academic culture during residency training. J Pediatr. 2008;152:599600, 600.e1.
                              41. Jibson MD, Hilty DM, Arlinghaus K, et al. Clinician–educator tracks for residents: Three pilot programs. Acad Psychiatry. 2010;34:269276.
                                42. Back SE, Book SW, Santos AB, Brady KT. Training physician–scientists: A model for integrating research into psychiatric residency. Acad Psychiatry. 2011;35:4045.
                                  43. Brubaker L, Kenton K. Clinical research education study teams: A research curriculum for obstetric and gynecology residents. Obstet Gynecol. 2011;117:14031407.
                                    44. Carek PJ, Dickerson LM, Diaz VA, Steyer TE. Addressing the scholarly activity requirements for residents: One program’s solution. J Grad Med Educ. 2011;3:379382.
                                      45. Noggle CA, Punwani M. Revitalization of research education in psychiatry: Implications for training during residency. Acad Psychiatry. 2012;36:419421.
                                        46. Wallner PE, Ang KK, Zietman AL, et al. The American Board of Radiology Holman Research Pathway: 10-year retrospective review of the program and participant performance. Int J Radiat Oncol Biol Phys. 2013;85:2934.
                                          47. Andrews JM, Abdolell M, Norman RW. Canadian urology resident scholarly performance. Can Urol Assoc J. 2013;7:E402E406.
                                            48. Palacio A, Campbell DT, Moore M, Symes S, Tamariz L. Predictors of scholarly success among internal medicine residents. Am J Med. 2013;126:181185.
                                              49. Wagner RF Jr, Raimer SS, Kelly BC. Incorporating resident research into the dermatology residency program. Adv Med Educ Pract. 2013;4:7781.
                                                50. Roane DM, Inan E, Haeri S, Galynker II. Ensuring research competency in psychiatric residency training. Acad Psychiatry. 2009;33:215220.
                                                  51. Ables AZ, Carek PJ, Dickerson LM, Abercrombie S, Hanlin RB. Promoting resident scholarship through a statewide symposium. Fam Med. 2010;42:440443.
                                                  52. Ahmad S, De Oliveira GS Jr, McCarthy RJ. Status of anesthesiology resident research education in the United States: Structured education programs increase resident research productivity. Anesth Analg. 2013;116:205210.
                                                  53. Basu Ray I, Henry TL, Davis W, Alam J, Amedee RG, Pinsky WW. Consolidated academic and research exposition: A pilot study of an innovative education method to increase residents’ research involvement. Ochsner J. 2012;12:367372.
                                                  54. Bhattacharya SD, Williams JB, de la Fuente SG, Kuo PC, Seigler HF. Does protected research time during general surgery training contribute to graduates’ career choice? Am Surg. 2011;77:907910.
                                                  55. Blake DJ, Lezotte DC, Yablon S, Rondinelli RD. Structured research training in residency training programs. The impact on the level of resident research activity. Am J Phys Med Rehabil. 1994;73:245250.
                                                  56. Byrnes AB, McCormack FX, Diers T, Jazieh AR. The resident scholar program: A research training opportunity for internal medicine house staff. J Cancer Educ. 2007;22:4749.
                                                  57. Chan RK, Lockyer J, Hutchison C. Block to succeed: The Canadian orthopedic resident research experience. Can J Surg. 2009;52:187195.
                                                  58. Dengel LT, Smith PW, Kron IL, Schirmer BD, Slingluff CL Jr, Schroen AT. Resident research forums stimulate novel research within general surgical training programs. J Surg Educ. 2009;66:146151.
                                                  59. Dunn JC, Lai EC, Brooks CM, Stabile BE, Fonkalsrud EW. The outcome of research training during surgical residency. J Pediatr Surg. 1998;33:362364.
                                                  60. Dunnick NR. Radiology residency training programs: Current status. Radiology. 1988;169:549552.
                                                  61. Durning SJ, Cation LJ, Ender PT, Gutierrez-Nunez JJ. A resident research director can improve internal medicine resident research productivity. Teach Learn Med. 2004;16:279283.
                                                  62. Elliott ST, Lee ES. Surgical resident research productivity over 16 years. J Surg Res. 2009;153:148151.
                                                  63. Ellis MC, Dhungel B, Weerasinghe R, Vetto JT, Deveney K. Trends in research time, fellowship training, and practice patterns among general surgery graduates. J Surg Educ. 2011;68:309312.
                                                  64. Fancher TL, Wun T, Hotz CS, Henderson MC. Jumpstarting academic careers with a novel intern research rotation: The AIMS rotation. Am J Med. 2009;122:10611066.
                                                  65. Fischer JL, Cation LJ. Impact of a residency research program on research activity, faculty involvement, and institutional cost. Teach Learn Med. 2005;17:159165.
                                                  66. Fisher C, Baker MK. Improving participation and quality of clinical research in a university-based general surgery residency program. Am Surg. 2010;76:741742.
                                                  67. Gutovich JM, Den RB, Werner-Wasik M, Dicker AP, Lawrence YR. Predictors of radiation oncology resident research productivity. J Am Coll Radiol. 2013;10:185189.
                                                  68. Hellenthal NJ, Ramirez ML, Yap SA, Kurzrock EA. Manuscript publication by urology residents and predictive factors. J Urol. 2009;181:281286.
                                                  69. Hepburn MJ, Battafarano DF, Enzenauer RJ, et al. Increasing resident research in a military internal medicine program. Mil Med. 2003;168:341345.
                                                  70. Hillman BJ, Nash KD, Witzke DB, Fajardo LL, Davis D. The RSNA-AUR-ARRS introduction to research program for 2nd year radiology residents: Effect on career choice and early academic performance. Radiological Society of North America. Association of University Radiologists. American Roentgen Ray Society. Radiology. 1998;209:323326.
                                                  71. Hobson WL, Bruse J, Bale JF Jr. Creating flexibility in pediatric resident education. Pediatrics. 2011;127:10881093.
                                                  72. Konstantakos EK, Laughlin RT, Markert RJ, Crosby LA. Assuring the research competence of orthopedic graduates. J Surg Educ. 2010;67:129134.
                                                  73. Kurahara DK, Kogachi K, Yamane M, et al. A pediatric residency research requirement to improve collaborative resident and faculty publication productivity. Hawaii J Med Public Health. 2012;71:224228.
                                                  74. Levine RB, Hebert RS, Wright SM. Factors associated with citation of internal medicine residency programs for lack of scholarly activity. Teach Learn Med. 2005;17:328331.
                                                  75. Liese BS, Johnson CA, Govaker DA, O’Dell ML. Increasing research productivity in a university-based residency program: A case study. Kans Med. 1988;89:143147.
                                                  76. Lohr J, Smith JM, Welling R, Engel A, Hasselfeld K, Rusche J. Stimulating resident research in a general surgery residency community program. Curr Surg. 2006;63:426434.
                                                  77. Mezzacappa E, Hamoda HM, DeMaso DR. Promoting scholarship during child and adolescent psychiatry residency. Acad Psychiatry. 2012;36:443447.
                                                  78. Millis SR, Campagnolo DI, Kirshblum S, Elovic E, Jain SS, DeLisa JA. Improving resident research in physical medicine and rehabilitation: Impact of a structured training program. J Spinal Cord Med. 2004;27:428433.
                                                  79. Morgan PB, Sopka DM, Kathpal M, Haynes JC, Lally BE, Li L. First author research productivity of United States radiation oncology residents: 2002–2007. Int J Radiat Oncol Biol Phys. 2009;74:15671572.
                                                  80. Morrison JC, Meeks GR, Martin JN Jr, Cowan BD, Whitworth NS, Wiser WL. Resident research projects: Frequency of presentation and publication in a national forum. Am J Obstet Gynecol. 1994;170:777781.
                                                  81. Robertson CM, Klingensmith ME, Coopersmith CM. Long-term outcomes of performing a postdoctoral research fellowship during general surgery residency. Ann Surg. 2007;245:516523.
                                                  82. Ruiz J, Wallace EL, Miller DP, et al. A comprehensive 3-year internal medicine residency research curriculum. Am J Med. 2011;124:469473.
                                                  83. Sabir M, Penney DG, ReMine SG, Mittal VK. Scholarly activities—essential to surgical education. Curr Surg. 2003;60:459462.
                                                  84. Sakai T, Karausky PL, Valenti SL, Sandusky SL, Hirsch SC, Xu Y. Use of a problem-based learning discussion format to teach anesthesiology residents research fundamentals. J Clin Anesth. 2013;25:434438.
                                                  85. Segal NA, Garvan CW, Basford JR. Factors influencing involvement in research and career choice: A survey of graduating physical medicine and rehabilitation residents. Arch Phys Med Rehabil. 2006;87:14421446.
                                                  86. Strauss GD, Yager J, Offer D. Research training in psychiatry: A survey of current practices. Am J Psychiatry. 1980;137:727729.
                                                  87. Taniguchi MH, Johnson PD. Rehabilitation resident academic productivity. Report on 1993 graduates. Am J Phys Med Rehabil. 1994;73:240244.
                                                  88. West CP, Halvorsen AJ, McDonald FS. Scholarship during residency training: A controlled comparison study. Am J Med. 2011;124:983987.e1.
                                                  89. Wilson JL, Redman RW. Research policies and practices in family practice residencies. J Fam Pract. 1980;10:479483.
                                                  90. Yang G, Zaid UB, Erickson BA, Blaschko SD, Carroll PR, Breyer BN. Urology resident publication output and its relationship to future academic achievement. J Urol. 2011;185:642646.
                                                  91. Segal LS, Black KP, Schwentker EP, Pellegrini VD. An elective research year in orthopaedic residency: How does one measure its outcome and define its success? Clin Orthop Relat Res. 2006;449:8994.
                                                  92. Alguire PC, Anderson WA, Henry RC. Teaching research skills: Development and evaluation of a new research program for residents. Teach Learn Med. 1993;5:3743.
                                                  93. Kirchner JE, Owen RR, Nordquist CR, Clardy JA. Developing clinician–scientists through integrated research training in psychiatry. Teach Learn Med. 1998;10:183187.
                                                  94. Levitt MA, Terregino CA, Lopez BL, Celi C. Factors affecting research directors’ and residents’ research experience and productivity in emergency medicine training programs. Acad Emerg Med. 1999;6:356359.
                                                  95. Accreditation Council for Graduate Medical Education. ACGME program requirements for graduate medical education in orthopaedic surgery. Effective July 1, 2016. https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/260_orthopaedic_surgery_2016.pdf. Accessed February 10, 2016.
                                                  96. Accreditation Council for Graduate Medical Education. The self-study and 10-year site visit: Clarifying the timeline and action steps: Webinar for designated institutional officials. 2015. https://www.acgme.org/acgmeweb/Portals/0/PDFs/Webinars/DIOWebinarSelfStudy03112015.pdf. Accessed February 10, 2016.
                                                  97. Simpson D, Yarris LM, Carek PJ. Defining the scholarly and scholarship common program requirements. J Grad Med Educ. 2013;5:539540.
                                                  98. Carek PJ, Simpson D, Yarris LM. Comments: Authors’ response to letters regarding “defining the scholarly and scholarship common program requirements.” J Grad Med Educ. 2014;6:392.
                                                  99. Grady EC, Roise A, Barr D, et al. Defining scholarly activity in graduate medical education. J Grad Med Educ. 2012;4:558561.
                                                  100. Sakai T. Residents’ scholarly activity: A cost analysis with regard to its effects on departments. Curr Opin Anaesthesiol. 2015;28:180185.
                                                  Appendix 1
                                                  Appendix 1:
                                                  Summary of the Effect of Interventions on Resident Scholarly Activity Productivity (RSAP) in 54 Studies Pertaining to Medical and Surgical Specialties, Published as of October 2013
                                                  Appendix 2
                                                  Appendix 2:
                                                  Detailed Results of the Effect of Interventions on Resident Scholarly Activity Productivity in 23 U.S. and Canadian Studies Pertaining to Medical Specialties, Published as of October 2013
                                                  Appendix 3
                                                  Appendix 3:
                                                  Detailed Results of the Effect of Interventions on Resident Scholarly Activity Productivity in 12 U.S. and Canadian Studies Pertaining to Surgical Specialties, Published as of October 2013

                                                  Supplemental Digital Content

                                                  Copyright © 2016 by the Association of American Medical Colleges