Dyrbye, Liselotte N. MD; Davidson, Laurie W. MLIS; Cook, David A. MD, MHPE
Many medical schools in the United States and Canada offer elective research experiences for their students,1–7 with a handful of schools requiring students to complete a research project before graduation.8,9 On the Association of American Medical Colleges Graduation Questionnaire, more than half of all 2006 graduates reported that they had participated in a research project.10 Students who had conducted research during medical school reported gains in knowledge and skills in appraising the literature,3–5,11 analyzing data,3,4,9 and writing for publication,3–5 along with more positive attitudes toward future research.3–5,12,13 Only a few studies report more tangible measures of research productivity such as publications and presentations at extramural research conferences.3–5,9,10,13,14 Less is known about research products resulting from required research experiences, in which students might be less invested than they would have been if they had chosen to conduct research.
Looking more closely at required research experiences, only one study reports published articles (17 publications from 201 student participants).14 Two studies using self-report surveys report anticipated publications following required research experiences (46 “expected” publications for 60 students in one study and 7 submitted manuscripts for 63 students in another),3,9 and another study reports that one third of the students presented their research at a regional research conference.15 Given the limited information on actual productivity from required research experiences, additional study is warranted. Furthermore, reports published thus far have not clarified important features of the required experience such as its duration or how the experience might be related to subsequent research productivity. With these considerations in mind, we conducted the study at Mayo Medical School (MMS) described below.
MMS has had a required research experience since the establishment of the school in 1972. Students identify their research project mentor with help from MMS faculty, deans, and the student’s advisor, and write a research proposal. The research coordinating committee reviews proposals for scientific merit and feasibility and provides feedback to students before students start their research. The faculty members who mentor the student research projects are supported for up to 8% of their time for each student mentee. Mentors also receive up to $750 for laboratory expenses per student. The role of the faculty is to supervise the preliminary design of the student’s research project, approve the student’s research proposal and sponsor the proposed project when presented to the research coordinating committee, supervise the student’s research project, and evaluate the student’s performance using a preestablished evaluation form. Students at MMS are required to write a “medical communication report” organized as a standard research manuscript (i.e., title, abstract, introduction, materials and methods, results, discussion, literature cited, tables, legends to figures, and figures) before graduation. Most students base the report on work completed during the required research experience. Some students write review articles or report on research findings resulting from elective research experience. Students fulfill the requirement for the medical communication report when they submit a reprint of a published article, a copy of a manuscript accepted for publication, or a manuscript draft.
The required research experience has always taken place during the third year of medical school, but the time allotted for research has decreased over the years to accommodate various curricular changes. Initially 21 weeks long, the experience was shortened to 18 weeks in 1992 and to 17 weeks the following year, and then to 13 weeks in 2002. The research weeks are taken consecutively, and the students have no concurrent classroom or clinical responsibilities. Despite the reduction in curricular time for the required research experience, the organizational structure of the required research experience and expectations of students have not changed. For a detailed description of the required research experience, see (http://www.mayo.edu/mms/md-research-quarter.html).
The purpose of our study was to determine (1) students’ research productivity, as measured by research reports and research abstracts published and extramural presentations delivered based on the required research experience at MMS, (2) whether this research productivity has varied depending on the duration of the research experience (21, 17–18, or 13 weeks), and (3) whether having a research product resulting from the required research experience (published research reports, published research abstract, or a research presentation during medical school) is associated with greater short-term productivity (research reports published through the first three years after graduation). Understanding the immediate research productivity resulting from a required medical school research experience and the association between research products during medical school and future research productivity will be helpful to guide stakeholders, such as medical school administrators and curriculum planners and medical school applicants, all of whom must consider the utility of a required research component.
Study design, subjects, and setting.
We conducted a retrospective cohort study of all 1,105 medical students graduating between 1976 (the first graduating class) and 2003 from MMS, a small private Midwestern medical school. We excluded graduates who had been excused from the required research component. The Mayo Clinic institutional review board deemed this study exempt.
Data sources and variables.
Four independent variables were defined for all eligible participants: year of graduation from medical school, gender, duration of required research experience (i.e., 13, 17, 18, or 21 weeks), and type of research project. We classified research types on the basis of project title as patient based/public health research, laboratory-based biological/medical research, medical education research, or other. We defined three dependent variables related to research productivity: number of published research reports, number of published research abstracts, and number of extramural research presentations.
We defined published research reports as full-length articles, including original research and reviews, authored or coauthored by the graduate. Letters to the editor, opinion pieces, program descriptions, etc., were excluded. For each graduate, we searched MEDLINE by author (i.e., last name—maiden and married where applicable—first name, and middle initial) for research reports. We limited the search to research reports published between the year of matriculation into MMS and three years after graduation, to allow for publishing delay. To avoid erroneous credit due to common names, we attributed research reports to the graduate if the name matched and at least one of the following criteria was met: (1) the subject matter of the publication corresponded to the title of the graduate’s required research project, (2) the subject matter of the publication corresponded to the graduate’s medical specialty (e.g., urology), (3) the author’s previous or current institutional address corresponded to the graduate’s address near the time of publication, or (4) the report was coauthored by the graduate’s research mentor. We ascertained graduates’ medical specialties and institutional affiliations and associated dates from existing Mayo databases, the American Medical Association’s “Find a Doctor” database (http://dbapps.ama-assn.org/aps/amahg.htm), the American Board of Medical Specialties database, or online personal curricula vitae. Graduates who were authors on research reports but for whom we were unable to include or exclude authorship on the basis of the above criteria (ambiguous attribution) were excluded from the data set when analyzing the number of research reports published.
Published research abstracts.
We defined published research abstracts as abstracts authored or coauthored by the graduate and presented at scientific meetings and published in journals. We did not distinguish who presented the abstract, because this information was typically not available. We searched the Mayo Authors Database by graduate name for abstracts authored or coauthored by the graduate and published in journals while the graduate was an MMS student. The Mayo Authors Database is maintained by Mayo librarians, who find and verify published work (including journal articles, published abstracts of presentations prepared for scientific meetings, letters to editors, and books) authored by physician and nonphysician staff, residents, and students during their tenure at Mayo Clinic. Abstracts were attributed to the graduate if the name matched and at least one of the following criteria were met: (1) the subject matter of the publication corresponded to that indicated by the title of the graduate’s required research project, or (2) the abstract was coauthored by the graduate’s research mentor. Ambiguous attribution was avoided by using the Mayo Authors Database.
Research presentations by the graduate.
Because any of an abstract’s authors could have delivered the presentation at a scientific meeting, we measured a separate outcome for research presentations. Research presentations were identified by (1) requests for absenteeism from medical school to present at a scientific meeting, (2) requests for reimbursement for travel to present at a scientific meeting outside of Rochester, Minnesota (the location of the medical school), or (3) requests for reimbursement for conference registration reimbursement. Medical school absenteeism requests are required for all absences, including presentation trips. Because absenteeism requests and travel reimbursement forms were available for the 1978 graduating class onward, we were unable to collect research presentation data from the classes of 1976 and 1977.
Research product related to the required research experience.
Not all identified research products were products of the required research experience, because some graduates did elective research in medical school or conducted research after graduation. To determine which research reports and abstracts were related to the required research experience, we compared each graduate’s publication title against the title of their required research project. If the research report or abstract title was similar to the research project title, it was designated as a research product related to the required research experience. Two of us (L.N.D. and D.A.C.) independently classified titles as related or unrelated to the required research experience for a random sample of 10% (119) titles. The interrater agreement (kappa = 0.88) was excellent.16 One of us (L.N.D.) coded the remaining titles. We were unable to determine whether research findings presented at extramural meetings resulted from the required research experience or from other research projects conducted during medical school, because the information pertaining to the content of the research presentation was not recorded on the absenteeism requests or travel reimbursement forms.
Because only one graduating class had 18 weeks of research, these graduates were combined with those who had 17 weeks of research for statistical analyses. Demographic variables and research productivity were summarized using descriptive statistics. Because of the skewed distributions for some measures, nonparametric statistical procedures were used for data analysis. Research productivity was compared between the different durations of research experience (21, 17–18, or 13 weeks) using the Kruskal-Wallis test, with pairwise comparisons using the Wilcoxon rank-sum test for significant models. The proportion of graduates generating research products was compared using the chi-square test. The Wilcoxon rank-sum test was used to compare the number of published research reports unrelated to the required research experience between those who did and did not publish at least one research report related to their research experience. The number of published articles (both total and unrelated) was also compared between those who did and did not have a published abstract or present their research extramurally.
Of the 1,105 students enrolled in MMS between 1976 and 2003, 107 (10%) were excused from the required research experience (see Figure 1 for details); those graduates were excluded from this study.
Among the 998 MMS graduates participating in the required research experience, 630 (63%) were male. For their required research project, 614 (62%) had conducted patient-based or public health research, 372 (37%) had done laboratory-based research, and fewer than 1% had done medical education research (3; <1%) or another type of research (9; <1%). There were 594 graduates in the graduating classes of 1976 to 1991 who spent 21 weeks doing required research, 336 from 1992 to 2001 who spent 17 to 18 weeks doing required research, and 68 graduates in 2002 and 2003 who spent 13 weeks doing required research. Figure 1 shows the number of graduates included in our analysis of published research reports, published abstracts, and presentations, both in total and for each duration of required research.
Overall research productivity
Overall, the required research experience resulted in 554 research reports and 258 abstracts published during the 26 years of the program. The graduate was the first author on the majority of these research reports (360; 65%) and abstracts (187; 73%). In addition, there were 416 extramural research presentations.
Of the 998 graduates participating in the required experience, 17 were excluded when considering authorship of research reports because of ambiguous attribution of authorship. Of the remaining 981 graduates, 400 (41%) published one or more research reports related to their required research experience (see Table 1). Approximately one out of six graduates (176/998, 18%) published one or more abstracts related to their required research project. Forty-four percent of graduates (444/998) published a research report or abstract related to their required research experience. Among the 920 graduates from 1978 to 2003 (data not available for 1976–1977), 375 (41%) presented their research at an extramural meeting at least once. Ninety percent (337/375) had one research presentation, 35 (9%) had two, and three (1%) had three presentations. Sixty-six percent of graduates (594/907; classes of 1976 and 1977 excluded and 13 graduates not included because of ambiguous attribution of authorship) had at least one research report, abstract, or research presentation. There were differences in published research reports by gender, with more male students publishing research reports related to the required research curriculum (278/623; 45% males versus 122/358; 34% females, P = .001) and being the first author on such a publication (211/623; 34% versus 93/358; 26%, P = .01). The mean number of research reports and abstracts related to the required research curriculum and extramural presentations are shown in Table 2.
Research productivity by duration of required research
There was no difference in the proportions of graduates publishing a research report or abstract or presenting at a scientific meeting between the three duration groups (see Table 1). There was a significant difference with respect to first authorship (P = .005 overall), with more graduates in the 21-week group being first authors (203/584; 35%) than those in the 17/18-week group (80/329; 24%, P = .001). A similar pattern was seen for mean numbers of research reports, abstracts, and presentations. See Table 2 for additional details.
Relationship between publications related to the required research experience and other publications within three years of graduation
To explore the relationship between productivity in the required experience and overall research productivity, we compared those who did and did not have a research report or abstract related to the required experience. Graduates who published a research report related to their required research had more nonrelated research reports (we looked at nonrelated research reports only, to avoid colinearity) published in the eligible time frame than their peers who did not publish a research report related to the required experience (0.5 ± 1.1 versus 0.3 ± 0.9, P < .0001). Similarly, publishing an abstract related to required research was associated with publishing more total (1.6 ± 1.7 versus 0.8 ± 1.3, P < .0001) and nonrelated research reports (0.5 ± 0.9 versus 0.4 ± 1.0, P ≤ .04). Students with research presentations were also more likely to have more total (1.3 ± 1.5 versus 0.7 ± 1.4, P < .0001) and nonrelated research reports (0.5 ± 1.1 versus 0.3 ± 0.9, P = .001).
We found that 594/907 (66%) of the graduates participating in the MMS required medical school research experience had a tangible research product (published research report, published abstract, or extramural presentation) to show for their effort, and 41% published a research report directly related to their required research experience. However, research productivity, with the exception of graduates who had lead authorship, did not vary in relation to time allotted for the research experience. (Regarding those with lead authorship, there was a significant difference.) Graduates with a research product related to the required research experience or a research presentation during medical school had more research products within three years of graduation, suggesting that having a research product during medical school might stimulate pursuit of further research and submission of research reports, at least in the short term. This information is useful to medical school leaders and curriculum planners contemplating a required research component, and to students wondering what research products could be expected if they engaged in research during medical school.
Despite the reduction in the duration of the required research experience, most measures of research productivity did not change over time. The only notable change consisted of differences in the proportions of graduates with lead authorship on a research report, with first authorship occurring less often with shorter research experiences. Being granted the first author position by one’s collaborators suggests substantial contributions to the research design, project implementation, data analysis, and manuscript preparation. The decrease in lead authorship found in relation to the duration of the research experience suggests that fewer graduates were able to engage as fully during the shorter time periods. This could signify fewer gains for the graduate with respect to research knowledge and skills. There may be alternative explanations for the decline in first authorship among students, such as a greater need for other research team members (e.g., fellow research members of the lab) to be academically productive. Regardless, other productivity measures were similar over time, so this effect, if present, is likely small.
We also found an association between graduates publishing a research report or abstract related to their required research experience or having a presentation during medical school, and overall publishing productivity through the first three years after graduation. One could speculate that doing required research during medical school fosters knowledge and skills needed to conduct other research and motivates graduates to do so. Some retrospective cohort studies have reported associations between medical school research and postgraduate research17 as well as between medical school research and academic careers, academic rank, and number of citations to published work.18
Our finding that 41% of the students had a publication is consistent with findings of prior studies in which the number of students with anticipated or published research reports following required research experiences varied from about 10%9,14 to 76%.3 The variability in outcomes was likely attributable to differences in measured outcomes (the number of confirmed publications is, as expected, lower), research interests or abilities of students at studied schools, and/or curricular design. Yet, taken together, these findings suggest that a school administrator or prospective student might reasonably expect that somewhere between 10% and 70% of students will have a research report published following a required research experience, and a slightly higher percentage will have a product of any sort (e.g., publication, abstract, or oral presentation). Students who choose to do elective research experiences4–7,19 seem to publish more frequently than do those in required research, with all but one study of elective experiences showing that more than 30% of students generated a publication. This is not surprising, because students who choose to do research are likely more motivated. However, the research requirement ensures that students who might otherwise not engage in research, perhaps because of inaccurate perceptions of research-related tasks, have the opportunity to do so. Such experience is considered important to professional development and to the promotion of interests in biomedical science research.20,21 Our findings further suggest that most students in required research will successfully complete and disseminate the results of a research project. The research requirement also conveys the value a medical school (and, by extension, society) places on research, a message that may help maintain the pipeline of physician scientists.22,23
The curriculum design that optimally stimulates productivity research during the required research experience cannot be ascertained from published reports. However, research on elective experiences suggests that features such as timing (preclinical electives6,7,19 seem less productive than experiences in the clinical years5), funding (research support from endowments4 or the National Institutes of Health19 are more productive), and curricular design (structured didactic components19and flexible curricula that allow for more than four years of medical school4 improve productivity) may explain some of the variability in research productivity resulting from elective research experiences. Our results suggest that a required research curriculum consisting of close review of students’ research proposals, support for researcher’s time and research expenses, and 13 weeks of dedicated time for research in the third year without the addition of formal classroom research training enables most students to implement their research studies, analyze the results, and report the findings in some venue.
This study has several limitations. First, it was conducted at a single institution. Since students may have chosen to come to MMS because of the structured research experience, their interest and motivation may not be typical of that of other medical students. Second, as is typical of retrospective cohort studies, we did not have data on factors that could affect research productivity such as curricular changes or variations in the number of venues to publish or present research work. Third, if students published research reports or abstracts resulting from their required research project in journals not indexed in MEDLINE or more than three years after graduation from medical school, these reports were not included this study. However, because the vast majority of students whose projects were identified did patient-based, public health, or laboratory-based research, we believe that the number not included is unlikely to be substantial. Fourth, the number of published abstracts and extramural research presentations reported may underestimate actual research productivity because of limitations in data-collection methods.
There are several strengths in this study. First, the sample studied was three times as large as those of the three previous studies of required research experiences combined.3,9,14 Second, we report objectively verified research productivity outcomes and, thus, avoid error from self-reporting bias, manuscripts reported as “expected publications” that were never published, and low survey-response rates. Third, we report research productivity for all students regardless of their interest in conducting research. This reduces selection bias that might result from students participating in elective research opportunities. Fourth, we made a distinction between research reports and abstracts that were and were not related to the required research experience, and we were able to do so with high interrater reliability. Finally, our study describes research outcomes from a required research experience that has spanned nearly three decades.
Findings from this study suggest that a required medical school research experience seems to facilitate tangible research products and greater subsequent research productivity. Shorter experiences (13 weeks) seem to yield outcomes similar to those typical of longer experiences. Future research is needed to explore whether conducting research during medical school leads to the acquisition of knowledge, skills, and attitudes that students would not otherwise have obtained from the curriculum. If so, what are the optimal experiences and duration of experiences needed to maximize benefits in an already crowded undergraduate curriculum? It would also be a worthwhile endeavor to explore gender differences in research projects and mentor–mentee relationships. Well-designed, multicentered studies exploring the role of required research in sparking sustained interest in careers as physician–scientists could benefit the continued advancement of medical science and, ultimately, our patients.
The authors thank Tanya Hoskin for assistance with statistical analysis.
Investigators received salary support from institutional sources, none of which were involved in the study design, data collection and analyses, or data interpretation, or in the decision to submit the manuscript for publication.
2 Zier K, Stagnaro-Green A. A multifaceted program to encourage medical students’ research. Acad Med. 2001;76:743–747.
3 Houlden RL, Raja JB, Collier CP, et al. Medical students’ perceptions of an undergraduate research elective. Med Teach. 2004;26:659–661.
4 Jacobs CD, Cross PC. The value of medical student research: the experience at Stanford University School of Medicine. Med Educ. 1995;29:342–346.
5 Frishman WH. Student research projects and theses: Should they be a requirement for medical school graduation? Heart Dis. 2001;3:140–144.
6 Godkin MA. A successful research assistantship program as reflected by publications and presentations. Fam Med. 1993;25:45–47.
7 Gonzales AO, Westfall J, Barley GE. Promoting medical student involvement in primary care research. Fam Med. 1998;30:113–116.
8 Schor NF, Troen P, Kanter SL, et al. The Scholarly Project Initiative: Introducing scholarship in medicine through a longitudinal mentored curricular program. Acad Med. 2005;80:824–831.
9 Smith FG, Harasym PH, Mandin H, et al. Development and evaluation of a research project program for medical students at the University of Calgary Faculty of Medicine. Acad Med. 2001;76:189–194.
11 Ebbert J. A retrospective evaluation of research in the medical curriculum. J Med Educ. 1960;35:637–643.
12 Paiva RE, Donnelly C, Haley HB, et al. Factors related to medical students’ research activities. J Med Educ. 1975;50:339–345.
13 Kassebaum DG, Szenas PL, Ruffin AL, et al. The research career interests of graduating medical students. Acad Med. 1995;70:848–852.
14 Rhyne RL. A scholarly research requirement for medical students: The ultimate problem-based learning experience. Acad Med. 2000;75:523–524.
15 Rosenblatt RA, Desnick L, Corrigan C, et al. The evolution of a required research program for medical students at the University of Washington School of Medicine. Acad Med. 2006;81:877–881.
16 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
17 Segal S, Lloyd T, Houts PS, et al. The association between students’ research involvement in medical school and their postgraduate medical activities. Acad Med. 1990;65:530–533.
18 Brancati F, Mead LA, Levine DM, et al. Early predictors of career achievement in academic medicine. JAMA. 1992;267:1372–1376.
19 Solomon SS, Tom SC, Pichert J, et al. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003;51:149–156.
20 Association of American Medical Colleges. Report IV. Contemporary Issues in Medicine: Basic Science and Clinical Research. Medical School Objectives Project. Available at: (http://www.aamc.org/meded/msop/msop4.pdf
). Accessed July 9, 2007.
21 Zemlo TR, Garrison HH, Partridge NC, et al. The physician–scientist: Career issues and challenges at the year 2000. FASEB J. 2000;14:221–230.
22 Association of American Medical Colleges. Promoting translational and clinical science: The critical role of medical schools and teaching hospitals. Report of the AAMC’s task force II on clinical research. Available at: (www.aamc.org/publications
). Accessed October 25, 2007.
23 Ley TJ, Rosenberg LE. Removing career obstacles for young physician–scientists—loan-repayment programs. N Engl J Med. 2002;346:368–371.