Goldhamer, Mary Ellen J. MD, MPH; Cohen, Amy P.; Bates, David W. MD, MSc; Cook, E Francis ScD; Davis, Roger B. ScD; Singer, Daniel E. MD; Simon, Steven R. MD, MPH
Concerns about the declining numbers of physicians conducting clinical research and their higher rates of attrition and failure to achieve grant funding, compared with nonphysician investigators, have led the Association of American Medical Colleges (AAMC) to identify physician–scientists as a vulnerable population.1 Multiple factors have contributed to the declining population of physician clinical investigators, including debt acquired during medical training, long training periods required for research careers, uncertain prospects for success, and the challenge of excelling in multiple fields.2–8 These factors combine with difficulty obtaining grant funding, lack of protected research time, and more lucrative clinical opportunities, deterring physicians from pursuing careers in clinical research1,9–11 and causing them to abandon research careers even after achieving early success.3,12
Background: The Harvard Program in Clinical Effectiveness
Several paths can lead to a research career for the physician–scientist. For example, medical students can pursue an MD/PhD combined degree through the Medical Scientist Training Program (MSTP) sponsored by the National Institutes of Health (NIH),2,9 or they can seek a one-year research-intensive program through the Howard Hughes Medical Institute (HHMI)13; both programs emphasize basic science research.
A variety of approaches have emerged to address the impediments facing physicians who pursue careers in clinical research.11 One strategy has involved the temporal and programmatic linkage of research training with clinical training, aiming to provide physician researchers early in their careers with the knowledge, skills, and experience necessary to compete with their nonphysician peers.1 The Harvard Program in Clinical Effectiveness (PCE) began in 1986 as a novel effort to train physicians to conduct clinical research, with a particular emphasis on clinical trials, clinical epidemiology, and health services research.14–16 During the past two decades, more than 1,400 physicians have completed the PCE, but their success as clinical researchers has not been systematically evaluated.
Since its inception in 1986, the PCE has provided research training concurrent with physicians’ clinical training. The program began in 1986 with three physician–students and now enrolls more than 150 participants each summer. There have been a total of 1,489 graduates during the program’s first 20 years (Figure 1). The PCE is a daily 6.5-week intensive summer program designed for physicians seeking rigorous training in clinical research.14,15 Housed in the Harvard School of Public Health (HSPH), the program admits generalist and specialist physicians from the United States and abroad, typically in the midst of completing their fellowship training. All students complete core courses in epidemiology and biostatistics and submit a grant proposal as a final epidemiology project. Elective courses include advanced biostatistics and epidemiology, health policy, quality improvement, and decision sciences. All students complete 15 graduate credits and may apply to earn an advanced degree (MSc or MPH) by completing additional courses part-time for an additional 25 credits during two academic years, or full-time during two subsequent summers. The core biostatistics and clinical epidemiology curricula are also used by the more translationally oriented Harvard K30 Program that also leads to a master’s degree (http://grants2.nih.gov/training/k30.htm). Full HSPH per-credit tuition is required; for the vast majority of PCE enrollees, tuition is provided by sponsors, such as fellowship program funds.
We undertook the present study to evaluate PCE graduates’ academic achievements and to identify participant and program features that predict success in obtaining NIH grant funding. Because obtaining federal funding is often considered the sine qua non of a successful research career, this achievement was used as the major outcome in our study. We hypothesized that participants who completed an advanced degree in research methods, who attended the PCE concurrently with their clinical training (e.g., residency/fellowship), and who ultimately conducted and published independent projects originally submitted to satisfy course requirements would be more likely to obtain NIH funding.
Survey development and administration
The 20th-anniversary PCE 48-item survey was developed by the coauthors of this paper, most of whom are either professors in the PCE program or are program graduates. The survey was pretested and revised in the setting of “works in progress” research meetings during a six-month period. The survey included the following content areas: demographic characteristics, including year of PCE attendance, year of medical school graduation, age by decade during PCE, gender, ethnicity, and country of citizenship during program enrollment; academic rank at the time of program enrollment; tuition sponsorship; motivation for enrolling in the PCE; modification of career goals postprogram, specifically, decisions to pursue an advanced research degree, negotiate protected research time, or pursue federal and nonfederal grant funding; program influence on connection with mentors and/or potential collaborators; core epidemiology, biostatistics, and elective course experience, specifically if submitted course projects were subsequently conducted and published; mentoring during the PCE; postprogram career experiences, including current work environment, academic appointments, specialty training, and administrative roles; current percentage of time devoted to patient care, research, teaching, and administration; type of research pursued postprogram; federal and nonfederal grant awards; number and types of publications; and past and present mentoring experiences. Survey questions were formatted as either multiple choice, check all that apply, five-point Likert scale, or open ended. Where possible, questions offered an “other” answer choice, with the option for an open-ended answer.
We used program records and other sources (Google Scholar, PubMed, and the AMA Physician Masterfile) to obtain current contact information for program graduates. The survey was sent by e-mail (Zoomerang) to 1,189 graduates and by post to 300 graduates without known e-mail addresses in 2006. All respondents were automatically enrolled in a lottery to win 1 of 20 gift certificates worth $50.00 from an online bookstore. All nonrespondents in the United States and Canada received two follow-up phone calls to encourage participation. Nonrespondents in other countries did not receive follow-up phone calls.
The Human Studies Committee of Harvard Pilgrim Health Care determined that this study was exempt from human subjects protection regulations per 46 CFR 101 (b) (2). The PCE funded the survey deployment and research assistants who located graduates’ contact information and verified reported federal grants on the NIH Computer Retrieval of Information on Scientific Projects (CRISP) Web site (http://crisp.cit.nih.gov).
Statistical analyses and data verification
Success obtaining grant support was a primary outcome of the study. When respondents reported having served as principal investigator on a U.S. federal grant, we searched the NIH CRISP Web site and recorded the date of disbursement of the first federal grant. The distribution of verified NIH grant awards by category were as follows: K23, 32; R01, 31; K08, 29; F32, 12; K07, 12; R03, 7; R21, 6; M01, 5; P01, 3; U01, 3; D43, 2; G13, 2; K01, 2; R25, 2; U18, 2; and one each of K16, K22, K24, N01, P20, P60, R13, R44, and T32. Because of different follow-up times, we used Cox proportional hazards regression to examine characteristics of students and features of the PCE associated with successful NIH grant funding. Three hundred forty-two respondents were excluded from this analysis for the following reasons: 229 were not U.S. citizens, 63 with self-reported federal grant awards were not found on CRISP, 39 had already received a federal grant award before enrolling in the PCE, and 7 were not a resident, fellow, or faculty member when enrolled in the PCE, and 4 did not provide the year of PCE attendance. Variables assessed included age at program enrollment (by decade); ethnicity, gender, and academic status at enrollment (resident, fellow, or faculty status); generalist or specialist; tuition sponsor; pursuit of an advanced research degree (master of science, master of public health, PhD or equivalent); and publication of independent projects developed in the epidemiology and elective courses. All variables were examined using standard univariate analysis. On the basis of these results and the published literature, all variables were included in the omnibus Cox multivariable model. Time of entry into the study was defined as the summer of PCE attendance. The outcome was first federal grant, verified on CRISP. Time to first federal grant was defined as the difference between the time of entry and the disbursement date of the first federal grant. Follow-up ended at the date of survey deployment. Proportional hazards assumptions were checked using time-varying covariates, and violations were not found. SAS 9.1 statistical software (SAS Institute Inc., Cary, North Carolina) was used to perform analyses.
Among 1,489 graduates of the program, we identified valid e-mail or postal addresses for a total of 1,365 (92%). A total of 994 surveys were completed, resulting in a response rate of 73% of those with identifiable addresses. The demographic characteristics of the study population are shown in Table 1. The survey respondents included 401 women (40%), 299 nonwhites (30%), 229 non-U.S. citizens (23%), 388 generalists (39%), and 606 specialists (61%). Respondents spanned five decades of age and all academic ranks.
Reasons for program enrollment, program experience, and immediate program outcomes
When asked why they matriculated in the PCE, 656 respondents (66%) indicated that research training was needed to advance in their chosen career path and that the PCE provided the opportunity to explore research training methods. A total of 239 respondents (24%) said that they enrolled in the program because it was an expected or required part of their clinical fellowship program.
The modified grant proposal that served as the final paper for the clinical epidemiology course led to an actual research project in part or full by 646 respondents (65%); a total of 413 (64%) of the completed projects were published, and 278 projects (43%) ultimately received grant funding. Recently, several elective courses in the PCE (e.g., survival analysis, meta-analysis, research with large databases) have also required students to submit final course projects where course concepts are directly applied to real-world research; 298 graduates (30%) reported publishing papers based on these independent elective projects, and 139 respondents (14%) reported successful grant funding of these projects. In addition, 527 respondents (53%) reported that projects submitted in epidemiology and/or elective courses were part of a mentored research experience. Although formal mentoring is not an explicit objective of the program, 497 respondents (50%) reported that they found a new mentor or collaborator influential to their career and research projects while enrolled in the PCE.
Long-term program outcomes
When asked to cite the program’s impact on individual career goals and objectives, 577 respondents (58%) said that they continued with their research career as planned prior to participating in the PCE. A total of 348 respondents (35%) decided to pursue a degree (MS, MPH, PhD, or equivalent) that was not planned prior to the program; 10 respondents (1%) decided against a research career; and 179 respondents (18%) either stated that they were unsure whether PCE had affected their career plans or that the PCE did not alter their career plans. Overall, 755 survey respondents (76%) had either completed or were pursuing an advanced research degree. Specifically, 517 graduates (52%) pursued a MPH degree, 239 (24%) pursued an MS, and 30 respondents (3%) pursued doctoral training (PhD, ScD, DrPH, or equivalent).
Six hundred eighty-six program graduates (69%) indicated that the PCE had been important in successfully obtaining grant support. Yet, 199 respondents (20%) noted that writing grants was not applicable to their current work. Half of respondents, or 507 graduates (51%), reported that they had sought protected time for research, and 380 (75%) of these respondents were successful in obtaining it. Three hundred twenty-eight PCE survey respondents attended the program as faculty; 52 faculty respondents (16%) reported they were able to participate in more research projects, and 199 faculty respondents (61%) were able to negotiate a reduction in clinical time into their schedules. The following categories of research were pursued by survey respondents post-PCE: 537 respondents (54%) pursued epidemiology, 437 (44%) pursued clinical trials, 408 respondents (41%) pursued health services research, 149 (15%) pursued translational research, 119 (12%) pursued medical education, 109 (11%) pursued qualitative research, 89 (9%) pursued medical informatics (9%), and 40 (4%) pursued basic science research.
When asked to describe their current professional position, respondents reported the following: 318 (32%) were currently researchers in academic medicine, 20 (2%) were researchers in industry, 388 (39%) were working as clinician educators/teachers, 50 (5%) were administrators at an academic institution, and 50 (5%) were working in private practice. In addition, 89 (9%) indicated that they were currently in training (medical student, resident, or fellow).
A considerable number of graduates reported having served in various leadership roles during their careers since the PCE. A total of 31 respondents (3%) had chaired a department, and 101 (10%) had served as chief of a division. Program graduates included 236 (24%) residency program directors, 193 (19%) medical directors, 19 (2%) chief executive officers or chief medical officers, and 6 (0.6%) deans. A total of 220 (22%) reported that they had held other administrative positions.
Correlates of successful NIH funding
Overall, 338 respondents (34%) reported receiving federal grant funding, and 626 respondents (63%) reported receiving nonfederal grant funding. After excluding ineligible subjects, 652 respondents remained eligible for analysis, of which 156 respondents (24%) had a verified NIH grant. The characteristics of this subgroup are shown in Table 2. Respondents represented more than 20 specialties and subspecialties, with the majority of graduates trained in general internal medicine, general pediatrics, cardiology, and hematology–oncology (Figure 2). We explored correlates of success in obtaining NIH grant support. The hazard ratio (HR) point estimates were similar in the univariable and multivariable analyses, but the effect of earning an advanced degree was no longer statistically significant in the multivariable model (Table 2). Multivariable Cox proportional hazards regression (Table 2) found that age less than 40 years at program enrollment (HR 1.87, CI 1.03, 3.41), generalist status (HR 1.57, CI 1.14, 2.16), and publishing coursework from either epidemiology or electives (HR 1.65, CI 1.19, 2.31) were significantly correlated with NIH grant funding. Gender, academic status at enrollment, ethnicity, tuition sponsorship (i.e., self versus institutional pay), and earning an advanced degree were not significantly associated with success in obtaining NIH grant funding.
The AAMC has identified physician–scientists as a vulnerable population,1 and others have questioned whether physician clinical investigators are an “endangered species.”9,10,12,17–25
The Harvard PCE seeks to provide needed core skills in research methodologies broadly applicable to clinical research and experience in generating research proposals. In general, graduates of the PCE report substantial success pursuing academic research careers. Many have been able to successfully negotiate protected research time, publish research work, obtain funding, gain promotions, and rise to leadership roles in public health, academic medicine, and the private sector.
Because obtaining federal funding is an important benchmark of a successful research career, we sought to identify correlates of receiving federal research funding among a subset of program graduates. We found that physicians who were age 40 years or younger at the time of program enrollment were awarded federal grant funding at nearly twice the rate of older attendees. Whereas the AAMC recommends targeting medical students, residents, and fellows for clinical research training in patient-oriented research,1 our study suggests that age at time of training, rather than academic status, may be particularly important for research and federal funding success. It may be that older participants have more distracting personal and professional responsibilities that compete with research endeavors. Our findings regarding age at PCE enrollment should be viewed as exploratory, and we emphasize that older enrollees also achieved funding success.
Future studies might include a non-PCE control group to better define the role that age may play in federal funding success. We considered surveying applicants who did not matriculate into the PCE, but comprehensive records of such individuals were not available. Additional research is needed to better define the personal and professional environments that are conducive to a successful academic research career. The finding that generalists are more likely than specialists to obtain NIH grant funding is somewhat counterintuitive. Generalists enrolled in the PCE may be more committed to a research career because many participate primarily as trainees in fellowship programs dedicated solely to research. In contrast, many specialist trainees enroll in the PCE during a research year(s) that is a component of a clinical fellowship. All PCE participants plan to conduct research; however, participants’ individual dedication to research varies and was not measured by our survey. Our observed effect may also reflect the difference in earning potential among generalists and specialists. The financial advantages of a clinical career rather than a research career may be less pronounced for a generalist physician.
Whereas publishing independent coursework in epidemiology and electives may be a marker of a highly motivated individual, the positive association between publishing PCE research projects and NIH grant funding suggests that the opportunity to pursue real-world practical projects during clinical research training, coupled with close mentoring by professors and advisors, serves as a springboard to independent investigation.
A review of the literature identifies two distinct paths to a research career from which the future physician–investigator typically may choose. The first path is available to medical students who pursue either an MD/PhD combined degree through the NIH-sponsored MSTP2–9 or a research-intensive experience for one year through the HHMI.13 Several medical schools offer variations on these programs,7,26–30 and their evaluations and success have been well documented.5,31 The second path is taken by physicians who decide during residency or fellowship training, or later, to pursue a career in research; these physician–investigators have been labeled “late bloomers.”2,6,32 They traditionally graduate from medical school and complete their residency, followed by a subspecialty fellowship consisting of one to two years of clinical training, followed by two to six years of training in research. Physician–investigators on the MSTP/HHMI path primarily pursue either basic science or disease-oriented research,2,9 whereas late bloomers traditionally pursue clinical research, including clinical trials, epidemiology, translational, and health services research.6 The NIH-sponsored National Research Service Awards granted to late bloomer generalists seeking advanced training in research methods have been extensively studied and evaluated.33–35 There is considerable concern that there are more physicians being trained who will ultimately conduct basic science rather than patient-oriented research, losing the potential for translational research with direct applications to individuals and populations.4,18,36
Training physicians in programs such as the PCE represents one solution that may increase the numbers of physician–investigators trained in patient-oriented research. Twenty-four percent of survey respondents from the United States were verified on the CRISP Web site as principal investigators on an NIH grant. This finding supports the assertion that the PCE is effective in training this critical late bloomer subset of physician–investigators who primarily conduct patient-oriented research, including epidemiology, clinical trials, translational, and health services research. The program currently trains more than 150 physicians per summer. Programs such as the PCE have the potential to narrow the gap between the numbers of physician–scientists trained to pursue basic science and those who pursue patient-oriented research.26
Although this study identified physician age and generalist status as correlates of NIH grant funding, there are many factors necessary to produce a successful independent physician–investigator.37 The results of our study may be confounded by unmeasured variables; for example, program participants may have had a variety of career aspirations, and not all attendees may have planned to pursue a career in independent investigation. Further research is needed to determine whether our results are generalizable to other training programs. We did not have a means to measure self-motivation or dedication to long-term research, but we recognize that it is likely that these characteristics are important determinants of success. Choosing a non-PCE control group would have been ideal but was beyond the scope of the study, and it would be important in future studies. In addition, our study was not designed to look for a possible cohort effect on receiving NIH research grants through the 20-year span of the enrollment period; this would be important to consider in future work. Our study was limited by the fact that the CRISP database does not list coprincipal investigators of NIH grants or program projects, or principal investigators on Veterans Affairs awards, which accounted for 38% of unverified grants. This limitation has been recognized, and the AAMC has recommended the modification of the CRISP database to include coinvestigators of funded grants.1 In addition, CRISP does not log grants from other non-NIH federal sources (e.g., the Department of Defense) or equivalent research or research career development awards from foundation or other nonfederal sources.
The PCE provides rigorous training in statistical methods, epidemiology, and health policy that positions graduates to conduct patient-oriented research and compete successfully for NIH grant funding. The development of short, intensive programs in clinical research methods, such as the PCE, that target clinical research training to late bloomers may provide a solution to the problem of declining physician–scientists dedicated to patient-oriented research.26,38 This vital group of clinically trained physician–scientists can help lead the nation’s patient-oriented research mission, translating knowledge learned from basic science into direct applications that improve the health of individuals and populations.9,17,18,39 Further research is needed to determine whether programs such as the PCE, offered to medical students, could aid in reversing the trend of declining numbers of physician–investigators who conduct patient-oriented research, or whether knowledge that comes only with years of clinical training is first needed to benefit from this kind of training experience.
The authors wish to thank lead research assistant Lynette English and supporting assistants Matthew Raber, Alexander T. Shaw, and Yamylex Belis. A special thanks to Dr. John Steiner, University of Colorado, for sharing the instrument used to survey all National Research Service Awardees, and to Dr. Maureen Connelly, Harvard Medical School (HMS), for sharing the instrument used to survey HMS faculty on career satisfaction.
Dr. Goldhamer was supported on an institutional National Research Service Award 5 T32HP11001-19. The Program in Clinical Effectiveness provided funds for programming the Internet survey and research assistants who assisted in locating all graduates’ addresses and in verifying NIH grants on CRISP.
1 Dickler HB, Korn D, Gabbe SG. Promoting translational and clinical science: The critical role of medical schools and teaching hospitals. PLoS Med. 2006;3:e378.
2 Ley TJ, Rosenberg LE. Removing career obstacles for young physician–scientists—loan-repayment programs. N Engl J Med. 2002;346:368–372.
3 Cech TR, Egan LW, Doyle C, et al. The biomedical research bottleneck. Science. 2001;293:573.
4 Pober JS, Neuhauser CS, Pober JM. Obstacles facing translational research in academic medical centers. FASEB J. 2001;15:2303–2313.
5 Jeste DV, Halpain MC, Trinidad GI, Reichstadt JL, Lebowitz BD. UCSD’s short-term research training programs for trainees at different levels of career development. Acad Psychiatry. 2007;31:160–167.
6 Ley TJ, Rosenberg LE. The physician–scientist career pipeline in 2005: Build it, and they will come. JAMA. 2005;294:1343–1351.
7 Nathan DG, Wilson JD. Clinical research and the NIH—a report card. N Engl J Med. 2003;349:1860–1865.
8 Schrier RW. Ensuring the survival of the clinician–scientist. Acad Med. 1997;72:589–594.
9 Zemlo TR, Garrison HH, Partridge NC, Ley TJ. The physician–scientist: Career issues and challenges at the year 2000. FASEB J. 2000;14:221–230.
10 Faxon DP. The chain of scientific discovery: The critical role of the physician–scientist. Circulation. 2002;105:1857–1860.
11 Hollander JE, Singer AJ. An innovative strategy for conducting clinical research: The academic associate program. Acad Emerg Med. 2002;9:134–137.
12 Dickler HB, Fang D, Heinig SJ, Johnson E, Korn D. New physician–investigators receiving National Institutes of Health research project grants: A historical perspective on the “endangered species.” JAMA. 2007;297:2496–2501.
13 Fang D, Meyer RE. Effect of two Howard Hughes Medical Institute research training programs for medical students on the likelihood of pursuing research careers. Acad Med. 2003;78:1271–1280.
14 Goldman L, Cook EF, Orav J, et al. Research training in clinical effectiveness: Replacing “in my experience …” with rigorous clinical investigation. Clin Res. 1990;38:686–693.
15 Hiatt H, Goldman L. Making medicine more scientific. Nature. 1994;371:100.
16 Simon SR, Shaneyfelt TM, Collins MM, Cook EF, Fletcher RH. Faculty training in general internal medicine: A survey of graduates from a research-intensive fellowship program. Acad Med. 1999;74:1253–1255.
17 Hirsch J. An association for patient-oriented research. Ann Intern Med. 1999;130:1014–1017.
18 Nathan DG. Educational-debt relief for clinical investigators—A vote of confidence. N Engl J Med. 2002;346:372–374.
19 Neilson EG. The role of medical school admissions committees in the decline of physician–scientists. J Clin Invest. 2003;111:765–767.
20 Rosenberg L. Physician–scientists—Endangered and essential. Science. 1999;283:331–332.
21 Rosenberg LE. Young physician–scientists: Internal medicine’s challenge. Ann Intern Med. 2000;133:831–832.
22 Schechter AN. The crisis in clinical research: Endangering the half-century National Institutes of Health consensus. JAMA. 1998;280:1440–1442.
23 Thompson JN, Moskowitz J. Preventing the extinction of the clinical research ecosystem. JAMA. 1997;278:241–245.
24 Wyngaarden JB. The clinical investigator as an endangered species. N Engl J Med. 1979;301:1254–1259.
25 Zucker S, Crabbe JC, Cooper Gt, et al. Veterans Administration support for medical research: Opinions of the endangered species of physician–scientists. FASEB J. 2004;18:1481–1486.
26 Moskowitz J, Thompson JN. Enhancing the clinical research pipeline: Training approaches for a new century. Acad Med. 2001;76:307–315.
27 Schor NF, Troen P, Kanter SL, Levine AS. The Scholarly Project Initiative: Introducing scholarship in medicine through a longitudinal, mentored curricular program. Acad Med. 2005;80:824–831.
28 Zier K, Stagnaro-Green A. A multifaceted program to encourage medical students’ research. Acad Med. 2001;76:743–747.
29 Abelmann WH, Nave BD, Wilkerson L. Generation of physician–scientists manpower: A follow-up study of the first 294 graduates of the Harvard–MIT Program of Health Sciences and Technology. J Investig Med. 1997;45:272–275.
30 Mark AL, Kelch RP. Clinician scientist training program: A proposal for training medical students in clinical research. J Investig Med. 2001;49:486–490.
31 Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician–scientists. J Investig Med. 2003;51:149–156.
32 Whitcomb ME, Walter DL. Research training in six selected internal medicine fellowship programs. Ann Intern Med. 2000;133:800–807.
33 Steiner JF, Curtis P, Lanphear BP, Vu KO, Main DS. Assessing the role of influential mentors in the research development of primary care fellows. Acad Med. 2004;79:865–872.
34 Steiner JF, Curtis P, Lanphear BP, Vu KO, Reid A. Program directors’ perspectives on federally funded fellowship training in primary care research. Acad Med. 2000;75:74–80.
35 Steiner JF, Lanphear BP, Curtis P, Vu KO. The training and career paths of fellows in the National Research Service Award (NRSA) Program for Research in Primary Medical Care. Acad Med. 2002;77:712–718.
36 Nathan DG. Careers in translational clinical research—Historical perspectives, future challenges. JAMA. 2002;287:2424–2427.
37 Levey GS, Sherman CR, Gentile NO, Hough LJ, Dial TH, Jolly P. Postdoctoral research training of full-time faculty in academic departments of medicine. Ann Intern Med. 1988;109:414–418.
38 Shine KI. Some imperatives for clinical research. JAMA. 1997;278:245–246.
39 Moses H 3rd, Dorsey ER, Matheson DH, Thier SO. Financial anatomy of biomedical research. JAMA. 2005;294:1333–1342.