Health services researchers play a crucial role in improving health care by studying how to enhance its organization, delivery, efficiency, and quality. With an aging health services research (HSR) workforce1 and health care reform challenges in the United States, the demand for well-trained HSR researchers can be expected to increase.2,3
To ensure future generations of health services researchers, the Department of Veterans Affairs (VA) Health Services Research and Development (HSR&D) Service, the U.S. National Institutes of Health (NIH), and the U.S. Agency for Healthcare Research and Quality (AHRQ) support career development programs that provide salary support, training, and mentoring. Given substantial investment in these programs and the future need for well-trained researchers, an evaluation of the academic advancement and research productivity of their awardees is warranted and timely.
Prior evaluations of academic/research career development programs
Most prior evaluations of career development programs have been single-group studies that documented the retention of award recipients in research or academia, their productivity (publications, grants), and/or their academic advancement/retention.4–9 Others have compared the achievements of award recipients with those of unfunded applicants, often without adjusting for baseline differences between groups.10–14 Even an evaluation of the NIH’s Individual Mentored Career Development Awards (CDA) program15 that attempted to minimize preexisting group differences noted that awardees were stronger applicants than nonawardees, accounting for some differences in subsequent productivity. The evaluation of the Burroughs Wellcome Career Awards in Biomedical Sciences Program16 concluded that it was impossible to isolate the causal impact of awards (see also Scriven and Coryn17) because awardees and nonawardees could not be balanced on preapplication characteristics.
This study’s approach
A more interpretable approach is to compare awardee outcomes across several highly regarded research career development programs. Thus, for the current evaluation we chose to compare the accomplishments of HSR&D CDA, NIH, and AHRQ K awardees in HSR. Specifically, we sought to determine whether there were any differences among awardee groups on academic advancement, publications, grants, recognition, and mentoring (see Figure 1, right-hand panel) after adjusting for number of years since award and covariates, including personal characteristics, prior training, and prior productivity (Figure 1, left-hand panel).
We focused on three study populations: fiscal year (FY) 1991 through FY2010 recipients of HSR&D CDAs; FY1991–FY2010 recipients of individual-mentored NIH K awards in HSR; and FY2000–FY2010 recipients of individual-mentored AHRQ K awards (the K award program began in FY2000).
HSR&D CDA recipients.
Applications submitted to the HSR&D CDA program18 are evaluated by a review committee using multiple criteria, including the candidate’s training and research experience; publications; appropriateness of the research and training plans; qualifications of the proposed mentors; relevance to the VA of the planned research; commitment of the applicant to a VA career; significance, merit, and feasibility of the proposed research; and the likely long-term contributions of the candidate to VA. The CDA program has provided salary support and mentoring in early-career awards, midlevel three- to five-year awards, and more advanced awards. Progress reports are reviewed for acceptable productivity annually during the award. An HSR&D midlevel award was considered the “relevant award” for individuals receiving more than one HSR&D CDA because these awards are most comparable to NIH and AHRQ individual-mentored K awards. As of October 1, 2010, 244 surviving individuals had received and accepted one or more CDAs; 219 (90%) provided a curriculum vita (CV).
NIH HSR K awardees.
Three individual-mentored NIH K awards, each lasting three to five years, provide salary support and protected research time: The K01 Mentored Research Scientist Development Award, the K08 Mentored Clinical Scientist Research Career Development Award, and the K23 Mentored Patient-Oriented Research Career Development Award. Entering as search terms the three types of K awards, the time period FY1991–FY2010, and “health services” into the NIH RePORTer19 database yielded records for 758 K award recipients.
Review of the award titles and, in some cases, abstracts (by J.F. with either M.C. or R.C.) indicated that 440 awards were either not on HSR topics (e.g., biomedical or clinical science topics) or not on HSR topics comparable to those which might be conducted by HSR&D CDAs. Thus, we excluded K awardees who focused, for example, on different insurance payment plans (not relevant for Veterans Health Administration [VHA] care), care for children (not provided by VHA), or care for citizens in other countries. We imposed this latter eligibility criterion because we wanted to select K awardees who would be more likely to publish in the same journals and compete for the same grant funding from the same sources (except VA grants) as HSR&D CDAs. Initially, 318 K awardees were identified with “VA-comparable” HSR topics. Because the number of NIH K awardees from FY1991 to FY1999 was small relative to the number of HSR&D CDAs during that period, we selected all early NIH K awardees and randomly selected NIH K awardees between FY2000 and FY2010, resulting in a sample of 226 NIH K awardees, of whom 154 (68%) provided a CV.
AHRQ K awardees.
All AHRQ K awards are for HSR. Most have been clinician-focused K08 awards, with K01 awards for nonclinician researchers only being offered beginning in FY2009. Data on AHRQ K awardees are included in the NIH RePORTER19 and on the AHRQ Web site.20 We (J.F. and R.C.) examined the research topics (titles and abstracts) of 120 AHRQ K01 and K08 awardees from FY2000 through FY2010 and reached consensus that 29 were not focused on VA-comparable HSR topics. Thus, we solicited CVs from 91 AHRQ K awardees, of whom 69 (76%) provided a CV.
Data were derived mainly from CVs obtained from awardees. We contacted awardees by e-mail and, if necessary, by telephone or faxed letter. In eight cases (two HSR&D CDAs, four NIH K awardees, and two AHRQ K awardees), we obtained from awardees only an abbreviated biosketch. Because the percentage of awardees submitting an abbreviated biosketch was small, we use “CV” to encompass all documents. Variables from CVs were coded by project assistants and reviewed and confirmed by the primary investigator (PI) (J.W.F.), co-PI (R.C.C.), and/or data managers (X.B., N.C.M.).
We used the Hirsch21,22h-index (described below) from the Thomson Reuters Web of Knowledge database to determine how frequently awardees’ publications had been cited. Finally, we used the NIH research rankings of the academic institution from which awardees had obtained their highest degree. Because we could not locate NIH research rankings prior to FY1990, for the ranking of the university conferring the highest degree we used the ranking of the degree-granting university for the year the CDA or K award was received. Rankings were based on the NIH support received by the medical school associated with the university.
Variables and measures
Baseline descriptive variables and covariates.
Information on award recipients prior to the year of award included demographics (age at award, assuming a bachelor’s degree at age 22; gender, minority group status, defined as white-Hispanic or nonwhite); highest degree (e.g., MD, PhD); NIH research ranking in the year the award was received for the university conferring the highest degree; number of years since highest degree or completion of a residency to (a) receiving an award and (b) through 2010; and receipt of postdoctoral training (yes/no) prior to the CDA. In addition, we recorded grants (see below) as PI and major (see below) journal articles published prior to the year of the award.
Academic advance ment was indexed by the highest attained tenure-track rank in 2010. We assumed that an academic position was a tenure-track position, unless the position title included a modifier (e.g., “clinical” professor, “research” professor).
Grant-related outcomes were based on research grants of $5,000 or more (if the funding amount was provided; grants without funding amounts indicated were included) from the year of award through 2010. We chose this minimum level to include local pilot funding that may have been helpful in obtaining subsequent larger grants. We calculated the number of grants as PI and the number of grants as PI over $100,000. Funding amounts were not inflation adjusted, and it was often impossible to determine whether indirect costs were included.
For publications, the two primary outcomes were the number of major journal articles and the number of major journal articles as first or sole author. “Major journal articles” were defined as reports, reviews, monographs, essays, etc., and excluded editorials, brief commentaries, letters to the editor, book reviews, etc. In addition, the impact of publications through calendar year 2010 was operationalized by the Hirsch21h-index, defined as total number of publications with at least h citations, with none of the author’s remaining publications having been cited more than h times.21 For example, a researcher with an h-index of 25 would have at least 25 articles, each of which had been cited at least 25 times, but no other article that was cited more than 25 times.
We also gathered information on whether the awardee had received the following two types of recognition: held at least one editor, associate editor, guest editor of a special section or issue, or editorial board position (labeled a “journal editorship position”); or served as a standing member of at least one grant review panel for a large funding entity (e.g., VA, NIH, other federal agencies, Robert Wood Johnson Foundation).
To assess the extent of mentoring provided by awardees, we recorded the number of postdoctoral researchers that awardees had mentored through calendar year 2010. Finally, because a goal of the VA HSR&D CDA program is to retain researchers in the VA, we tracked the number of awardees still affiliated with the VA and the number involved in VA research in 2010. For descriptive purposes, we also coded information on the other academic and research positions held by awardees in 2010.
Analyses and methodological notes
We conducted separate analyses for the HSR&D and NIH awardees from FY1991, and for the samples of HSR&D, NIH, and AHRQ awardees from FY2000 and later. To adjust for preaward differences among the groups of awardees, analyses of covariance were used for continuous outcomes and (multinomial) logistic regressions for dichotomous outcomes, controlling for all of the covariates noted in Figure 1.
In some cases, more than one CDA had been mentored by the same individual (124 senior researchers had been the primary mentors for the 219 HSR&D CDAs returning a CV). Because we only knew the primary mentors of HSR&D CDAs and did not know the possible “clustering” of NIH and AHRQ K awardees within mentors, we did not employ mixed-effects analyses22 to compare the three awardee groups. Ignoring clustering underestimates standard errors, so comparisons in such analyses are more likely to be significant; because most comparisons among awardees were not significant, the impact of ignoring clusters was minimal.
We had minimal missing data (generally less than 5%) across variables. Thus, although the approach has drawbacks with more substantial missing data,23 we substituted the mean for the appropriate group for missing data,24 unless otherwise noted.
Awards received and preaward characteristics
In all, 67 (31%) of the HSR&D CDA awardees participating in the evaluation received more than 1 award, yielding a mean of 1.32 awards. Of the 154 NIH K awards, 84 (49%) were K23s, 41 (26%) were K01s, and 29 (18%) were K08s. Of the 69 AHRQ awards, 57 (83%) were K08s and 12 (17%) were K01s.
Information on the characteristics of awardees is presented in Table 1. The three groups of awardees were 37 to 38 years of age, on average, when they received the awards. Among HSR&D CDAs between FY1991 and FY2010, men were more predominant than among FY1991–2010 NIH awardees; for the awardees from FY2000 or later, majorities of the HSR&D CDAs and AHRQ K awardees were male. Researchers from minority (white-Hispanic or nonwhite) groups were proportionately fewer among all HSR&D CDAs in comparison with all NIH K awardees; this difference held for these two groups among awardees from 2000 and later, with AHRQ K awardees having an intermediate percentage of minority group researchers. Almost two-thirds of all HSR&D CDAs and NIH K awardees had an MD degree. Among the three groups of awardees from 2000 and later, the percentage of HSR&D CDAs with an MD dropped to 56% versus 64% of NIH K awardees and 71% among AHRQ K awardees.
Comparing academic advancement and research productivity
Academic advancement and other positions.
The unadjusted percentage of each group of awardees with tenure-track positions at the end of 2010 is presented in Table 2, along with their covariate-adjusted mean tenure-track academic ranks. No significant group differences emerged on this outcome, with all groups having an adjusted average academic rank around assistant professor.
We also present in Table 2 descriptive information on other types of positions held by the awardees in 2010. In all, 75% of HSR&D CDA awardees, 81% of all NIH K awardees, and 67% of the AHRQ awardees held academic positions (some were administrative positions), including those with tenure-track positions noted above. For the HSR&D CDAs, 80% of awardees had VA positions, with 74% holding VA research positions in 2010 (we included those with active CDAs in these percentages). Overall, 93% of HSR&D CDA awardees, 97% of the NIH K awardees, and 100% of the AHRQ K awardees had research positions in 2010, again counting those with active awards as holding research positions.
Grants as PI.
The 219 HSR&D CDA awardees had been PI on 1,147 grants of $5,000 or more, the 154 NIH K awardees had been PI on 564 grants, and the 69 AHRQ K awardees had been PI on 146 grants. For those grants with funding amounts reported, the number of grants of $100,000 or more was 585 for the full sample of HSR&D CDA awardees, 226 for the NIH K awardees, and 57 for the AHRQ K awardees. No significant differences emerged in the adjusted number of grants as PI, either between all HSR&D and NIH awardees or among the three groups of awardees since 2000 (see Table 3). However, for those reporting funding amounts, the HSR&D CDA awardees, on average, had been PI on significantly more grants of $100,000 or more than the NIH K awardees.
Major journal articles.
The 219 HSR&D CDA awardees had published 9,350 major journal articles, 2,480 as first/sole author; the 154 NIH K awardees published 4,234 major journal articles, 1,497 as first/sole author; and the 69 AHRQ K awardees published 1,271 major journal articles, 527 as first/sole author. Relative to NIH K awardees, HSR&D CDA awardees had published a significantly higher mean number of major journal articles (see Table 3); among the three groups of awardees since FY2000, however, differences were not significant. Likewise, we found no significant differences in any comparisons in the adjusted mean number of major journal articles as first/sole author or h-indices.
Recognition and mentoring.
No significant differences emerged in any comparisons in either the likelihood of having had a journal editorship position or a grant review committee membership, or in any comparisons on the adjusted mean number of mentees (data not reported).
In contrast to prior evaluations of research and faculty career development programs that have compared successful versus unsuccessful applicants, we compared the career advancement and productivity of awardees of three prominent HSR career development programs. We believe that this approach has a significant advantage over prior approaches in that the three programs are likely to draw from comparable applicant pools and have similar awardee selection criteria.
Characteristics of awardees
At the time of the award, HSR&D CDAs and NIH and AHRQ K awardees typically were between ages 37 and 38, were six to seven years beyond receipt of their doctoral degrees or completion of their residencies, and were 5.4 to 6.9 years into their research careers (postresidency for MDs and postdoctoral degree for others).
Although 43% of all the HSR&D CDA awardees were women, more women (54%) were included in the full NIH K awardee sample. That gender difference held in the samples of 2000 and later awardees, although the percentage of female CDA awardees rose to 47%, with the AHRQ K awardees having a similar percentage. A total of 14% of the HSR&D CDA awardees were from minority racial/ethnic groups versus 24% for the full sample of NIH K awardees. This difference held in the 2000 and later samples of awardees, with the AHRQ group also having a higher percentage of minority group researchers than did the HSR&D CDA sample. These differences in part may reflect targeted NIH K awards for minority applicants. Intervention at earlier stages in potential applicants’ careers (e.g., during medical or graduate training) likely will be necessary to achieve the sought-after goal of substantial diversity.25
Outcomes for awardees from the three programs
Overall, our evaluation found similar outcomes for the VA HSR&D, NIH, and AHRQ career development awardees in academic rank, grants as PI, publication of major journal articles, citation of publications, serving in journal editorship positions and research review panels for large funding entities, and in subsequent mentoring. The exceptions to these no-difference findings were that HSR&D CDA awardees from FY1991–FY2010 who reported grant funding amounts had been PI on significantly more grants of $100,000 or more than NIH K awardees who reported grant funding in their CVs and had a significantly higher adjusted mean number of major journal articles than did the full sample of NIH K awardees. These significant differences may reflect chance findings arising from multiple significance tests, not taking into account clustering of HSR&D CDAs within mentors, and/or the multiple CDAs received by some HSR&D CDA awardees (we did not track multiple K awards).
This evaluation provides important findings but is not without limitations. Although the productivity of all groups is likely overestimated by using data from only those who returned CVs, the overestimation should be less for the HSR&D CDA awardees whose participation rate was higher. On the other hand, average time since award was greater for HSR&D CDA awardees than for NIH K awardees, which may have allowed them to exhibit greater productivity, even after linear control for years since award. Another limitation is that, for the comparisons of outcomes for HSR&D, NIH, and AHRQ awardees, although nine covariates were controlled, omitted variables may have biased the comparisons.
Our findings indicate that the VA HSR&D CDA, NIH K award, and AHRQ K award programs are effectively selecting and mentoring promising applicants, as indicated by their advancing in academic positions, securing grants, publishing, garnering recognition, and mentoring young investigators. Expansion of these programs will ensure additional HSR capacity to improve the quality and delivery of high-value care in the present challenging era of health care reform.
Acknowledgments: The awardees who participated in the evaluation, as well as Brandy Smith, Susan Macus, Robert Small, and Kirsten Unger-Hu, who made important contributions to the project, are gratefully acknowledged.
1. McGinnis S, Moore J.. The health services research workforce: Current stock. Health Serv Res. 2009;44:2214–2226
2. Coalition for Health Services Research. . A report from the Coalition for Health Services Research: “Health reform: What it means for health services research.” Health Serv Res. 2010;45:ix–xxi
3. Thornton C, Brown JD.. Employer demand for health services researchers in the year 2020. Health Serv Res. 2009;44:2242–2254
4. Escobar-Alvarez SN, Myers ER.. The Doris Duke Clinical Scientist Development Award: Implications for early-career physician scientists. Acad Med. 2013;88:1740–1746
5. Rings EH, Escher JC, Büller HA, Heymans HS.. 20 years of scientific training of Dutch medical students in an American academic division for pediatric gastroenterology and nutrition: Impact on career development. J Pediatr Gastroenterol Nutr. 2008;46:419–422
6. Steiner JF, Lanphear BP, Curtis P, Vu KO.. The training and career paths of fellows in the National Research Service Award (NRSA) program for research in primary medical care. Acad Med. 2002;77:712–718
7. Thompson RW, Schucker B, Kent KC, et al. Reviving the vascular surgeon–scientist: An interim assessment of the jointly sponsored Lifeline Foundation/National Heart, Lung, and Blood Institute William J. von Liebig mentored clinical scientist development (K08) program. J Vasc Surg. 2007;45(suppl A):A2–A7
8. Todd RF 3rd, Salata RA, Klotman ME, et al. Career outcomes of the graduates of the American Board of Internal Medicine research pathway, 1995–2007. Acad Med. 2013;88:1747–1753
9. Tsai AC, Ordóñez AE, Reus VI, Mathews CA.. Eleven-year outcomes from an integrated residency program to train research psychiatrists. Acad Med. 2013;88:983–988
10. Fang D, Meyer RE.. Effect of two Howard Hughes Medical Institute research training programs for medical students on the likelihood of pursuing research careers. Acad Med. 2003;78:1271–1280
11. Lichtman MA, Oakes D.. The productivity and impact of the Leukemia & Lymphoma Society scholar program: The apparent positive effect of peer review. Blood Cells Mol Dis. 2001;27:1020–1027
12. Mahoney MC, Verma P, Morantz S.. Research productivity among recipients of AAFP foundation grants. Ann Fam Med. 2007;5:143–145
13. Mavis B, Katz M.. Evaluation of a program supporting scholarly productivity for new investigators. Acad Med. 2003;78:757–765
14. Smith WH, Rogers JG, Hansen TN, Smith CV.. Early career development in academic pediatrics of participants in the APS-SPR medical student research program. Pediatr Res. 2009;65:474–477
15. Discovery Logic, Inc. National Institutes of Health individual mentored career development awards program. August, 29, 2011. http://grants.nih.gov/training/K_Awards_Evaluation_FinalReport_20110901.pdf
. Accessed August 30, 2015.
16. Pion GM, Cordray DS.. The Burroughs Wellcome Career Award in the Biomedical Sciences: Challenges to and prospects for estimating the causal effects of career development programs. Eval Health Prof. 2008;31:335–369
17. Scriven M, Coryn CLS.. The logic of research evaluation New Dir Eval. 2008;118:89–105
18. U.S. Department of Veterans Affairs. Health Services Research and Development. VA HSR&D research career development program. http://www.hsrd.research.va.gov/cdp/default.cfm
. Accessed September 1, 2015
19. National Institutes of Health. RePORTER (Research Portfolio Online Reporting Tool). http://projectreporter.nih.gov
. Accessed September 1, 2015
20. U.S. Department of Health and Human Services. Agency for Healthcare Research and Quality. Career development award grants archive. http://archive.ahrq.gov/fund/training/kawarch.htm
. Accessed September 1, 2015
21. Hirsch JE.. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci USA. 2005;102:16569–16572
22. Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 20022nd ed Thousand Oaks, Calif Sage Publications
23. Schafer JL, Graham JW.. Missing data: Our view of the state of the art. Psychol Methods. 2002;7:147–177
24. Rubin LH, Witkiewitz K, Andre JS, Reilly S.. Methods for handling missing data in the behavioral neurosciences: Don’t throw the baby rat out with the bath water. J Undergrad Neurosci Educ. 2007;5:A71–A77
25. Halvorson M, Maisel N, Amundson EO, et al. Improving the VHA Health Services Research and Development Career Development Award Program: Interviews With Key Stakeholders. June 2014 Menlo Park, Calif VA Palo Alto Health Care System, Center for Innovation to Implementation