Individual Mentored Career Development Awards, or K awards, provide support and protected time for early-career investigators to engage in mentored research and career development, and thus aim to promote a pathway to independence. This study focuses on the National Institutes of Health (NIH) K01, K08, and K23 awards. K01 awards support primarily postdoctoral researchers and early-career faculty with PhD or DVM degrees. K08 and K23 awards support primarily early-career clinician scientists holding MD or MD/PhD degrees and also a small number of clinical scientists with PhD degrees.
We focus on early-career scientists because there is now growing consensus in the scientific community that hypercompetition for research funding may be impeding scientific research and endangering the scientific enterprise.1,2 There is also evidence that this hypercompetition has greater negative impact on those early in their scientific careers.2,3 A recent National Academies of Sciences, Engineering, and Medicine (NASEM) report urges the NIH to support independent research among early-career scientists, including an expansion in the number of Career Development Awards (NASEM, Recommendation 4.2).4 Furthermore, the 21st Century Cures Act establishes the Next Generation of Researchers Initiative, which instructs the NIH to “Develop, modify, or prioritize policies … to promote opportunities for new researchers and earlier research independence.”5
Prior evaluations of the K award programs have indicated positive impact on subsequent research support and publications, with particularly strong impact for clinician scientists.6–8 However, to our knowledge, previous research has not examined the role that K awards may play in different funding environments, when competition for funding may differ, such as during the doubling of the NIH budget (1998–2003) and post doubling (2004–2016). This analysis examines outcomes for K awardees during and after the NIH budget doubled, to assess the role of K awards during different funding environments. It is not clear from prior analyses whether the beneficial effects of K awards extend beyond success in attaining a first R01 equivalent award (R01 Eq.) or Research Project Grant (RPG). We test whether there is an effect of K awards on second R01 Eq. or RPG by comparing the outcomes among those who received a previous major independent award. Our analysis estimates the effect of a K award by using a regression discontinuity design (RDD) with matching, and by controlling for many variables that could affect the outcomes of interest. We also explore whether the different K award types (K01, K08, and K23) may have heterogeneous effects on subsequent NIH funding.
Method
We used data from the NIH’s Information for Management, Planning, and Coordination II (IMPAC II) administrative database, and examined data for K01 and K23 applicants and awardees from fiscal years (FY) 2000 to 2009 and K08 applicants and awardees from 1990 to 2009. We examined the likelihood of subsequent R01 Eq. and RPG awards, and the number of R01 Eq. and RPG awards after having received or applied for a career development (K) award. We included all new competing first R01 Eq. or RPG grants between the FY of the participant’s last K application and FY 2016. To examine the role of K awards on second R01 Eq. and RPGs, we considered all competing second awards after having received a first independent award. Finally, we examined the time (in years) between K award (or last application for K award) and first R01 Eq. and RPG and second RPG.
One would predict that those receiving a K award would be more successful in receiving subsequent NIH grants because K awards provide protected time to conduct research and generate publications and additional research ideas, which should increase the likelihood of a future RPG award or R01 Eq. Figure 1 provides a schematic representation of a basic logic model for our analysis.7
;)
Figure 1: Logic model highlighting K award inputs, activities, and outcomes, from a study of NIH K awards and early-career scientist research trajectories, 1990–2016. Abbreviations: NIH indicates National Institutes of Health; R01 Eq., R01 equivalent award; RPG, Research Project Grant.
Simple comparisons between those researchers who were funded versus unfunded could overestimate the effect of K award on subsequent outcomes because such comparisons cannot distinguish the effect of the award itself from the effect of unobserved characteristics of those funded. Therefore, to estimate an effect that may more closely approximate the causal effect of K awards, we used a RDD with matching and accounted for an array of variables that could affect the likelihood of subsequent independent funding.
The RDD compares applicants below and above a scoring/funding cutoff used by NIH Institutes/Centers (IC) to make funding decisions. The RDD assumes that applicants near the cutoff should be similar in observed and unobserved characteristics. The cutoff can serve as a randomization mechanism where similar applicants are “randomly” assigned to treatment (funded) and control (unfunded) groups. This approach removes individuals with highly dissimilar scores and characteristics from the analysis. Previous literature has found reduction in bias in the effect of “treatment” by combining RDD with matching for particular characteristics.9–11 We therefore combined RDD with coarsened exact matching and matched on gender, race, ethnicity, activity code (K01, K08, K23), degree type, previous NIH support through National Research Service Awards (NRSAs), FY, and IC.12
Our treatment and control groups included matched individuals who are within 100 points of the funding cutoff. There is no predetermined cutoff for funding that can be applied to all applications; rather, the actual cutoff depends on the IC and council round in which an application competes. Like Jacob and Lefgren,13,14 we defined the cutoff () as the score of the last funded applicant in each IC (j), council round, and year (c). The cutoff score is subtracted from each individual applicant’s priority score () to generate a normalized priority score . The normalized score of zero represents the cutoff. Figure 2 shows the probability that a K application was funded as a function of the normalized priority score for researchers within 100 points of the cutoff. Scores range from −100 (competitive and funded) to 100 (less competitive and unfunded). There is an inverse relationship between normalized priority score and the likelihood of K funding. There is also evidence that there is a discontinuity in the likelihood of receiving a K award at the cutoff. The RDD method, which requires a discontinuity at the cutoff to be used, clearly divides individuals into funded and unfunded K participants. Individuals scoring below the cutoff are the “treatment” group receiving a K award, whereas those scoring above the cutoff (“control”) are not funded even though individuals in both groups have otherwise very similar priority scores.
Figure 2: The relationship between normalized priority score and the probability of K award funding, from a study of NIH K awards and early-career scientist research trajectories, 1990–2016.
RDD and matching may still yield a biased estimate of the K award on outcomes. Some imbalance between treatment and control groups remains because there are variables that we did not match on. We further reduced this potential bias by controlling for confounders via regression.15 In the regression we controlled for linear and quadratic normalized priority scores, gender, race, ethnicity, degree type, age, previous NRSA support (pre- or postdoctoral), indicators for years of evaluation, and indicators for NIH IC. We also examined outcomes controlling for linear, quadratic, and cubic priority scores, which did not change the findings. We conducted all analyses using Stata/MP 15 statistical software (StataCorp LLC, College Station, Texas). Because the data consisted of administrative records that awardees consent to the NIH using for evaluating program effectiveness, no ethical approval was required.
Results
Subsequent funding
Table 1 compares unconditional means of outcome variables for those within 100 points in each direction of the funding cutoff. Funded researchers were approximately 1.5 times more likely to receive a first R01 Eq. (37.3% vs. 22.9%) or RPG (47.0% vs. 31.5%) than unfunded researchers. K awardees also had a higher number of R01 Eq. (0.69 vs. 0.33) and RPG awards (1.1 vs. 0.42) and a higher likelihood of second R01 Eq. (17.0% vs. 9.8%) or RPG (26.7% vs. 16.4%). Supplemental Digital Appendices 1 and 2 (available at https://links.lww.com/ACADMED/A619) visually represent the correlation between normalized priority score and first and second R01 Eq. Overall, those funded through a K award received a first R01 Eq. or RPG later, but this delay was reduced for the second R01 Eq. or RPG. Table 1 also compares independent variables of K applicants and awardees for the characteristics we matched on, demonstrating that there were no significant differences in other determinants affecting subsequent funding. Detailed definitions of analysis variables are included in Supplemental Digital Appendix 3, available at https://links.lww.com/ACADMED/A619.
Table 1: Characteristics for Matched Sample of Mentored Career Development Participants in Fiscal Year 1990–2009 and Subsequent National Institutes of Health Grants Activity, From a Study of K Awards and Early-Career Scientist Research Trajectories, 1990–2016
Differences in means in the RDD matched sample could still be biased; thus, we estimated regression-adjusted (conditional) effects. Table 2 first presents estimates from the regression analysis for a combined sample of all three K award types (entire sample), then by stratification of data to assess the effect by K award type. We also summarized effects during the NIH budget doubling and post doubling. Results on the entire sample indicated positive effects of receiving a K award on the likelihood of receiving a first R01 Eq. and first RPG award. K awardees were 24% more likely to receive R01 Eq. or RPG funding and received a larger number of R01 Eq. awards, an effect significant at the 10% level. Combined data for all three K awards demonstrated no evidence that K awards were linked to increased probability of receiving a second R01 Eq. or RPG award.
Table 2: Conditional Regression Discontinuity Design Models of Receiving Subsequent National Institutes of Health Grants Among Mentored Career Development Participants in Fiscal Year 1990–2009, From a Study of K Awards and Early-Career Scientist Research Trajectories, 1990–2016a
To examine whether Career Development Awards exerted a differential effect under different funding environments, we evaluated cohorts that submitted applications during and after the NIH budget doubling. Estimates during the doubling suggest that receiving a K award was associated with an increased likelihood of subsequent R01 Eq. by 29.6% and RPG award by 32.2% as well as an increased number of subsequent awards. However, we found no evidence that K awards improved the likelihood of second independent awards. Our analysis of postdoubling data found similar effects. K awards were linked to increased likelihood of first R01 Eq. and RPG award by 26.6% and 22.8%, respectively, and a higher number of R01 Eq. awards. We did not calculate the impact of K awards on second R01 Eq./RPG post doubling because for some participants insufficient time had elapsed to receive subsequent second awards.
Up to this point our analysis combined data for three different types of K awards, but these combined results do not address potential heterogeneity in the impact of different K awards on subsequent outcomes. A previous evaluation examining data from FY 1990–2005 found that K awards improved subsequent grant activity for K08 and K23 awardees, but K01 awards had less impact on subsequent independent grants.6
Examining K01 awards, we found that K01 awardees were more likely to receive first independent R01 Eq. (by 22.7%) and RPG awards (25.1%). We did not find an effect on the number of independent grants or second independent awards. Data for the NIH budget-doubling period demonstrate a null effect of K01 awards on subsequent grant activity, but estimates covering years after the NIH doubling indicate that K01 awards were associated with improved outcomes among those receiving funding. Those funded through K01 awards post NIH doubling were predicted to be 44% more likely to receive an R01 Eq. and 44.1% more likely to receive an RPG award, and had a higher number of R01 Eq. awards.
Examining outcomes for K08 awardees, we found that those researchers receiving funding were more likely to be funded in subsequent first R01 Eq. and RPG awards. This effect was particularly significant for K08 awardees during the NIH budget doubling, where the K award was associated with an increased probability of a first R01 Eq. by 53.6% and RPG award by 63.6%, and a larger number of R01 Eq. and RPG grants. However, in the postdoubling period, estimates indicate that K08 awards were not correlated with subsequent funding. It is important to note that the sample from which these data were drawn is small, and we did find an effect of K08 awards when we ran a less restrictive model that did not include regression. Therefore, the reduced impact of K08 awards post doubling must be interpreted carefully, and additional data may be required to definitively determine the impact of the K08 awards on subsequent outcomes.
These findings imply particularly large positive effects of K23 awards on subsequent outcomes. Results including data pre and post doubling (all years) indicate that funded applicants were more likely to receive first R01 Eq. by 61.4% and RPG funding by 41.3% and received a larger number of subsequent awards. The effects were larger than for K01 or K08 awards. Importantly, the effects were present during and after NIH budget doubling. During the NIH doubling, a K23 increased the likelihood of R01 Eq. or RPG funding by almost 100%. Post doubling, K23 awardees were also more likely to receive a subsequent R01 Eq. (55.4%) or RPG (31.3%) award. K23 awardees were also more likely to receive a second R01 Eq. or RPG award in the period of the NIH budget doubling.
Where do K awards make the biggest difference? In examining data from all time periods, we found that the relative effect of NIH funding was largest among K23 awardees. Estimates for K01 and K08 awardees demonstrate significant effects of the K01 only post doubling, and the K08 only during the doubling.
Overall, we found no significant effect of K awards on improving the likelihood of second independent awards. This finding is consistent with the idea that the main driver of second R01 Eq. or RPG awards is likely the receipt of independent funding through a first R01 Eq./RPG. Supplemental Digital Appendix 4, available at https://links.lww.com/ACADMED/A619, presents estimates on unmatched data that use RDD and regression. These findings serve as a robustness check to findings from the matched sample. The estimates based on the matched data are the more conservative estimates.
Time to award
From a policy perspective, it is important to establish whether funding through a K award may delay the receipt of an independent research award. We estimated the time to R01 Eq. or RPG award for K awardees as the time in years between the FY they received the K award and the FY they receive subsequent funding. For those not receiving a K award, the time variable is the difference between the FY of their last K application and the FY of their R01 Eq. or RPG award. We report results of unmatched data because sample sizes get very small in the matched sample. We summarize findings in Table 3. Overall, results for the combined sample indicate that receiving a K award was associated with increased time to a first R01 Eq. and a first RPG award by 0.85 and 0.86 years, respectively. The difference ranged between 0.66 years (for K23) and 1.17 years (for K01 awardees). For first awards, these findings imply that among awardees, independent funding was received after more time for K01 awardees than for K08 and K23 awardees.
Table 3: Conditional Regression Discontinuity Design Models of Mentored Career Development Participants in Fiscal Year 1990–2009 Predicting Time to Subsequent Award for Those Researchers With Subsequent Awardsa
The results suggest a likely “catch-up” between K awardees and K applicants in the time to receipt of a second RPG award (Table 3). There were no significant differences in time to second award between awardees and applicants for K01 and K23 awards. To confirm this catch-up observed in the entire sample and study period, we assessed time to second RPG award during the NIH budget-doubling period. If a catch-up in time to second RPG did occur between K awardees and applicants, it should be more apparent during the doubling because a larger group of applicants had received several independent awards, and sufficient time had elapsed to be able to identify this catch-up in our data. The data indicate that during the doubling, there was no difference in time to second RPG award between K awardees and K applicants except for a delay among K08 awardees, which was significant only at the 10% level.
The average estimated time to first R01 Eq. for the NIH budget doubling and postdoubling cohorts were almost identical (0.93 and 0.9 years). This means that while on average K awardees may have experienced a delay in time to receipt of independent funding, there was not an increase in the delay between the doubling and postdoubling period. Comparing different K award types, we observed that for R01 Eq. a delay in the postdoubling period was largest (over a year) for K01 awardees, and smallest for K08 and K23 awardees (under a year).
Discussion
This study found that individual Mentored Career Development Awards (K awards) are linked to a researcher’s enhanced success in competing for first R01 Eq., a critical milestone of successful transition to independence. They are also predictive of receiving RPG awards as well as the number of subsequent R01 Eq. and RPG awards. Our findings have significant policy implications, given concerns about a declining pipeline of early-career investigators.2 The findings indicate that extending funding to candidates with meritorious scores close to but below the funding cutoff may enhance and sustain the careers of early-career biomedical researchers. Increasing the number of K awardees is also consistent with recent recommendations from a NASEM study.4
Our study uncovered differences in the effect between clinician scientists who received K08 and K23 awards, and K01 awardees who were primarily PhDs. We also found that these effects changed during the NIH budget doubling and post doubling. K23 awards had the largest effect overall both during and post doubling. This has important policy implications because patient-oriented research is critical to the NIH mission to improve health. Our estimates suggest that protected time for research and mentored career development may be particularly important for clinicians doing patient-oriented research. The K08 award, which also supports clinician scientists in basic and translational research, was associated with success in subsequent R01 Eq. or RPG awards, but the effect was not as great as for the K23 and was more apparent during the budget-doubling than post doubling period. Although our study did not identify factors driving the reduced effect of K08 awards post doubling, it is important to consider that clinician scientists may face a choice between practicing medicine for a larger income or continuing research in a more competitive funding environment.
K01 awards were highly predictive in promoting transition to independence post doubling, which differed from the data during the NIH budget doubling. This may suggest that K01 awards had greater impact during the period of enhanced competition for independent funding experienced after the NIH doubling. Among those researchers who received a first R01 Eq. or RPG award, we found no effect of K awards on the probability of receiving a second major independent award.
Finally, concern has been expressed about the increase in age at which biomedical researchers achieve their first substantial independent award.16–19 Our findings suggest that a K award may delay a researcher’s transition to independence. However, the fact that the time to second RPG award did not significantly increase for K01 and K23 awardees indicates that an initial delay in time to independence for K awardees does not affect subsequent productivity for a K awardee. The fact that there may still be a delay for K08 awardees to second RPG award warrants further consideration, such as whether clinicians may need improved support on K awards to ensure that they remain in a successful research career.
Our analysis has largely focused on outcomes of K applicants and awardees as related to subsequent NIH funding. Important questions remain regarding to what extent funding through a K award affects publication productivity, and funding through other federal or nonfederal entities. Furthermore, our analysis does not test the relationship between K awards and tenure and promotion. These are limitations of our analysis that future research may want to address.
Overall, our findings provide new evidence that K awards promote success in the careers of both PhDs and physician scientists. Extending K award funding to candidates with meritorious scores right below the cutoff could increase the likelihood that more scientists go on to successful, independent careers. In FY 2017, the NIH supported 2,604 individuals and spent a total of $400.2 million on competing and noncompeting K01, K08, and K23 awards, which represents a small fraction (< 1.1%) of the NIH budget. Our findings indicate that even a small increase in investment to support more K awardees could provide a significant return by strengthening the biomedical and physician scientist pipeline.
Acknowledgments: The authors wish to thank the anonymous referees, the editor of Academic Medicine, and the editorial office for reviewing and assisting in the preparation of the manuscript. The authors thank Jennifer Sutton, Shoshana Kahana, Lisa Evans, Henry Khachaturian, Walter T. Schaffer, Marguerite Matthews, and Kristen Kirkham within the National Institutes of Health (NIH) Division of Biomedical Research Workforce; participants of the Data Policy Seminar in the Office of the Director/Office of Extramural Research; the Evaluation Training Advisory Committee; and participants of the 2017 American Evaluation Association. The authors would like to thank Rachael Walsh, Lindsey Scott, Natalie Graham, Katie Patel, Robert Moore, Deepshikha Roychowdhury, Katrina Pearson, Paul Jordan, Cassandra Spears, Charles Wu, Joy Wang, and Jamie Doyle of the Statistical Analysis and Reporting Branch for assistance on data-related questions. The authors would like to thank Mike Lauer for providing comments. The authors wish to thank Kendra Williams for assistance with data visuals as well as Julie Mason, Ming Lei, Jessica Faupel-Badger, Erica Ginsburg, Yvette Seger, Leo DiJoseph, Joshua Schnell, and Jonathan Wiest for the logic model. Any omissions or errors are the authors’.
References
1. Alberts B, Kirschner MW, Tilghman S, Varmus H. Rescuing U.S. biomedical research from its systemic flaws. Proc Natl Acad Sci U S A. 2014;111:5773–5777.
2. Lauer M, Tabak L, Collins F. Opinion: The next generation researchers initiative at NIH. Proc Natl Acad Sci U S A. 2017;114:11801–11803.
3. Nikaj S, Roychowdhury D, Lund PK, Matthews M, Pearson K. Examining trends in the diversity of the U.S. National Institutes of Health participating and funded workforce [published online ahead of print June 19, 2018]. FASEB J. doi: 10.1096/fj.201800639
4. National Academy of Sciences, Engineering, and Medicine. The Next Generation of Biomedical and Behavioral Sciences Researchers—Breaking Through. 2018.Washington, DC: National Academies Press.
5. 21st Century Cures Act of 2016. H.R.34. 114th Congress. Public Law No. 114-255.
https://www.congress.gov/bill/114th-congress/house-bill/34. Accessed October 13, 2018.
6. National Institutes of Health. Individual Mentored Career Development Awards Program.
https://grants.nih.gov/training/K_awards_evaluation_finalReport_20110901.pdf. Published 2011. Accessed October 13, 2018.
7. Mason JL, Lei M, Faupel-Badger JM, et al. Outcome evaluation of the National Cancer Institute career development awards program. J Cancer Educ. 2013;28:9–17.
8. Jagsi R, Griffith KA, Jones RD, Stewart A, Ubel PA. Factors associated with success of clinician–researchers receiving career development awards from the National Institutes of Health: A longitudinal cohort study. Acad Med. 2017;92:1429–1439.
9. Jenkins JM, Farkas G, Duncan GJ, Burchinal M, Vandell DL. Head Start at ages 3 and 4 versus Head Start followed by state pre-K: Which is more effective? Educ Eval Policy Anal. 2016;38:88–112.
10. Linden A, Adams JL. Combining the regression discontinuity design and propensity score–based weighting to improve causal inference in program evaluation. J Eval Clin Pract. 2012;18:317–325.
11. Keele L, Titiunik R, Zubizarreta JR. Enhancing a geographic regression discontinuity design through matching to estimate the effect of ballot initiatives on voter turnout. J R Stat Soc Ser A Stat Soc. 2015;178:223–239.
12. Iacus SM, King G, Porro G. Causal inference without balance checking: Coarsened exact matching. Polit Anal. 2012;20:1–24.
13. Jacob BA, Lefgren L. The impact of NIH postdoctoral training grants on scientific productivity. Res Policy. 2011;40:864–874.
14. Jacob BA, Lefgren L. The impact of research grant funding on scientific productivity. J Public Econ. 2011;95:1168–1177.
15. Imbens GW, Wooldridge JM. Recent developments in the econometrics of program evaluation. J Econ Lit. 2009;47:5–86.
16. Matthews K, Calhoun K, Lo N, Ho V. The aging of biomedical research in the United States. PLoS One. 2011;6:e29738.
17. Sinatra R, Wang D, Deville P, Song C, Barabási AL. Quantifying the evolution of individual scientific impact. Science. 2016;354:aaf5239.
18. Eblen MK, Wagner RM, Roychowdhury D, Patel KC, Pearson K. How criterion scores predict the overall impact score and funding outcomes for National Institutes of Health peer-reviewed applications. PLoS One. 2016;11:e0155060.
19. Charette MF, Oh YS, Maric-Bilkan C, et al. Shifting demographics among Research Project Grant awardees at the National Heart, Lung, and Blood Institute (NHLBI). PloS One. 2016;11:e0168511.
