Secondary Logo

Journal Logo

Research Reports

Toward Independence

Resubmission Rate of Unfunded National Heart, Lung, and Blood Institute R01 Research Grant Applications Among Early Stage Investigators

Boyington, Josephine E.A. PhD, MPH, CNS; Antman, Melissa D. PhD; Patel, Katherine C. MSPH; Lauer, Michael S. MD

Author Information
doi: 10.1097/ACM.0000000000001025
  • Free

Abstract

Sally Rockey,1 the immediate past deputy director for extramural research at the National Institutes of Health (NIH), began the April 29, 2014, post of her blog “Rock Talk” writing, “The strength of the biomedical research enterprise depends on new researchers becoming independent NIH-funded researchers”; however, for many new investigators, receipt of an NIH independent research (R01) award presents a significant challenge.2,3 Richard Harris, in his recently posted National Public Radio story “When scientists give up,”4 described the pursuits of two early career scientists—both of whom unsuccessfully sought independent research grant awards and subsequently abandoned their research careers. National Heart, Lung, and Blood Institute (NHLBI) staff members have heard similar stories from other early career investigators. One possible manifestation of these junior researchers’ frustration might be their decision not to resubmit an application after failing to secure funding with the first peer-reviewed R01 grant application. We decided, therefore, to measure and ascertain possible correlates for the resubmission of unfunded R01 applications submitted to the NHLBI by early stage investigators (ESIs) during fiscal year (FY) 2010, 2011, and 2012.

Terms, Considerations, and R01 Review Processes

We focused on the R01 grant mechanism because it is the hallmark of research independence and because NIH affords funding benefits to ESIs through this mechanism.2,5–7 NIH defines an ESI as any applicant who is within 10 years of completion of his/her terminal research degree or medical residency and has never received an R01 grant (or equivalent).5,8 Policies related to early career scientists include the clustering and review of ESIs’ applications apart from those from established investigators,5,8 the provision of opportunities to resubmit unfunded applications earlier and thereby reduce the time to resubmission by four months,9 and fiscal policies that ensure that new investigators are funded at the same rate as established investigators.10 Within the confines of these NIH policies, individual NIH institutes freely develop and implement strategies that ensure a scientifically diverse and workforce-balanced portfolio.

During FY 2010–2012, all unfunded NHLBI ESI R01 applications were eligible for special funding considerations if their percentile scores (see below) were within 10 points of the payline (i.e., the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available).11,12

All applications submitted to NIH are reviewed, but not all applications are discussed at peer review meetings. Discussed applications receive criterion scores for five core variables, an overall impact score, and a percentile score. Applications that are not discussed receive criterion scores but no overall impact score and no percentile score. It is generally assumed that these applications are above the 50th percentile of the applications submitted for review.

The percentile score13 is a measure of an application’s relative rank compared with other applications reviewed by a particular study section (i.e., a panel of scientific experts [i.e., “reviewers”], convened by the NIH Center for Scientific Review or an NIH institute, to provide initial scientific review14). It reflects reviewers’ global assessment of the application’s scientific and technical merit. The score ranges from 1st through 99th percentile, and favorable scores are below the 50th percentile.13

The overall impact score reflects reviewers’ judgment of a project’s ability to influence its field, and it is determined by reviewers’ ratings of the overall merit of an application, particularly after taking into consideration the five core peer review criteria—(1) significance, (2) approach, (3) investigator, (4) innovation, and (5) environment—in addition to other application-relevant factors such as human subjects protections.

In their consideration of significance, reviewers reflect on the extent to which a research “project address[es] an important problem or a critical barrier to progress in the field,”15,16 and in their consideration of approach, they assess how well a research project’s “strategy, methodology, and analyses [are] well-reasoned and appropriate to accomplish the stated objectives of the project.”15,16 Reviewers considering the investigator criterion ask the following key questions: “Are the PD/PIs [program director / principal investigators], collaborators, and other researchers well suited to the project?” and “If Early Stage Investigators or New Investigators, or in the early stages of independent careers, do they have appropriate experience and training?”15 The consideration of innovation focuses on “how much a project can shift the current research or clinical paradigms by utilizing novel theoretical concepts, approaches, or methodologies, instrumentation, or interventions,” and, finally, the review of environment centers on whether “the scientific environment in which the work will be done contribute[s] to the probability of success”; whether “the institutional support, equipment and other physical resources available to the investigators [are] adequate for the project proposed”; and whether the project would “benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?”15,16

The overall impact score is not an average of the five core criteria scores; rather, it is the average of the individual ratings of overall merit given by all reviewers after assessment of the five scored criteria listed above, plus, as mentioned, additional concerns such as the protection of human subjects, the welfare and care of vertebrate animals, and the consideration of biohazards.17,18 The relationship between overall impact score and percentile score is that the percentile score normalizes the application’s overall impact score across study sections and thereby attenuates unintended variations in scoring.19

Method

In our effort to examine resubmissions (our primary end point), we extracted and deidentified all unfunded, competing, new (A0) R01 applications that had undergone peer review during FY 2010–2012. We extracted the applications from the NIH electronic research administration “Information for Management, Planning, and Coordination II (IMPAC II)” database. As potential predictors of resubmission, we considered application-based factors (e.g., year of original submission and peer review scores), applicant-based factors (e.g., demographics, training, prior funding), and institution-based factors (e.g., ranking in receipt of NIH funding, and type of institution [research, medical, other]).

We determined variable frequencies with descriptive analyses, and we used random forest methodology,20 a robust machine-learning-based predictors’ ensemble approach, to identify the most important predictor variables for inclusion in subsequent logistic regression models. We first analyzed the overall pool of applications and subsequently the subset of applications that were discussed and percentiled. We performed all analyses at the NIH, using R3.1.0 (packages Hmisc, rms, ggplot2, and randomForestSRC).

Results

We extracted, for applications reviewed in FY 2010–2012, a total of 34,240 unfunded A0 NIH grant applications. Of the 4,587 NHLBI applications, 833 (18.2%) were ESI applications. (Comparatively, of the 29,653 remaining NIH applications, 5,268 [17.8%] were ESI applications.) We focused our analyses on the 821 NHLBI ESI applications that were not formally withdrawn. Table 1 summarizes application-, applicant-, and institution-based characteristics, and Table 2 provides the mean review criterion scores for the applications.

Table 1
Table 1:
Frequencies of Baseline Application-, Applicant-, and Institution-Based Variables for NHLBI Unfunded Early Stage Investigator Applications, FY 2010–2012a
Table 2
Table 2:
Mean Review Criterion Scores for NHLBI Unfunded Early Stage Investigator Applications, FY 2010–2012

Resubmitted applications

The overall resubmission rate was 51.4% (422 of 821). Of the pool of 821 applications, 382 (46.5%) were discussed and given a percentile score, and 294 (35.8%) scored less than 50 (Table 1). Of the applications with a percentile score less than 50, 82.3% (242 of 294) were resubmitted, whereas among the 527 applications with a percentile score equal to or greater than 50, only 180 (34.2%) were resubmitted (P < .001). Using the random forest machine-learning approach on the entire pool of applications, we observed that the most important correlate, by far, of resubmission was a percentile score less than 50. The investigator, approach, significance, innovation, and environment criterion scores were less correlated with resubmission (Figure 1). Neither applicant- nor institution-based variables emerged as important correlates.

Figure 1
Figure 1:
Variable importance and probability of resubmission for all National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 applications, by application-, applicant-, and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer-reviewed criteria on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant award; K-award indicates career development grant awards; and T-grant means training grant award. ICR indicates “initial council review” (the NIH institutes generally make grant-funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “special emphasis panel,” which is a review group set up to conduct a one-time review of grants; RPG indicates “research project grant,” which is an award that supports scientific research projects. “NIH institutional rank” is the rank quintile in which an institution falls, based on the total amount of funds received from NIH in the last five years.

In the corresponding logistic regression model, the only independent predictors were percentile score of less than 50 (P < .001) and the investigator criterion score (P = .01). We noted that 560 of 821 (68.2%) of the ESI applications were from male PIs and that the sample of applications from groups underrepresented in the biomedical sciences was small, thereby limiting our ability to conduct subgroup analyses.

Discussed and percentiled applications

For the 382 applications that were both discussed and received a percentile ranking, we examined the association between overall impact score, raw percentile score, percentile distance from payline, and resubmission status. Overall impact score did not emerge as a key predictor variable; however, we found that the raw percentile score and the percentile distance from the payline were essentially equivalent predictors of resubmission. We calculated “percentile distance” as a derived variable by measuring the distance of an application’s percentile score from the operative NHLBI payline at the time of funding consideration. The payline at NHLBI generally changes over the course of the fiscal year, being more conservative (lower) at the beginning of the year and more generous (higher) toward the end. With the calculation of percentile distance, we were able to create a variable with seemingly greater face value than raw percentile score and to also account for the impact of payline changes on resubmission.

As such, we tested percentile distance along with other candidate predictors in a random forest model and found that it was, by far, the strongest correlate of resubmission, followed by the significance and approach criterion scores (Figure 2). In the logistic regression model including percentile distance, the five criterion scores, requested budget, and gender, the only independent predictor was percentile distance (P < .001). We further examined the relationship between percentile distance and the probability of resubmission and found that as percentile distance increased, the probability of resubmission decreased (Figure 3).

Figure 2
Figure 2:
Variable importance and probability of resubmission for discussed and percentiled National Heart, Lung, and Blood Institute (NHLBI) FY 2010–2012 early stage investigator unfunded A0 R01 applications, by application-, applicant-, and institution-based factors. Source: National Institutes of Health (NIH) Information for Management, Planning, and Coordination II (IMPAC II) database. Approach, Investigator, Significance, Innovation, and Environment are the five peer-reviewed criteria on which NIH grant applications are scored. Requested budget is the total budget requested in an application. FY is the fiscal year in which the application was reviewed. F-grant indicates fellowship training grant; K-award indicates career development grant award; and T-grant means training grant. ICR indicates “initial council review” (the NIH institutes generally make grant funding decisions after grants have been presented at one of their advisory council meetings). SEP indicates a “special emphasis panel,” which is a review group set up to conduct a one-time review of grants. RPG indicates “research project grant,” which is an award that supports scientific research projects. Payline is the cutoff point set by NHLBI for funding grant applications, after balancing projected grant numbers, total grant budgets, and funds available. Distance from payline, a derived variable, is the calculated distance of the difference between the overall impact score and the payline operative at the time funding decisions are made. “NIH institutional rank” is the rank quintile in which an institution falls, based on the total amount of funds received from NIH in the last five years.
Figure 3
Figure 3:
Random-forest-adjusted probability of resubmission for percentiled applications according to distance from payline. The shaded space connotes the confidence interval of the points in the plotted line.

Unfunded applications eligible for NHLBI special funding consideration FY 2010–2012

During FY 2010–2012, NHLBI considered 342 total (new and resubmitted) ESI R01 applications for funding. Two hundred two (59%) of these had percentile scores above, but within 10 points of, the NHLBI R01 payline and were, therefore, awarded funding based on NHLBI’s special funding consideration for ESIs.

Discussion

In the current budget-driven fiscal climate, early career investigators may be frustrated by failure to secure an R01 grant and may express this frustration by not resubmitting unfunded applications. We found that slightly more than half of NHLBI ESIs (51.4%) resubmitted their unfunded applications, and that in the regression model of the overall pool of applications, a percentile score of less than 50 and the investigator criterion score were the two significant predictors of resubmission.

As noted above, percentile reflects the relative rank of an application compared with others reviewed by a particular study section, and the investigator criterion score reflects reviewers’ judgment of a PI and his/her team’s competence to execute a proposed project. A higher investigator criterion score, therefore, indicates the need for the PI to gain more practical and relevant research skills, the need for a stronger and more complementary research team, or both. Compared with other factors that could affect the investigator criterion score, skill set and an appropriate research team are more easily changeable than, for example, demographics (race/ethnicity) or degree type (MD versus PhD). A recent internal NIH analysis using overall NIH data determined that degree type has no measurable impact on R01 success rates.21 In contrast, a recent NIH-commissioned external study that sought to understand the relationship between race/ethnicity and grant receipt documented differential probabilities of R01 grant receipts by race/ethnicity.22 Because of insufficient data, the study did not report institute-specific, race-based results, but it did show that overall, Hispanic and black applicants were less likely than others to resubmit a revised application and to receive a grant-award.22

Our study differs from previous work in that we focused specifically on ESIs and on a critical antecedent behavior for grant receipt—that is, resubmission of unfunded R01 applications. Although we focused on the vulnerable group of early career investigators, we were underpowered to conduct subgroup analyses (e.g., race/ethnicity) because of our small sample size. We found correlations between application-based characteristics and resubmission of an unfunded R01 application but no independent association of applicant- or institution-based characteristics with resubmission. We did observe twice as many male as female applicants in our cohort of 821. Ley and Hamilton23 and Dunbar and colleagues24 have reported similar observations and have commented that observed differences are evidence of the greater exodus of females at the independent career stage and not a result of gender-based differences in success rates. Apart from the proportional differences in the number of applicants by gender, we found that gender was not a significant predictor of resubmission.

The finding that discussed (thus percentiled) applications were resubmitted at a higher rate is reassuring because it indicates that an objectively derived application attribute (i.e., percentile score) seemingly drives resubmission behavior. As noted earlier, we determined percentile distance to be an essentially equivalent surrogate for raw percentile score and to be the only significant predictor of resubmission for discussed applications. Consequently, this finding suggests that the primary driver in an applicant’s decision to resubmit should be the application’s overall percentile score and less so the individual criterion or overall impact scores.

NHLBI funded 342 ESI R01 applications in FY 2010–2012. Of these, 202 (59%) were within 10 points of the NHLBI payline that was operative when these applications were being considered for funding and were therefore awarded on the basis of NHLBI’s special funding consideration for ESIs. We were unable to assess whether knowledge of the funding advantage offered by NHLBI and of other related NIH policies (e.g., favorable resubmission time allowances, clustered review, rapid processing of new investigator applications) affected resubmission behavior. Given these policies—along with the most recent NIH application submission policy25 that allows resubmission of a previously unsuccessful application—we predict that resubmission of new applications will continue at the same or higher rate.

We note three important limitations of this study. First, we could not use data prior to FY 2010 because at the NIH, the American Recovery and Reinvestment Act, the designation of the ESI category, and the new scoring and summary statement percentiling approaches were all implemented in 2009.5,26,27 Second, we did not analyze outcomes for other NIH institutes and centers that may have differing ESI policies, so we cannot speak to ESI resubmission behavior outside of the NHLBI. Lastly, we had too few foreign/international applicants (n = 2) and too few applicants from groups underrepresented in the biomedical sciences to allow for robust subgroup analyses.

In the future, we plan to assess the long-term research career trajectory and performance of NHLBI newly independent investigators, the relative effect of NHLBI special funding advantage on research career performance among beneficiaries and nonbeneficiaries, and the combined effect of NIH ESI-focused policies on research career independence.

Conclusions

The NIH and the biomedical research communities must contend with a stringent budget-driven climate, resulting in shared concerns about the future of new investigators.28,29 For the early career biomedical research scientist, receipt of an NIH R01 (or equivalent) award constitutes a major step toward career independence2,6,7; however, reaching this goal is a challenge for many.2,28,29 Collectively, the length of time between research idea and receipt of an NIH R01 research project award, the multiple years of flat NIH budgets, the ever-increasing pool of applications, and the current grant success and funding rates1,30 all portend continued, substantial challenges for early career scientists seeking research independence. Consequently, for many whose cache of research ideas, funding options, and opportunities for establishing career independence are few, resubmission of R01 grant applications could constitute a critical step toward independent research careers.

As the third largest NIH institute (with an FY 2014 budget of approximately 2.9 billion dollars31), the NHLBI recognizes the immense challenges facing young investigators pursuing heart-, lung-, blood-, and sleep-related research. Within the confines of NIH policies, NHLBI has engaged strategies, including favorable funding advantages for ESIs, to facilitate their timely entry into the biomedical research pipeline.

Acknowledgments: The authors thankfully acknowledge Dr. Lawrence Fine, Dr. Paul Sorlie, and Mr. Matthew Eblen, all of the National Institutes of Health, for their support and invaluable assistance with various aspects of this project.

References

1. Rockey S.. A look at programs targeting new scientists National Institutes of Health Office of Extramural Research Web site. April 29, 2014 http://nexus.od.nih.gov/all/2014/04/29/a-look-at-programs-targeting-new-scientists/. Accessed September 30, 2015
2. Alberts B, Kirschner MW, Tilghman S, Varmus H.. Rescuing US biomedical research from its systemic flaws. Proc Natl Acad Sci U S A. 2014;111:5773–5777
3. Harris A.. Young, brilliant and underfunded N Y Times. October 2, 2014 http://www.nytimes.com/2014/10/03/opinion/young-brilliant-and-underfunded.html?_r=3. Accessed September 30, 2015
4. Harris R.. When scientists give up National Public Radio Web site. September 9, 2014 http://www.npr.org/blogs/health/2014/09/09/345289127/when-scientists-give-up?refresh=true. Accessed September 30, 2015
5. National Institutes of Health, Office of Extramural Research. Encouraging early transition to research independence: Modifying the NIH new investigator policy to identify early stage investigators. NOT-OD-08-121. Updated August 12, 2009. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-121.html. Accessed September 30, 2015
6. National Instutes of Health, Advisory Committee to the Director.Draft report of the Advisory Committee to the Director Working Group on Diversity in the Biomedical Research Workforce. June 13, 2012 http://acd.od.nih.gov/Diversity%20in%20the%20Biomedical%20Research%20Workforce%20Report.pdf. Accessed September 30, 2015
7. Garrison HH, Deschamps AM.. NIH research funding and early career physician scientists: Continuing challenges in the 21st century. FASEB J. 2014;28:1049–1058
8. National Institutes of Health, Office of Extramural Research.New and early stage investigator policies: Definition of new investigator. Updated February 10, 2014. http://grants.nih.gov/grants/new_investigators/#definition. Accessed September 30, 2015
9. National Institutes of Health, Office of Extramural Research.Full implementation to shorten the review cycle for new investigator R01 applications reviewed in Center for Scientific Review (CSR) recurring study sections. NOT-OD-07-083. Updated October 5, 2011. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-07-083.html. Accessed September 30, 2015
10. National Institutes of Health, Office of Extramural Research.Revised new and early stage investigator policies. NOT-OD-09-013. Updated May 8, 2013. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-013.html. Accessed September 30, 2015.
11. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). NIAID glossary of funding and policy terms and acronyms—P. http://www.niaid.nih.gov/researchfunding/grant/strategy/pages/7payline.aspx. Accessed October 6, 2015
12. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). Research funding: Understand paylines and percentiles. http://www.niaid.nih.gov/researchfunding/glossary/pages/p.aspx. Accessed October 6, 2015
13. National Institutes of Health, Office of Extramural Research.Grants and funding glossary and acronym list: Percentile. Updated March 20, 2013. http://grants.nih.gov/grants/glossary.htm#Percentile. Accessed September 30, 2015
14. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). NIAID glossary of funding and policy terms and acronyms—S. Study section. http://www.niaid.nih.gov/researchfunding/glossary/pages/s.aspx#ss. Accessed October 30, 2015
15. National Institutes of Health, Office of Extramural Research.Grants and funding: Definitions of criteria and considerations for research project grant (RPG/R01/R03/R15/R21/R34) critiques. Updated April 15, 2015. https://grants.nih.gov/grants/peer/critiques/rpg.htm. Accessed September 30, 2015
16. National Institutes of Health.Enhancing peer review: The NIH announces enhanced review criteria for evaluation of research applications received for potential FY2010 funding. NOT-OD-09-025. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-025.html. Accessed October 7, 2015
17. National Institutes of Health. Scoring system and procedure. https://grants.nih.gov/grants/peer/guidelines_general/scoring_system_and_procedure.pdf. Accessed October 7, 2015.
18. National Institutes of Health, National Institute of Allergy and Infectious Diseases (NIAID). Research funding: Overall score. http://www.niaid.nih.gov/researchfunding/glossary/pages/o.aspx#overallscore. Accessed October 7, 2015
19. National Institutes of Health Office of Extramural Research.Grants and funding glossary and acronym list: Impact score. Updated March 20, 2013. http://grants.nih.gov/grants/glossary.htm#ImpactScore. Accessed September 30, 2015
20. Biau G.. Analysis of a random forests model. J Mach Learn Res. 2012;13:1063–1095
21. Rockey S.. Does your institution matter? National Institutes of Health Office of Extramural Research Web site. May 14, 2012 http://nexus.od.nih.gov/all/2012/05/14/does-your-institution-matter/. Accessed September 29, 2015
22. Ginther DK, Schaffer WT, Schnell J, et al. Race, ethnicity, and NIH research awards. Science. 2011;333:1015–1019
23. Ley TJ, Hamilton BH.. Sociology. The gender gap in NIH grant applications. Science. 2008;322:1472–1474
24. Dunbar CE, Abkowitz JL, Berliner N, Shurin SB.. “Ask and ye shall receive”—women, stand up and be counted. Hematologist. May 30, 2014 http://www.hematology.org/Thehematologist/Op-Ed/2883.aspx. Accessed September 30, 2015
25. National Institutes of Health, Office of Extramural Research.NIH and AHRQ announce updated policy for application submission. Updated April 22, 2014. http://grants.nih.gov/grants/guide/notice-files/NOT-OD-14–074.html. Accessed September 30, 2015
26. National Institutes of Health. The NIH director: NIH’s role in the American Recovery and Reinvestment Act (ARRA). http://www.nih.gov/about/director/02252009statement_arra.htm. Accessed October 7, 2015
27. National Institutes of Health. NIH and the American Recovery and Reinvestment Act: NIH grants funded by the American Recovery and Reinvestment Act of 2009. http://report.nih.gov/recovery/ARRAFunding.aspx. Accessed October 7, 2015
28. Langer JS.. Enabling scientific innovation. Science. 2012;338:171
29. Ioannidis JP.. More time for research: Fund people not projects. Nature. 2011;477:529–531
30. U.S. Department of Health and Human Services.. NIH Data Book: Success rates and funding rates, R01-equivalent investigators: Competing applicants, awardees, and funding rates. NIH Research Portfolio Online Reporting Tools (RePORT). Updated June 15, 2015. http://report.nih.gov/nihdatabook/. Accessed September 30, 2015
31. 113th Congress of the United States. Consolidated and Further Continuing Appropriations Act of 2015. Public Law No: 113–235. December 16, 2014 https://www.congress.gov/113/plaws/publ235/PLAW-113publ235.pdf. Accessed October 1, 2015
© 2016 by the Association of American Medical Colleges