Home Current Issue Previous Issues Published Ahead-of-Print Collections For Authors Journal Info
Skip Navigation LinksHome > May 2010 - Volume 85 - Issue 5 > Commentary: New Guidelines for NIH Peer Review: Improving th...
Academic Medicine:
doi: 10.1097/ACM.0b013e3181d7e130
Commentary

Commentary: New Guidelines for NIH Peer Review: Improving the System or Undermining It?

Spiegel, Allen M. MD

Free Access
Article Outline
Collapse Box

Author Information

Dr. Spiegel is Marilyn and Stanley M. Katz Dean, Albert Einstein College of Medicine, Bronx, New York, and former director, National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, Maryland.

Editor's Note: This is a commentary on Costello LC. Perspective: Is NIH funding the “best science by the best scientists?” A critique of the NIH R01 research grant review policies. Acad Med. 2010:85;775–779.

Correspondence should be addressed to Dr. Spiegel, Albert Einstein College of Medicine, Belfer 312, 1300 Morris Park Avenue, Bronx, NY 10461; telephone: (718) 430-2801; e-mail: allen.spiegel@einstein.yu.edu.

Collapse Box

Abstract

The National Institutes of Health (NIH) peer review system has been viewed as the best way to guarantee the scientific independence of biomedical research in the United States, and it has been emulated internationally. The system, however, is subject to a variety of stresses, and these have always been exacerbated at times of flat NIH funding, as in the past five years. To address several of these stresses, NIH first conducted a “diagnostic self-study” of the peer review system and then implemented a number of changes. Costello, in a Perspective in this issue of Academic Medicine, argues that two of these changes, special consideration for new investigators and emphasis of the criterion of “innovation,” undermine the stated goal of funding the “best science by the best scientists.” In this commentary on Costello's Perspective article, the author examines the issue of NIH funding of new investigators from a historical perspective, in the context of overall NIH priority setting in resource allocation. The related issue of innovation as a criterion in NIH peer review is also addressed, and the commentary concludes with an affirmation of the need to measure outcomes in assessing the impact of changes in the NIH peer review system.

In his provocative and thoughtful Perspective article in this issue, Costello1 argues that recently issued guidelines for the National Institutes of Health (NIH) grant review process are in conflict with the stated objectives of NIH for the funding of research grants, namely to “fund the best science by the best scientists.” Specifically, he contends that the new guidelines introduce discrimination into the review process by favoring “new” over “established” investigators. The result, he argues, will be funding of lesser-quality grant proposals, to the ultimate detriment of the best interests of the biomedical research community and of the health and welfare of the public at large. While he freely admits that his thesis is not based on data or statistics, he bases his views on “experience of nearly 50 years, during which [he] received about 30 research grants, served on grant review committees for NIH and other agencies, and presented grantsmanship lectures and workshops at numerous medical institutions.” This lengthy experience also led him to question the value of contemporary education and training of biomedical scientists in an earlier Perspective article published in Academic Medicine.2 He hearkens back to this theme in trying to explain the relative lack of success of the current generation of new investigators in obtaining NIH grants, and he also invokes it in characterizing the declining quality of reviewers on NIH study sections. Finally, he singles out “innovation” as a particularly misguided criterion in judging the quality of research grant applications. In considering Costello's arguments, I believe it is useful to place some of the issues he raises in a broader, historical context.

Back to Top | Article Outline

Priority Setting at NIH

Costello's objection to NIH's creation of what he terms a “young investigator development program” overlooks the fact that NIH has always had to balance a number of competing priorities. Indeed, how NIH sets priorities has been a perennial theme of Congressional hearings and of studies and reports by independent, “blue-ribbon” panels. The executive summary of NIH's 2007–2008 Peer Review Self-Study3 nicely captured the tensions inherent in this process:

Above all, it is critical that the NIH maintain the core values of peer review: scientific competence, fairness, timeliness, and integrity. When striving to fund the “best” science, the NIH must consider many factors, including scientific quality, public health impact, the mission of an NIH Institute or Center, and the current NIH portfolio.

Although some have argued that investing solely in high-quality, investigator-initiated, curiosity-driven research offers the best prospect of discoveries that will ultimately lead to improved health, the public and its elected representatives in Congress, as well as some members of the biomedical research community, have insisted that NIH must do more than fund individual, investigator-initiated research grants. Institute directors were reminded often enough by members of Congress who control the NIH appropriation that they represented the National Institutes of Health, not Science. Notwithstanding the efforts of successive NIH directors to resist disease-specific “earmarking,” mandates for support of research on HIV/AIDS, cancer, type 1 diabetes, and autism, to name just a few, were enacted over the years. In the aftermath of the attacks on September 11, 2001, and the subsequent anthrax attacks, nearly 40% of the NIH budget increase for 2003, the last year of the five-year doubling of the NIH budget, was earmarked for “biodefense” research. In the face of pressure from patient and disease-specific organizations, discipline-specific professional organizations, and many other types of advocacy groups, the NIH leadership has always faced the difficult, some would say impossible, challenge of allocating finite resources in a way that best serves the interests of the public whose tax dollars support the entire enterprise.

NIH priority setting involves not only decisions on resource allocation between various diseases but also decisions on resource allocation for centers versus individual research grants, for clinical or translational research (however those are defined) versus basic research, and for training grants versus research grants. A recent example is the allocation of NIH funds for “comparative effectiveness research,” at least partly in response to the health care reform effort. It is in this overall context of priority setting that we need to approach the issue of NIH's allocating specific resources for funding of new investigators.

Back to Top | Article Outline

NIH Funding for New Investigators: Past and Present

Special effort by NIH to ensure adequate funding of new investigators is not a new phenomenon, and the first question to be addressed is why this should be a high priority for NIH. One reason often cited, and difficult to prove, is that new investigators bring a fresh outlook to important biomedical research problems, so that failure to provide adequate support to them risks losing “breakthrough” contributions. Perhaps the most cogent justification, though, is the incredibly difficult nature of the problems biomedical research addresses, and the realization that, much as we might wish for progress to be rapid, the reality is that it is often agonizingly slow, taking decades or more to improve public health. Recognizing this reality mandates that we ensure a continued supply of well-trained, highly motivated and dedicated investigators whose work will provide the key advances of the future. No matter how powerful new technologies and research tools become, human capital will always be the key ingredient at the heart of the biomedical research enterprise, and we cannot afford to allow this vital resource to die out.

Although the size of NIH's annual budget increases have fluctuated widely over the years, the success rate for traditional R01 research grants and the proportion of awards going to previously unfunded investigators have markedly declined since 1963 when they were 58% and 35%, respectively.4 Since 1977, NIH has introduced a series of measures and mechanisms to enhance the career prospects of new investigators. Some of these, such as the R29 (First Independent Research Support and Transition Award), were eventually abandoned when it was determined that the level of support provided was inadequate to launch a successful independent career. Beyond use of special mechanisms, many if not most institutes at NIH implemented special emphasis programs for new investigators, with funding of applications at several points beyond the general pay line. Despite such measures, by 2006, the percentage of R01s going to new investigators continued to decline, perhaps not surprising given overall success rates below 21%. In effect, a substantial increase in the size of the investigator pool that occurred in response to the doubling of the NIH budget between 1999 and 2003 collided with a flat NIH budget (actually declining in real dollars) postdoubling to yield a mismatch between numbers of grant applications and available funds.

It is against this background that NIH in 2009 implemented a series of measures to improve the peer review system. A discussion of all the changes implemented is beyond the scope of this commentary. For a more comprehensive view, I refer the interested reader to a provocatively titled piece: “NIH peer review reform—Change we need, or lipstick on a pig?”5 Here I will focus on the program to ensure adequate funding of new investigators and the “innovation” criterion to which Costello objects. The new NIH policy encourages institutes to maintain approximately equal success rates for new and established investigators on new (Type 1) research project grant applications. Success rates on competing renewal (Type 2) applications are unaffected, providing a measure of stability for successful projects of established investigators, a point Costello evidently overlooks.4 By clustering applications from new investigators during peer review, NIH is attempting to “level the playing field” between new and established investigators on Type 1 applications. Costello claims this practice compromises the quality of the science NIH will fund. It is not clear his assertion is correct. What is clear is that this practice is a deliberate setting of NIH priorities for reasons outlined above.

In implementing this new policy, NIH is also trying to address another inexorable trend in NIH funding demographics over the years: the increase in age of investigators at time of first R01 award (Figure 1). A variety of factors, including the elongation of training periods (particularly for MD/PhDs, a key pool of physician–scientists) and the delays in granting independent faculty rank, have led to an increase in average age at first award of an R01 to greater than 42. Is this in fact a problem? At least in the mathematical and physical sciences, there is more than anecdotal evidence that major achievements occur at relatively early ages and career stages. This is less clear in the biological sciences. Nonetheless, not only NIH, but also private funding foundations (e.g., the Howard Hughes Medical Institute's Early Career Scientist support program) have taken steps to increase the pool of “Early Stage Investigators” (defined as those within 10 years of completion of terminal degree or medical residency) on the assumption that such support will pay dividends in enhanced productivity. In addition to giving special consideration to Early Stage Investigators, NIH has launched additional programs such as the Pathways to Independence awards (K99/R00) which provide two years of postdoctoral support followed by three years of independent research grant support to encourage early transition to independent grant careers.

Figure 1
Figure 1
Image Tools
Back to Top | Article Outline

NIH Support for Innovative Research

As pay lines and success rates decline with prolonged flat NIH budgets, a perennial murmur critical of the NIH peer review system, contending that it is too conservative and tends to fund only incremental research advances, is amplified to a roar. Costello decries the explicit mandate in the new NIH peer review criteria for “innovation,” correctly pointing out that true innovation is hard to define and that it most certainly does not represent merely the use of new technologies to address a research problem. Nonetheless, there are clear differences in innovation between the research that led, for example, to the discovery of small interfering RNAs and that which extends observations on a particular signal transduction pathway to yet another cell type. In response to the perceived need to shift the balance of funding toward high-risk/high-reward research, NIH has not only highlighted innovation in its peer review criteria but has also launched a number of new award mechanisms, such as the NIH Director's Pioneer Awards, Transformative R01s, and New Innovator Awards. These awards attempt to counter the inherent tendency of any system relying on prospective review of research to default toward a risk-averse bias, a bias exacerbated at times by strikingly low success rates.

Back to Top | Article Outline

Evaluating the Performance of the NIH Peer Review System

Ideally, the changes NIH has implemented would be monitored, not only in terms of process but, more important, in terms of outcomes. Only in this way would we acquire the evidence base supporting modifications of the NIH peer review system. Unfortunately, this is difficult for a number of reasons. Measuring the impact of not funding investigators in terms of lost productivity is highly problematic if they are lost to the biomedical research enterprise. Comparing groups of funded investigators (e.g., new versus established investigators, or new investigators who would have been funded without special consideration versus those funded through the “reach” goals) is also not straightforward. What parameters are the most important to monitor? Publications? Prizes and awards? Changes in actual health outcomes? All would require years of follow-up. Nonetheless, I would argue that it is incumbent on NIH to evaluate its new mechanisms of research support and criteria for peer review to determine whether these deserve to be continued, expanded, or abolished. This, in fact, was explicitly articulated as one of the seven challenges facing the NIH peer review system: “Meeting the need for continuous review of peer review.”3 Only in this way will NIH be able to ensure that it is funding the “best science by the best scientists.”

Back to Top | Article Outline

Acknowledgments:

The author gratefully acknowledges Arturo Casadevall, MD, PhD, for helpful suggestions.

Back to Top | Article Outline

Funding/support:

None.

Back to Top | Article Outline

Other disclosures:

None.

Back to Top | Article Outline

Ethical approval:

Not applicable.

Back to Top | Article Outline

References

1Costello LC. Perspective: Is NIH funding the “best science by the best scientists?” A critique of the NIH R01 research grant review policies. Acad Med. 2010:85;775–779.

2Costello LC. The effect of contemporary education and training of biomedical scientists on present and future medical research. Acad Med. 2009;84:459–463.

3National Institutes of Health. Executive summary. In: 2007–2008 Peer Review Self-Study. Available at: http://enhancing-peer-review.nih.gov/meetings/NIHPeerReviewReportFINALDRAFT.pdf. Accessed January 28, 2010.

4Schaffer WT, Rockey SJ. NIH continues to support the best science through R01s. The Scientist. 2009;23:25–27.

5Fang FC, Casadevall A. NIH peer review reform—Change we need, or lipstick on a pig? Infect Immun. 2009;77:929–932.

© 2010 Association of American Medical Colleges

Login

Article Tools

Images

Share