Share this article on:

The Relationship Between OREF Grants and Future NIH Funding Success

Hegde, Vishal, MD1; Johansen, Daniel, BS1; Park, Howard Y., MD1; Zoller, Stephen D., MD1; Hamad, Christopher, BS1; Bernthal, Nicholas M., MD1,a

doi: 10.2106/JBJS.16.01278
The Orthopaedic Forum
Disclosures

Background: The Orthopaedic Research and Education Foundation (OREF) is the leading specialty-specific nongovernmental organization providing orthopaedic funding in the United States. As extramural research funding has become increasingly difficult to acquire, one mission of the OREF is to support investigators to generate data needed to secure larger extramural funding from agencies such as the National Institutes of Health (NIH). The objectives of this study were to evaluate the rate of translating OREF faculty-level grants into subsequent NIH funding and to determine if there are identifiable factors that increase the rate of converting an OREF grant into NIH funding.

Methods: This is a retrospective review of OREF grants awarded to full-time faculty orthopaedic surgeons between 1994 and 2014. Grants were analyzed on the basis of award type and were categorized as basic science, clinical, or epidemiological. Sex, individual scholarly productivity, and publication experience were evaluated. All awardees were assessed for subsequent NIH funding using the NIH RePORTER web site.

Results: One hundred and twenty-six faculty-level OREF grants were awarded to 121 individuals. Twenty-seven OREF grant awardees (22%) received NIH funding at a mean of 6.3 years after OREF funding. Nineteen (46%) of 41 Career Development Grant winners later received NIH funding compared with 10 (12%) of 85 other award winners. OREF grants for basic science projects were awarded more often (58%) and were more than 4 times as likely to result in NIH funding than non-basic science projects (odds ratio, 4.70 [95% confidence interval, 1.66 to 13.33]; p = 0.0036). Faculty who later received NIH funding had higher scholarly productivity and publication experience (p < 0.05).

Conclusions: The OREF grant awardee conversion rate of 22% and, particularly, the 46% for Career Development Grant winners compares favorably with the overall NIH funding success rate (18% in 2014). Faculty-level OREF grants appear to achieve their purpose of identifying and supporting researchers who aim to secure subsequent federal funding.

Clinical Relevance: The goal of this study is to examine how successful faculty who have obtained OREF grants have been in securing NIH funding later in their careers. Although subsequent accrual of NIH funding is not the only goal of OREF funding, it can be used as an important benchmark to assess the development of orthopaedic clinician-scientists.

1Department of Orthopaedic Surgery, David Geffen School of Medicine at the University of California Los Angeles, Santa Monica, California

aE-mail address for N.M. Bernthal: nbernthal@mednet.ucla.edu

The Orthopaedic Research and Education Foundation (OREF) is the preeminent nongovernmental organization funding research across orthopaedic subspecialties in the United States. Since its inception in 1955, the OREF has awarded more than $143 million for 4,700 research and education initiatives1. One of the stated goals of the OREF is to promote career development for faculty in orthopaedic surgery who plan to make basic science research, clinical research, or epidemiological research a component of their careers. These grants were intended to assist junior faculty in establishing their research platforms, potentially laying the foundation to secure larger grants from other funding sources, such as the National Institutes of Health (NIH).

In light of the recent trend toward cutbacks for research support, the ability of faculty to secure extramural funding has become increasingly challenging2-4. The pursuit of progressively scarce research support compounds the challenges of an academic practice, particularly for junior faculty for whom the accrual of seed money to help to perform pilot studies and to establish research platforms is crucial. The OREF serves an important function in targeting and funding those researchers most likely to be successful in the future. Yet no study to date, to our knowledge, has evaluated the success of the OREF grant programs at enabling accrual of subsequent extramural funding or variables that may improve the rate of that success.

The purpose of this study was twofold: first, to determine the conversion rate of OREF grantees in subsequently garnering federal NIH support, and second, to identify potential attributes of OREF awardees that are associated with a higher likelihood of future NIH funding.

Back to Top | Article Outline

Materials and Methods

Rate of Conversion of OREF Award into Future NIH Funding Success

Data regarding all OREF grant awards from 1994 to 2014, including principal investigator, institution, project title, grant type, year awarded, and amount awarded, were generously provided by the OREF upon request from the authors. Grants afforded to part-time faculty, volunteer faculty, or residents were excluded in an effort to assess the translation of faculty-level grants in securing future federal funding. Five OREF grant categories were therefore included in our analysis: the Career Development Grant ($134,000 to $225,000), the Zimmer Orthopaedic Career Development Grant ($50,000), the Career Development Grant in Total Joint and Trauma Surgery ($50,000), the Young Investigator Grant ($50,000), and the New Investigator Grant ($50,000). This resulted in the inclusion of 128 total faculty-level OREF grant awards to 123 different individuals. The Career Development Grant was compared with the other 4 grants in a separate analysis given differences in the size and scope of the grants, with 3 years and $134,000 to $225,000 for the Career Development Grant compared with 1 year and $50,000 each for the other 4 grants.

The NIH RePORTER web site, which contains NIH grant information from 1992 to the present, was utilized to query all OREF awardees for NIH funding and type of NIH grant5. Only faculty who were principal investigators on an NIH grant were considered NIH award winners. Given the aim of the study to determine OREF grant translation into subsequent federal funding, of the 123 individuals who received grants, 2 faculty were excluded because they received NIH funding prior to receiving an OREF grant award, leaving 121 grants recipients for 126 grants.

Back to Top | Article Outline

OREF Awardee Variables Associated with Future NIH Funding Success

OREF grants were categorized in this study as basic science, clinical, or epidemiological in focus6, with basic science studies further subcategorized into those involving cell biology compared with biomechanical studies. All OREF awardees were searched using the American Academy of Orthopaedic Surgeons (AAOS) web site to determine their practice setting and subspecialty. Individual department web sites of recipients were queried for further practice details. Awardees were stratified into the following 9 commonly accepted orthopaedic subspecialties: sports, arthroplasty, oncology, pediatrics, trauma, hand, foot and ankle, shoulder and elbow, and spine. The subspecialty assigned to the awardees was taken on the basis of their primary subspecialty designation on their department web site.

The Scopus database was then queried for all included subjects to determine publication experience and scholarly impact, as measured by the h-index7. An individual’s h-index is defined as the number of an investigator’s publications (h) that have been cited at least h times8. The h-index as a measure of scholarly productivity and potential has been validated in several subspecialties within academic medicine, including neurosurgery, dermatology, and orthopaedic surgery9-15. An awardee’s h-index in the year in which he or she won the OREF award was recorded using the h-index graph in the Scopus database, which displays an individual’s h-index over time. Each awardee’s h-index was also recorded at the time of data collection, in September 2016, and was considered as the current h-index score. Publication experience was defined as the number of years from an individual’s first publication to the year of his or her OREF award.

Statistical analysis was performed using the Mann-Whitney U test for continuous data and the Pearson chi-square test for categorical data. A p value of <0.05 was considered significant. Odds ratios (ORs) were calculated using 2-by-2 frequency tables in Excel 2010 (Microsoft).

Back to Top | Article Outline

Results

On the basis of our methodology, of the 121 individual OREF grant recipients since 1994, all 121 recipients were full-time Doctor of Medicine (MD) faculty serving in an academic orthopaedic surgery department. Of these 121 recipients, 27 faculty-level OREF awardees (22%) later received 31 NIH funding grants (Fig. 1-A). Of the 27 NIH awardees, 8 (30%) had an MD and a Doctor of Philosophy (PhD). Of the remaining 94 OREF awardees who did not receive NIH finding, 7 (7%) had an MD and PhD. Thus, 53% (8 of 15) of the MD plus PhDs subsequently received NIH funding, and 18% (19 of 106) of the MDs subsequently received NIH funding. Five investigators received 2 different OREF awards. All 5 received a non-Career Development Grant followed by a Career Development Grant for cell biology basic science work. Two (40%) of these 5 awardees received a subsequent NIH award.

Fig. 1

Fig. 1

When stratified by award type, 19 (46%) of the 41 awardees for the 3-year Career Development Grant later received NIH funding (Fig. 1-B), compared with 10 (12%) of the 85 awardees for the other 1-year awards (Fig. 1-C) (OR, 6.47 [95% confidence interval (CI), 2.63 to 15.95]; p < 0.0001) (Table I). When stratified by the type of NIH award won, 24 grants (77%) were R grants, 2 grants (7%) were K grants, 2 grants (7%) were P grants, and 3 grants (10%) were I grants (Table II).

TABLE I - Number of OREF Grant Awardees and Subsequent NIH Awardees Stratified by OREF Grant Award Type
OREF Award No. of OREF Awardees No. of NIH Awardees Percentage of NIH Awardees
Career Development Grant 41 19 46%
Zimmer Orthopaedic Career Development Grant 50 7 14%
Young Investigator Grant 25 2 8%
Career Development Grant in Total Joint and Trauma Surgery 6 0 0%
New Investigator Grant 4 1 25%

TABLE II - NIH Awards Stratified by Type
NIH Grant Type No. of Award Winners
R01 15
R03 5
R13 2
R21 1
R29 1
K02 1
K08 1
P20 1
P50 1
I01 3
Total 31

Back to Top | Article Outline

Awards by Study Type

Of the 126 total OREF grants, 73 were categorized as basic science projects, 45 were categorized as clinical projects, and 8 were categorized as epidemiological projects. Of the basic science projects, 58 were categorized as cell biology and 15 were categorized as biomechanical. Of the 29 OREF grant recipients who later received NIH funding, 24 (83%) received OREF funding for a basic science project and 5 (17%) received OREF funding for a non-basic-science project (Fig. 2). Faculty members with a basic science project were more than 4 times as likely to receive NIH funding than faculty with a non-basic-science project (OR, 4.70 [95% CI, 1.66 to 13.33]; p = 0.0036). There were no differences in study type when comparing Career Development Grant awardees with non-Career Development Grant awardees.

Fig. 2

Fig. 2

Back to Top | Article Outline

Awards by Subspecialty

The proportion of OREF awardees receiving NIH funding was stratified by orthopaedic subspecialty: sports (5 of 17), arthroplasty (5 of 25), oncology (4 of 17), pediatrics (5 of 10), trauma (2 of 17), hand (2 of 8), foot and ankle (2 of 6), shoulder and elbow (2 of 7), and spine (0 of 14) (Fig. 3).

Fig. 3

Fig. 3

Back to Top | Article Outline

Publication Experience and Scholarly Impact of Awardees

Compared with OREF awardees who did not receive NIH funding, OREF awardees who received NIH funding had a significantly higher h-index both at the time of their OREF award, at 14.97 for OREF awardees receiving NIH funding compared with 9.31 for awardees who did not receive NIH funding (p = 0.003), and currently, at 24.48 for OREF awardees receiving NIH funding compared with 16.70 for awardees who did not receive NIH funding (p = 0.002). When comparing faculty receiving NIH funding by grant type, Career Development Grant winners had a significantly higher h-index than non-Career Development Grant winners both at the time of their OREF award, at 17.0 compared with 11.1 (p = 0.03) and currently, at 26.25 compared with 14.50 (p < 0.001). In addition, the mean time from first publication to the OREF grant was significantly longer (p = 0.0006) for those awardees who received NIH funding (12.16 years) than those who did not receive NIH funding (8.26 years). However, there was no significant difference (p = 0.61) in the mean time from first publication to OREF grant when comparing NIH-funded Career Development Grant winners (12.0 years) and NIH-funded non-Career Development Grant winners (13.4 years). OREF awardees obtained NIH funding at a mean time of 6.3 years after their OREF award. There was no difference (p = 0.76) between the time from OREF award to NIH award for winners of Career Development Grants (6.4 years) and winners of non-Career Development Grants (5.8 years).

Back to Top | Article Outline

Discussion

One of the stated goals of the OREF is to provide financial support to a broad variety of orthopaedic researchers to acquire preliminary data needed to apply for federal funding1. Between 1994 and 2014, the OREF awarded 121 faculty-level grants to fund 126 projects: 73 basic science projects, 45 clinical projects, and 8 epidemiological projects. In our analysis, 22% of OREF grant awardees successfully secured NIH funding, with a mean interval of 6.3 years between awards. Furthermore, 46% of Career Development Grant awardees later secured NIH funding. Given the limitation in the number of available NIH grants, our study sought to determine whether specific study types had a higher likelihood of acquiring NIH funding in the future. Interestingly, faculty with basic science projects that were funded by OREF were more than 4 times as likely to obtain NIH grants than faculty who had an OREF grant for a non-basic science project. Although the OREF may not prioritize basic science research, according to the data presented in this study, the NIH may have a preference for funding basic science research in the musculoskeletal area.

These outcomes are important given the current climate of restrictive funding within academic research, particularly concerning NIH grants2,4. In 2014, the NIH received 52,073 grant applications and accepted 9,241 of them, resulting in a 17.7% success rate3,16. The improved success rate of those orthopaedic faculty who have won OREF Career Development Grants, at a rate that is nearly triple the national average, points to the success of the OREF faculty grant programs in identifying promising scientists and supporting them as they establish their research careers.

Other studies within different specialties of medicine have also examined the relationship between NIH funding and their respective specialty-specific foundation funding. Eloy et al. looked at the relationship between full-time faculty members receiving a faculty grant from the American Academy of Otolaryngology-Head and Neck Surgery Foundation (AAO-HNSF) and showed that 39.6% of their investigators eventually later received NIH funding17. This is similar to the 22% of OREF faculty-level grant awardees and 46% of Career Development Grant awardees who eventually acquired NIH funding and reinforces the fact that specialty-specific foundations are crucial in supporting the development of academic research in their respective fields of medicine.

The 5 OREF grants analyzed in this study vary with respect to the amount of funding awarded. The Career Development Grant is given to faculty members with promising research careers to promote a greater commitment to research. The monetary value of this award reflects this, as it amounts to as much as 4 times the value of the other individual grants ($134,000 to $225,000, compared with $50,000). In our analysis, Career Development Grant winners had a significantly higher h-index at the time that they received an OREF grant than non-Career Development Grant winners. It is important to recognize that a high h-index, as a positive predictor of future academic productivity, likely reflects a commitment to academic medicine and is thus self-selecting, particularly when it comes to seeking NIH grant funding. Thus, it is no surprise that even though non-Career Development Grants are awarded more often, winners of the Career Development Grant are nearly 4 times more likely to later receive NIH funding. The fact that OREF awards more of the smaller non-Career Development Grants than the larger Career Development Grants is a reflection of their stated mission to support a broad variety of clinically impactful orthopaedic research. Nonetheless, it appears that the Career Development Grant above all has been an effective funding source for faculty whom the OREF has identified as having promising research and is the OREF grant that is most likely to eventually lead to larger and more prestigious grant funding, such as NIH funding.

There were several limitations to this study. Our investigation only explored the relationship between OREF grants and NIH funding. This may have introduced some selection bias, as those with a commitment to basic science may have traditionally been more likely to submit applications for NIH grants. Although it is one of the largest sources for academic research support, NIH funding is not the only major grant funding resource. The U.S. Department of Defense (DoD), for example, funds a wide spectrum of medical research and provides millions of dollars of support toward orthopaedic research18. In addition, the Agency for Healthcare Research and Quality (AHRQ) and Patient-Centered Outcomes Research Institute (PCORI) are major funding institutions that do not have a basic science funding focus. However, the NIH is considered to be the gold standard for funding medical research, is the largest funder of orthopaedic research, and offers web-based transparency of awards through a searchable database17,19. Therefore, although DoD funding and other federal sources were not considered, we believed that the clarity offered by a study investigating the conversion from an OREF grant to an NIH grant would nonetheless be of value. Future work could expand this project to assess other sources of subsequent funding. We also believe that understanding the relationship between OREF grant funding and NIH funding may provide valuable insight into academic success, and these findings may also be applied to other extramural sources of funding.

It is also important to acknowledge that the findings in this study are historical in nature. They imply correlation between funding types but do not demonstrate causality and thus are limited in their ability to inform future success or prognosticate the likelihood of success of research efforts. As the research landscape evolves and funding organizations become more receptive to patient-centered outcomes and other types of clinical research, funding patterns will likewise change. This relationship would be an important future direction of investigation.

In conclusion, the OREF grant awardee conversion rate of 22% and, particularly, the 46% rate for Career Development Grant winners demonstrate the importance of OREF grants as a source of initial funding for faculty researchers. Grants awarded by the OREF are not only important for supporting orthopaedic surgeons during the years that they are funded, but also achieve their purpose of identifying and supporting young researchers with the Career Development Grant, leading to an NIH funding rate that is nearly triple the 18% national average success rate for NIH funding applications.

Investigation performed at the Department of Orthopaedic Surgery, David Geffen School of Medicine at the University of California Los Angeles, Santa Monica, California

Disclaimer: The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosure: One author of this study (N.M.B.) received a grant (Award Number 5K08AR069112-01) from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health. On the Disclosure of Potential Conflicts of Interest forms, which are provided with the online version of the article, one or more of the authors checked “yes” to indicate that the author had a relevant financial relationship in the biomedical arena outside the submitted work (http://links.lww.com/JBJS/E310).

Back to Top | Article Outline

References

1. Wurth GR. OREF: discovering the future of orthopaedics. J Am Acad Orthop Surg. 2006 Jan;14(1):2-4.
2. LaBeaud AD, McKeating H. The largest drought in American history: funding for science is drying up. PLoS Negl Trop Dis. 2013 Aug 29;7(8):e2351.
3. Garrison HH, Deschamps AM. NIH research funding and early career physician scientists: continuing challenges in the 21st century. FASEB J. 2014 Mar;28(3):1049-58. Epub 2013 Dec 2.
4. Hu Y, Edwards BL, Brooks KD, Newhook TE, Slingluff CL Jr. Recent trends in National Institutes of Health funding for surgery: 2003 to 2013. Am J Surg. 2015 Jun;209(6):1083-9. Epub 2015 Mar 21.
5. NIH RePORTER.https://projectreporter.nih.gov/. Accessed 2017 Apr 3.
6. Röhrig B, du Prel JB, Wachtlin D, Blettner M. Types of study in medical research: part 3 of a series on evaluation of scientific publications. Dtsch Arztebl Int. 2009 Apr;106(15):262-8. Epub 2009 Apr 10.
7. Scopus. 2016.https://www.elsevier.com/solutions/scopus. Accessed 2017 Mar 22.
8. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72. Epub 2005 Nov 7.
9. Schoenfeld AJ, Bhalla A, George J, Harris MB, Bono CM. Academic productivity and contributions to the literature among spine surgery fellowship faculty. Spine J. 2015 Oct 1;15(10):2126-31. Epub 2015 Mar 21.
10. Lopez J, Susarla SM, Swanson EW, Calotta N, Lifchez SD. The association of the h-index and academic rank among full-time academic hand surgeons affiliated with fellowship programs. J Hand Surg Am. 2015 Jul;40(7):1434-41. Epub 2015 May 28.
11. Hunt GE, Cleary M, Walter G. Psychiatry and the Hirsch h-index: the relationship between journal impact factors and accrued citations. Harv Rev Psychiatry. 2010 Jul-Aug;18(4):207-19.
12. Ence AK, Cope SR, Holliday EB, Somerson JS. Publication productivity and experience: factors associated with academic rank among orthopaedic surgery faculty in the United States. J Bone Joint Surg Am. 2016 May 18;98(10):e41-41.
13. DeLuca LA Jr, St John A, Stolz U, Matheson L, Simpson A, Denninghoff KR. The distribution of the h-index among academic emergency physicians in the United States. Acad Emerg Med. 2013 Oct;20(10):997-1003.
14. Babineau M, Fischer C, Volz K, Sanchez LD. Survey of publications and the h-index of academic emergency medicine professors. West J Emerg Med. 2014 May;15(3):290-2. Epub 2013 Nov 5.
15. Stavrakis AI, Patel AD, Burke ZDC, Loftin AH, Dworsky EM, Silva M, Bernthal NM. The role of chairman and research director in influencing scholarly productivity and research funding in academic orthopaedic surgery. J Orthop Res. 2015 Oct;33(10):1407-11. Epub 2015 May 20.
16. U.S. Department of Health & Human Services. NIH research portfolio online reporting tools.https://report.nih.gov/success_rates/Success_ByIC.cfm. Accessed 2017 Mar 22.
17. Eloy JA, Svider PF, Kanumuri VV, Folbe AJ, Setzen M, Baredes S. Do AAO-HNSF CORE grants predict future NIH funding duccess? Otolaryngol Head Neck Surg. 2014 Aug;151(2):246-52. Epub 2014 May 20.
18. Department of Defense - Congressionally Directed Medical Research Programs. Program announcements.http://cdmrp.army.mil/funding/prorp. Accessed 2017 Apr 3.
19. Pool LR, Wagner RM, Scott LL, RoyChowdhury D, Berhane R, Wu C, Pearson K, Sutton JA, Schaffer WT. Size and characteristics of the biomedical research workforce associated with U.S. National Institutes of Health extramural grants. FASEB J. 2016 Mar;30(3):1023-36. Epub 2015 Dec 1.

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2017 by The Journal of Bone and Joint Surgery, Incorporated