Institutional members access full text with Ovid®

Share this article on:

Analysis of National Institutes of Health R01 Application Critiques, Impact, and Criteria Scores: Does the Sex of the Principal Investigator Make a Difference?

Kaatz, Anna PhD, MPH; Lee, You-Geon PhD; Potvien, Aaron MS; Magua, Wairimu PhD, MS; Filut, Amarette; Bhattacharya, Anupama; Leatherberry, Renee; Zhu, Xiaojin PhD, MS; Carnes, Molly MD, MS

doi: 10.1097/ACM.0000000000001272
Research Reports

Purpose: Prior text analysis of R01 critiques suggested that female applicants may be disadvantaged in National Institutes of Health (NIH) peer review, particularly for renewals. NIH altered its review format in 2009. The authors examined R01 critiques and scoring in the new format for differences due to principal investigator (PI) sex.

Method: The authors analyzed 739 critiques—268 from 88 unfunded and 471 from 153 funded applications for grants awarded to 125 PIs (76 males, 49 females) at the University of Wisconsin–Madison between 2010 and 2014. The authors used seven word categories for text analysis: ability, achievement, agentic, negative evaluation, positive evaluation, research, and standout adjectives. The authors used regression models to compare priority and criteria scores, and results from text analysis for differences due to PI sex and whether the application was for a new (Type 1) or renewal (Type 2) R01.

Results: Approach scores predicted priority scores for all PIs’ applications (P < .001), but scores and critiques differed significantly for male and female PIs’ Type 2 applications. Reviewers assigned significantly worse priority, approach, and significance scores to female than male PIs’ Type 2 applications, despite using standout adjectives (e.g., “outstanding,” “excellent”) and making references to ability in more critiques (P < .05 for all comparisons).

Conclusions: The authors’ analyses suggest that subtle gender bias may continue to operate in the post-2009 NIH review format in ways that could lead reviewers to implicitly hold male and female applicants to different standards of evaluation, particularly for R01 renewals.

A. Kaatz is director of computational sciences, Center for Women’s Health Research, University of Wisconsin–Madison, Madison, Wisconsin.

Y.G. Lee is associate researcher, Wisconsin Center for Education Research, University of Wisconsin–Madison, Madison, Wisconsin.

A. Potvien is a doctoral candidate, Department of Statistics, and researcher, Health Innovation Program, University of Wisconsin–Madison, Madison, Wisconsin.

W. Magua is postdoctoral research associate, Center for Women’s Health Research, University of Wisconsin–Madison, Madison, Wisconsin.

A. Filut is research assistant, Center for Women’s Health Research, University of Wisconsin–Madison, Madison, Wisconsin.

A. Bhattacharya is an undergraduate student and data science scholar, Center for Women’s Health Research, University of Wisconsin–Madison, Madison, Wisconsin.

R. Leatherberry is staff researcher, Center for Women’s Health Research, University of Wisconsin–Madison, Madison, Wisconsin.

X. Zhu is associate professor, Department of Computer Science, University of Wisconsin–Madison, Madison, Wisconsin.

M. Carnes is director, Center for Women’s Health Research, professor in the Departments of Medicine, Psychiatry, and Industrial and Systems Engineering, University of Wisconsin–Madison, and part-time physician, William S. Middleton Veterans Hospital, Madison, Wisconsin.

Funding/Support: This research was funded by the UW-Madison Department of Medicine and the National Institutes of Health grant #R01 GM111002.

Other disclosures: None reported.

Ethical approval: The UW-Madison institutional review board approved all aspects of this study. Protocol #SBS2012-1177.

Supplemental digital content for this article is available at http://links.lww.com/ACADMED/A369.

Correspondence should be addressed to Anna Kaatz, 700 Regent St., Suite #301, University of Wisconsin–Madison, Madison, WI 53715; telephone: (608) 263-9770; e-mail: akaatz@wisc.edu.

© 2016 by the Association of American Medical Colleges