Plus ça change, plus c’est la même chose. [The more things change, the more they stay the same.]
—Jean-Baptiste Alphonse Karr (Les Guêpes, January 1849)
Thirty-one years ago, the U.S. Public Health Service Task Force on Women’s Health Issues released a report outlining why the low representation of women in clinical trials had led to suboptimal women’s health care.1 In response, the National Institutes of Health (NIH) released a 1986 policy advising scientists to examine sex/gender differences in all clinical trial outcomes.2 With the NIH Revitalization Act of 1993, this policy became law.3 The act requires NIH-funded researchers to enroll women and ethnic/racial minorities, including women of childbearing age, into clinical research trials, and forbids the use of cost as a reason for their exclusion.3 Guidelines (amended in 2001) require that investigators discuss the inclusion of women and minorities in their applications for clinical research funding, report to the NIH the female and minority enrollment in funded studies, and conduct preliminary trials that provide enough information to inform the development of phase III trials, which are also required to test for sex differences as appropriate based on prior research.4 The ultimate goal of these recommendations is to ensure that all people, regardless of sex/gender or race/ethnicity, will fully benefit from advances in biomedical science.5,6
In 2009, the NIH convened an internal task force to determine best practices for monitoring and tracking the inclusion of women and minorities, which led to the appointment of an Inclusion Policy Officer to oversee enrollment and ensure that inclusion guidelines are adequately followed.2 The latest biannual inclusion report indicated that NIH clinical research enrollees were 57.1% female (51.8% in non-sex-specific studies) and 26.2% were from racial/ethnic minority groups. The report concluded that the NIH has succeeded in increasing representation of women and minorities in clinical research.7
Notwithstanding this achievement, a 2015 Government Accountability Office (GAO) report highlighted drawbacks to current tracking mechanisms that make it difficult to assess adherence to NIH recommendations on inclusion.5 Two critical shortcomings were the aggregation of biannual report numbers across NIH Institutes and Centers, which makes it difficult to discern whether specific areas are failing to meet inclusion requirements, and the lack of adequate reporting mechanisms at the NIH to evaluate whether or not researchers are analyzing and reporting outcomes by sex.5
The GAO report recommends evaluating the peer-reviewed publications of NIH-funded research to monitor compliance with policies on inclusion, analysis, and reporting.5 Our group has conducted two prior studies examining the inclusion of women and minorities in NIH-funded randomized controlled trials (RCTs).8,9 According to our findings, in 2004, 33% of sampled publications analyzed data by sex or explained why they did not, and in 2009, only 36% reported this information.8,9 This was the case for race/ethnicity in 18% of 2004 publications and 21% of 2009 publications.8,9 Further, as compared with 2004, our 2009 analysis found no significant change in enrollment of women or individuals from underrepresented racial/ethnic groups into clinical trials.8,9
Since the publication of our 2009 study, other researchers have examined the inclusion of women and minorities as well as reporting by sex or race/ethnicity in specific fields (e.g., HIV, cardiology)10,11 and in particular clinical trial types (e.g., phase III clinical trials),12,13 finding similarly inadequate adherence to NIH policies. The purpose of this study was to repeat our evaluation of NIH-funded RCTs in a broad sample of high-impact journals published in 2015 to determine whether rates of inclusion and reporting have changed since 2009, more than two decades since the passage of the NIH Revitalization Act.
We identified RCTs published in 2015 through a PubMed search of 14 of the leading U.S. medical journals in the fields of general internal medicine and a wide range of medical specialty and subspecialty areas. Journals were selected if they published at least 25 RCTs in 2015 and had a 2014 impact factor in the top five for that medical specialty or subspecialty. Impact factor is a rating determined by Journal Citation Reports using the frequency with which articles are cited in a given year (retrieved from ISI Web of Science).14 Our journal selection criteria were intended to identify the leading journals that regularly publish RCTs. Though not necessarily representative of the entire spectrum of RCTs, studies published in these journals may be more accessible to clinicians and the general public compared with less well-known or less widely read journals. Our PubMed search parameters included “randomized controlled trial,” English language, human subjects, and publication in calendar year 2015. In cases where an article was published online and also in print, we used the date of print publication for eligibility (see Table 1 for the journal names, impact factors, number of RCTs published in 2015, and number of eligible studies included in this analysis).
Several of the authors (A.R.K., P.R., A.F., E.H.) reviewed all articles meeting these criteria, identifying funding sources and dates of study recruitment. Letters, Brief Communications, and clinical trials that started recruitment before 1994, as well as studies with no NIH support, were excluded. We also excluded studies that were secondary reviews of previously completed RCTs, combined data from various RCTs, analyzed a subset of study subjects, included only subjects from outside the United States, or analyzed or randomized at a unit other than the individual.
For analysis of sex-based reporting, we excluded studies from obstetrics–gynecology journals as well as those that enrolled only males or females. We did not exclude studies on conditions that may disproportionately affect members of one sex (e.g., autoimmune diseases). Similarly, studies focused on veteran populations were only excluded if they addressed a condition found only in men (e.g., prostate cancer). We analyzed articles for the inclusion of women and minorities and for reporting of sex-specific and race/ethnicity-specific results. We further recorded whether sex and/or race or ethnicity were taken into consideration during outcomes analysis and whether the authors mentioned the potential impact of sex, race, or ethnicity on the findings and/or their generalizability. Obstetrics–gynecology articles were evaluated for race/ethnicity-specific results only.
The abstract, text, and tables of each eligible manuscript were reviewed. In addition, we examined any follow-up articles or commentaries published by the author(s) or other researcher(s) for information related to sex, race, and ethnicity. When an article’s reporting of sex-specific or race/ethnicity-specific analyses or results was ambiguous, multiple reviewers on our research team evaluated and discussed the article until we reached consensus. The absolute numbers and percent distribution of samples across sex and race/ethnicity were recorded for all articles. We included absolute numbers because sample size drives the ability to find statistical significance. We compared our findings against our 2004 and 2009 findings using chi-square and Fisher exact tests as appropriate. The race/ethnicity portion of the analysis is limited to black and Hispanic subgroup reporting, as other racial/ethnic minorities were rarely and inconsistently reported. All analyses were conducted using SAS statistical software, version 9.3 (SAS Institute, Cary, North Carolina).
To determine whether there was any relationship between the likelihood of adhering to NIH guidelines for inclusion of women in clinical research and the gender of the person leading the research, we assessed the gender of the first and senior (last) authors of studies in our final data set. We assessed gender using the author’s name and completing an Internet search to find photographs and text containing gendered pronouns (e.g., “She did her undergraduate work at …”) to verify the gender of the first and senior authors.
We identified 782 studies in the 14 journals that met our search criteria. Of these, 142 studies were eligible for this analysis (Figure 1). The most common reasons for exclusion were absence of NIH funding for the study, participants residing solely outside the United States, and reported results limited to a secondary or subgroup analysis of a parent study (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/A502).
Of the 142 studies, 13 were published in obstetrics–gynecology journals and were excluded from the analysis for reporting of sex differences. Of the remaining 129 studies, 22 enrolled only one sex, either because the study addressed a condition or disease that affects only one sex (e.g., pregnancy, prostate cancer) or tested an intervention targeted to one sex (e.g., estrogen modulation in Alzheimer disease, depression screening in obstetrics–gynecology clinics). Four (3.1%) of these sex-specific studies were restricted to male subjects, and 18 (14.0%) were restricted to female subjects.
Among the 107 studies that were not sex specific, the majority of studies (87; 81.3%) enrolled ≥ 30% women, and mean and median proportion of female enrollment was 47.3% and 46.0%, respectively. However, 16 (15.0%) of these studies enrolled less than 30% women, and, of those, 7 (6.5%) enrolled less than 15% women without explanations. Four studies (3.7%) that were not explicitly sex specific did not report the number of male or female subjects or offer reasons for not reporting the sex of their participants.
There was no statistical improvement since 2009 in the proportion of studies that addressed and reported sex in their analyses. Twenty-eight of the 107 studies (26.2%) reported at least one outcome by sex or explicitly included sex as a covariate in statistical analysis (Table 2). Two additional studies (1.9%) did not include sex in their analyses, but explained why they did not, a statistically significant decrease compared with 2009 (P = .01). The remaining 77 studies (72.0%) did not mention whether sex was included in their analysis, did not report any sex-specific outcomes, and did not provide explanation. In only 4 of the 16 studies (25%) with female enrollment below 30% did the authors note that their findings may not be generalizable to women. We found follow-up publications (e.g., letters, editorials) for 9 of the 16 studies with < 30% females enrolled; the only follow-up publication to comment on the low female enrollment was an editorial about a study of active-duty U.S. Army soldiers.
Among the 107 studies that included both sexes, 38 studies (35.5%) had a female lead author, and 68 (63.6%) had a male lead author. The gender of the lead author could not be determined for one study. There was no difference by author gender in the proportion of studies that analyzed or reported their outcomes by sex: 10 of the 38 studies (26.3%) with female lead authors compared with 20 of the 68 studies (29.4%) with male lead authors. Likewise, 6 of the 27 studies (22.2%) with female senior authors reported sex-specific outcomes compared with 24 of 80 studies (30.0%) with male senior authors.
There were no studies of a disease or condition that affected only individuals from a specific racial or ethnic background. The majority of studies reported the racial and/or ethnic composition of their participants (120; 84.5%) as well as the number of white or black participants (115 [81.0%] and 98 [69.0%], respectively), but Hispanic enrollment was not reported in over half of the studies (Table 3). Only 4 studies (2.8%) were conducted within a single racial or ethnic group. Among the 31 studies with < 10% black participants, 9 (29.0%) had follow-up published commentary, but none of the follow-up commentary mentioned limitations by race or ethnicity.
Analysis and reporting of study outcomes by race/ethnicity continued to be uncommon in 2015, with less than 5% (7) of studies reporting primary outcomes by race/ethnicity. In total, 19 studies (13.4%) either reported outcomes by race and/or ethnicity or included it in their analyses, a proportion statistically indistinguishable from that of earlier years. Similarly, the percentages of studies that did not report race- and/or ethnicity-specific outcomes or include race/ethnicity in their analyses in 2015 did not change as compared with 2004 and 2009.
In this study of peer-reviewed articles published within 2015 in high-impact journals and describing NIH-funded RCTs, we found that 15% of studies continued to enroll fewer than 30% women. Further, we found an overall underperformance in adherence to NIH guidelines in the analysis and reporting of women and individuals from racial and ethnic minority groups. There has been a slight increase, from 43% to 46%, in the median number of women enrolled in RCTs since 2004.
Our findings suggest that merely enrolling women and nonwhite participants in clinical studies does not translate to analysis and reporting by sex or race/ethnicity. More than two decades since passage of the NIH Revitalization Act, 72% (77) of studies in our review did not include sex in their analyses or provide explanations as to why not. As stated in the 2015 GAO report, the lack of adequate NIH reporting mechanisms to evaluate whether or not researchers analyze outcomes by sex remains a critical issue.5 In addition, reporting of the racial/ethnic composition of study participants did not improve since 2004.
Research on institutional equity directives and actual practice shows that formal policies often paradoxically result in greater bias because they diminish the onus on individuals to promote equity.15,16 Thus, it is possible that having an overt NIH policy requiring the inclusion and reporting of data from women and minorities may inadvertently relieve investigators of the responsibility of actually conducting equitable research. It is also not surprising that we found that men and women researchers performed equally poorly in inclusion, analysis, and reporting of women in the studies they conducted. In nearly all controlled experiments, men and women participants are found to demonstrate the same degree of gender bias in judgment and decision making.17,18
There is a growing consensus that the results of clinical trials should be analyzed and reported for male/men and female/women subjects separately.19–21 A Lancet panel19 proposes, and we support, that publishers work with funders to offer clear guidance to editors and authors on reporting results by sex and/or gender in clinical trials. The guidelines include correct use of the terms “sex” and “gender”; reporting sex and gender, where appropriate, for study participants, animals, and cells; analyzing data by sex and gender or making the raw data accessible for analysis; and studying the influence of sex and gender on results. Further, if sex and/or gender are not reported or analyzed, authors should justify this within the article. Regarding the correct usage of “sex” and “gender,” “sex” refers to the genetic differences between males and females, and “gender” is socially constructed. However, the bidirectional interactions between biological and sociobehavioral processes are complex. Therefore, for convenience, we have used “sex/gender” throughout this manuscript.
As the Lancet panel19 notes, failure to account for sex and/or gender may mask differences in outcomes for men and women, limit researchers’ ability to reproduce study findings, and contribute to less than adequate patient care. For example, an RCT of naltrexone for drug and alcohol dependence found that it decreased use and severity in men but increased use and severity in women.22 Had these results not been analyzed and reported by sex, the apparent effect of the drug would have been attenuated, and neither men nor women would have received appropriate care as a result.
Of the 14 journals studied, only the Journal of the American College of Cardiology has specific guidelines for authors to provide gender-specific data in reporting outcomes of epidemiologic analyses or clinical trials. Diabetes Care advises authors to report whether sex was included in statistical analysis, but does not require results to be reported by sex. None of the journals had a guideline for authors to report clinical trial results by race or ethnicity. The 2010 CONSORT Statement for RCTs23 does not include reporting results by sex/gender, race, or ethnicity.
The primary limitation of this study is that by selecting studies published in 14 journals, we do not examine reports from the full spectrum of NIH-funded RCTs. Because of this, our findings regarding enrollment, analysis, and reporting of women and minorities in RCTs may not be representative of the complete scope of NIH-funded RCTs and may differ from the rates reported in the NIH biannual inclusion report. However, we did select the leading 14 journals that publish RCTs from the fields of general internal medicine and a wide range of medical specialty and subspecialty areas. Our analysis included one large HIV-related international study, which may not reflect the pattern of recruitment of domestic studies.
We recommend the following remedial actions: open discussion among all members of research teams regarding implicit sex/gender and race bias that may affect participant recruitment and data analysis; creation of a sense of urgency by principal investigators leading clinical trials about the importance of broad representation among participants from the standpoints of equity and good science; strong author guidelines for reporting clinical trial results by sex, race, and ethnicity within peer-reviewed journals; and revision of NIH grant policy to score grant applications on their plans for representation, recruitment strategies, and whether they will be powered to detect differences between groups.
Our findings indicate that federal law alone is insufficient to change practice in the conduct of NIH-funded RCTs regarding the inclusion, analysis, and reporting of women and ethnic/racial minorities. The lack of progress toward the goal of better understanding sex-specific and race/ethnicity-specific differences in biomedical research limits our ability to provide the best clinical care to all people.
1. Public Health Service Task Force on Women’s Health. Report of the Public Health Service Task Force on Women’s Health Issues. 1985.Vol. 100. Washington, DC: Public Health Service
2. National Institutes of Health. Office of Research on Women’s Health. Report of the Advisory Committee on Research on Women’s Health, Fiscal Years 2013–2014: Office of Research on Women’s Health and NIH Support for Research on Women’s Health. 2015.Bethesda, MD: National Institutes of Health
3. National Institutes of Health. NIH guidelines on the inclusion of women and minorities as subjects in clinical research. Fed Regist. 1994;59:14508–14513.
5. U.S. Government Accountability Office. Better Oversight Needed to Help Ensure Continued Progress Including Women in Health Research. Rep to Congr Requesters. 2015; Washington, DC: Government Accountability Office; GAO-16-13.
6. Oh SS, Galanter J, Thakur N, et al. Diversity in clinical and biomedical research: A promise yet to be fulfilled. PLOS Med. 2015;12:e1001918.
7. National Institutes of Health. Office of Research on Women’s Health. Appendix E: Aggregate enrollment data tables and trend data. In: Report of the Advisory Committee on Research on Women’s Health, Fiscal Years 2013–2014: Office of Research on Women’s Health and NIH Support for Research on Women’s Health. 2015.Bethesda, MD: National Institutes of Health
8. Geller SE, Koch A, Pellettieri B, Carnes M. Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: Have we made progress? J Womens Health (Larchmt). 2011;20:315–320.
9. Geller SE, Adams MG, Carnes M. Adherence to federal guidelines for reporting of sex and race/ethnicity in clinical trials. J Womens Health (Larchmt). 2006;15:1123–1131.
10. Curno MJ, Rossi S, Hodges-Mameletzis I, Johnston R, Price MA, Heidari S. A systematic review of the inclusion (or exclusion) of women in HIV research: From clinical studies of antiretrovirals and vaccines to cure strategies. J Acquir Immune Defic Syndr. 2016;71:181–188.
11. Chen MS Jr, Lara PN, Dang JH, Paterniti DA, Kelly K. Twenty years post-NIH Revitalization Act: Enhancing minority participation in clinical trials (EMPaCT): Laying the groundwork for improving minority clinical trial accrual: Renewing the case for enhancing minority participation in cancer clinical trials. Cancer. 2014;120(suppl 7):1091–1096.
12. Foulkes MA. After inclusion, information and inference: Reporting on clinical trials results after 15 years of monitoring inclusion of women. J Womens Health (Larchmt). 2011;20:829–836.
13. Nolan MR, Nguyen TL. Analysis and reporting of sex differences in phase III medical device clinical trials—How are we doing? J Womens Health (Larchmt). 2013;22:399–401.
15. Castilla EJ, Benard S. The paradox of organizations. Science. 2010;55:543–576.
16. Kaiser CR, Major B, Jurcevic I, Dover TL, Brady LM, Shapiro JR. Presumed fair: Ironic effects of organizational diversity structures. J Pers Soc Psychol. 2013;104:504–519.
17. Valian V. Why So Slow? The Advancement of Women. 1999.Cambridge, MA: MIT Press
18. Isaac C, Lee B, Carnes M. Interventions that affect gender bias in hiring: A systematic review. Acad Med. 2009;84:1440–1446.
19. Schiebinger L, Leopold SS, Miller VM. Editorial policies for sex and gender analysis. Lancet. 2016;388:2841–2842.
20. Leopold SS, Beadling L, Dobbs MB, et al. Editorial: Fairness to all: Gender and sex in scientific reporting. Clin Orthop Relat Res. 2014;472:391–392.
21. Clayton JA, Tannenbaum C. Reporting sex, gender, or both in clinical research? JAMA. 2016;316:1863–1864.
22. Pettinati HM, Kampman KM, Lynch KG, et al. Gender differences with high-dose naltrexone in patients with co-occurring cocaine and alcohol dependence. J Subst Abuse Treat. 2008;34:378–390.
23. Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;8:18.
Supplemental Digital Content
© 2018 by the Association of American Medical Colleges