Secondary Logo

Journal Logo

Trial Registration and Outcome Reporting: A Bait and Switch?

Nizamuddin, Junaid MD; Nizamuddin, Sarah Latif MD; Shahul, Sajid MD, MPH

doi: 10.1213/ANE.0000000000002371
Editorials: Editorial
Free

From the Department of Anesthesia and Critical Care, University of Chicago, Chicago, Illinois.

Accepted for publication June 26, 2017.

Funding: None.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Sajid Shahul, MD, MPH, 5841 S Maryland Ave MC 4028, Chicago, IL 60637. Address e-mail to sajid.shahul@gmail.com.

It does not happen infrequently: a randomized controlled trial (RCT) is designed, primary and secondary outcomes are established, and the trial gets underway. As results trickle in, however, a researcher’s yearning to discover a positive effect or desired outcome may influence an alteration of the original study design. Perhaps new secondary outcomes are added, the primary outcome is changed, or sample size adjusted. Perhaps the primary and secondary outcomes are switched. The result? A new study that presents more “acceptable” results to the researcher and more desirable results to the journal in which it may be published.

RCTs provide the strongest evidence base for clinical decision making and enhance the quality of care delivered by clinicians. Often considered the gold standard for investigational studies, RCTs establish causality, minimize selection bias, and account for unmeasured cofounders. However, RCTs are prone to several types of bias, including publication bias and reporting bias. Publication bias refers to the selective submission and publication of studies with a statistically significant difference in outcome, and selective nonsubmission and nonpublication of studies that result in nonsignificant outcomes or outcomes detrimental to the studies’ authors and/or sponsors. Simply put, it is the notion that positive studies get published more often than negative studies. Reporting bias refers to the selective reporting of outcomes, often reporting and publishing outcomes with statistically significant results and omitting outcomes with results that are not statistically significant.

Trial registration in advance of patient enrollment helps minimize publication bias and reporting bias. Investigators are required to declare their intended goals and methodology, including primary and secondary outcomes and sample size. In 2004, the International Committee of Medical Journal Editors (ICMJE) issued a statement mandating that all prospective clinical trials commencing after July 1, 2005, be registered in a public clinical trials registry to be published by any of the ICMJE member journals.1 This was heralded as a landmark step in enhancing the integrity of clinical research. Investigators would have to declare their intent in advance, with trial registries available to be searched by journals’ editorial boards and readership alike. The top anesthesia journals adapted this recommendation several years later, with Anesthesiology and Anesthesia & Analgesia issuing mandates for trial registration for clinical trials that commenced after 2013 and 2014, respectively.

In the time since these declarations, however, these lofty standards have largely not yet been met. Prior work has shown that the anesthesiology literature—and the medical literature at large—remains plagued by bias. While trial registration rates have clearly increased, whether trials are appropriately reported as registered remains an entirely separate issue altogether. De Oliveira et al2 previously investigated publication bias in the anesthesia literature published during 2008–2009. In this study published in Anesthesia & Analgesia, the authors evaluated 1163 studies in 14 anesthesiology journals, finding that 72% of studies published in the 4 journals with the highest impact factor had positive findings, compared with 53% of studies in the remaining 10 lower impact factor journals. In a study published last year, Chong et al3 examined the fate of 1052 RCTs presented as abstracts at the American Society of Anesthesiology annual meetings from 2001 to 2004. The authors found that just over half of these abstracts proceeded to publication, with abstracts with positive results proceeding to publication at a higher rate than abstracts with negative findings (relative risk 1.42; 95% confidence interval, 1.22–1.66; P < .001). In addition to this publication bias, reporting bias is also prevalent in the anesthesia literature. In a study also published in Anesthesia & Analgesia, De Oliveira et al4 examined 210 studies published in 2013 from the 5 highest impact factor anesthesiology journals, finding that 64% of the studies had not been prospectively registered, and among registered trials, almost half had a major discrepancy between the published manuscript and the clinical trial registration.

In this issue of Anesthesia & Analgesia, Jones et al5 publish the results of a study that greatly expands on this previous study and highlight the prevalence of inadequate registration in the anesthesiology literature. The authors examined RCTs published in the top 6 anesthesiology journals by impact factor longitudinally in the years 2007, 2010, 2013, and 2015 with the goal of examining trends in adequate trial registration and outcome reporting discrepancies. The rate of adequate trial registration, defined as registration before enrollment of the first subject and identification of a clearly defined primary outcome, increased over the study period, with 38% of RCTs adequately registered in 2015. The authors found a striking rate of outcome discrepancies among the RCTs examined. Even in 2015, 92% of adequately registered trials had at least 1 primary or secondary outcome discrepancy, over half of which favored statistical significance.

While these findings should certainly be of concern, this is hardly unique to our specialty. Similar rates of trial registration and outcome reporting compliance have been shown in leading neurology,6 emergency medicine,7 and surgery8 journals, among others.9–13 Recently, transparency requirements have gone ever further than simply mandating registration in advance of enrollment; applicable clinical trials are now required to report results to the clinicaltrials.gov website,14 and the ICMJE recently mandated that all manuscripts submitted of clinical trials must include a statement on data sharing. While these are certainly positive developments in terms of encouraging even more transparency, it remains to be seen how stringently these requirements will be followed. Encouraging researchers to stay within the initially established trial design and assigned primary and secondary outcomes will require a culture change among the scientific community which encourages and rewards appropriate trial registration and reporting.

Perhaps the most concerning finding raised by this study is the unexpected one (of note, appropriately reported as a post hoc secondary outcome)—the number of RCTs published in the top anesthesiology journals steadily declined from 2007 to 2015. While the authors speculate that there are several potential reasons for this, including a lack of research funding and a shift in publication of RCTs to other journals, the finding is nonetheless concerning. Could the decrease in RCTs published be the result of fewer RCTs being performed in the era of increased regulation and scrutiny? Are investigators being pushed to do observational studies that are far less burdensome to perform—and do not require advance public registration? While these conclusions can certainly not be made as a result of this study—an observational one—they must be kept in mind as regulation is increased in the noble pursuit of transparency.

Back to Top | Article Outline

DISCLOSURES

Name: Junaid Nizamuddin, MD.

Contribution: This author helped draft and edit the article and approved the final version of the article.

Name: Sarah Latif Nizamuddin, MD.

Contribution: This author helped draft and edit the article and approved the final version of the article.

Name: Sajid Shahul, MD, MPH.

Contribution: This author helped draft and edit the article and approved the final version of the article.

This manuscript was handled by: Nancy Borkowski, DBA, CPA, FACHE, FHFMA.

Back to Top | Article Outline

REFERENCES

1. De Angelis C, Drazen JM, Frizelle FA, et al.; International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250–1251.
2. De Oliveira GS Jr, Chang R, Kendall MC, Fitzgerald PC, McCarthy RJ. Publication bias in the anesthesiology literature. Anesth Analg. 2012;114:1042–1048.
3. Chong SW, Collins NF, Wu CY, Liskaser GM, Peyton PJ. The relationship between study findings and publication outcome in anesthesia research: a retrospective observational study examining publication bias. Can J Anaesth. 2016;63:682–690.
4. De Oliveira GS Jr, Jung MJ, McCarthy RJ. Discrepancies between randomized controlled trial registry entries and content of corresponding manuscripts reported in anesthesiology journals. Anesth Analg. 2015;121:1030–1033.
5. Jones PM, Chow JTY, Arango MF, et al. Comparison of registered and reported outcomes in randomized clinical trials published in anesthesiology journals. Anesth Analg. 2017;125:1292–1300.
6. Rayhill ML, Sharon R, Burch R, Loder E. Registration status and outcome reporting of trials published in core headache medicine journals. Neurology. 2015;85:1789–1794.
7. Jones CW, Platts-Mills TF. Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012;60:458–64.e1.
8. Rosenthal R, Dwan K. Comparison of randomized controlled trial registry entries and content of reports in surgery journals. Ann Surg. 2013;257:1007–1015.
9. Wayant C, Scheckel C, Hicks C, et al. Evidence of selective reporting bias in hematology journals: a systematic review. PLoS One. 2017;12:e0178379.
10. Sims MT, Sanchez ZC, Herrington JM, et al. Shoulder arthroplasty trials are infrequently registered: a systematic review of trials. PLoS One. 2016;11:e0164984.
11. Huser V, Cimino JJ. Evaluating adherence to the International Committee of Medical Journal Editors’ policy of mandatory, timely clinical trial registration. J Am Med Inform Assoc. 2013;20:e169–e174.
12. Viergever RF, Karam G, Reis A, Ghersi D. The quality of registration of clinical trials: still a problem. PLoS One. 2014;9:e84727.
13. Hartung DM, Zarin DA, Guise JM, McDonagh M, Paynter R, Helfand M. Reporting discrepancies between the ClinicalTrials.gov results database and peer-reviewed publications. Ann Intern Med. 2014;160:477–483.
14. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov—the final rule. N Engl J Med. 2016;375:1998–2004.
Copyright © 2017 International Anesthesia Research Society