Publication bias can have substantial effects on clinical practice when negative results about a clinical intervention remain unpublished.1 Publication bias is common in anesthesia literature.2,3 Reporting bias refers to selective data reporting based on preference for positive results, while omitting negative findings.4 “Positive” could refer to a finding with P value <0.05 and “negative” could be a finding with P value >0.05. Or, positive could be a finding that a drug or device has benefit, while negative is a finding that a drug or device is associated with harm. Similar to publication bias, outcome reporting bias can have adverse effects on clinical practice when clinically significant benefit or risk is underreported.5 Although reporting bias has been evaluated in other medical specialties, it has yet to be examined in the anesthesiology literature.6
Clinical trial registries were initially created to reduce bias in reports of industry-funded studies.7 A significant aspect of this concern was that a commercial sponsor might initiate many trials, but only publish those with positive results.8,9 In 2005, the International Committee of Medical Journal Editors (ICMJE) launched a policy requiring researchers to provide information about randomized controlled trials (RCTs) before participant enrollment as a necessary condition for publication in member journals.8,9 Anesthesiology journals have followed that recommendation and currently most journals request registration of RCTs to publish study results. Registration of studies allows for the examination of discrepancies between original study design and final reporting of study results in the literature.
The objective of the current investigation is to evaluate discrepancies between original clinical trial registrations and the content of corresponding reports in high-impact anesthesiology journals. Specifically, we examined the rates of discrepancies involving primary outcome, sample size calculation, or study intervention. We hypothesized that more than half of currently published studies would contain a major discrepancy.
The 5 highest-impact factor anesthesiology journals (Anaesthesia, Anesthesia & Analgesia, Anesthesiology, British Journal of Anaesthesia, and Regional Anesthesia and Pain Medicine) were screened for RCTs published in 2013. Journals that examined only pain medicine were not included in the study. A PubMed search was performed in the individual journals by limiting articles to “randomized controlled trials” published during the year 2013. All full-text articles were extracted and re-evaluated to exclude any duplicate publications as well as studies under the title RCTs that were not revealed to be RCTs. Studies were further excluded if they were a secondary analysis of a previous study or if the trial registration was not in English. Selections of included studies were performed independently by 2 investigators. Differences in the selection of included studies were resolved by discussion between the 2 investigators. Nonetheless, those discussions were infrequent (<5% of the included studies).
The clinical trial registration number was obtained from the articles if available or the following registries were searched: ClinicalTrials.gov, the International Standard Randomized Controlled Trial Number Register, and the World Health Organization Clinical Trial Registry Platform (a registry that contains information of multiple other registries and is constantly updated). Data, including primary and secondary outcomes, sample size estimation, interventions, study population, funding source, and country of origin were extracted from the registry and final reports. For manuscripts that did not contain information regarding registration of the study, corresponding authors were contacted and asked about the presence of trial registration. The journals evaluated had different starting dates for enforcing the requirement for prospective registration of clinical trials as presented in Table 1.
The primary outcome was the overall rate of major discrepancies. A major discrepancy was defined as a difference in the content of the manuscripts compared with original entries in the registry in at least one of the 3 areas: primary outcome, target sample size, and study intervention. Those differences were classified as major discrepancies because of the possibility of ethical implications.10 Primary outcome discrepancies were further specified as (1) adding outcomes (a new outcome was introduced in the publication) or omitting outcomes (a registered outcome was omitted in the publication; e.g., myocardial infarction was registered as an outcome but never reported in the manuscript), (2) downgrading a primary outcome to secondary or upgrading a secondary outcome to primary (the published primary outcome was described as secondary in the registry or vice versa), and (3) changing the definition of the outcomes measure (e.g., the registered primary outcomes was myocardial infarction at 1 month, whereas the published primary outcome was myocardial infarction at 3 months). These definitions were similar to studies performed by other investigators in other specialties.11,12 Discrepancies in the primary outcome measures were independently evaluated by 2 investigators. If differences regarding the presence or absence of an outcome discrepancy were noted, a discussion between the 2 investigators resolved those differences. Secondary outcomes were also evaluated following the aforementioned methodology.
Sample size calculation estimated that 35 registered studies would achieve 90% power to detect a 10% discrepancy rate when compared with the null hypothesis and using a 2-sided test and alpha = 0.05. Data are presented as counts (percentages) and analyzed using Fisher exact test. The 95% binomial confidence interval (CI) for the discrepancy rates was calculated using the Jeffreys method. The coverage properties of that method are similar to others, but it has the advantage of being equal-tailed (e.g., for a 95% CI, the probabilities of the interval lying above or below the true value are both close to 2.5%).13 Inter-rater reliability of the assessment of discrepancy in the primary outcome was evaluated by Cohen κ statistic and demonstrated high agreements (κ = 0.88). The discrepancies between registry entries and final reports were compared among journals and between trials with prospective versus retrospective registration (registration before the start of the trial versus during the trial or after the trial has been completed), source of funding (industry-sponsored versus non–industry-sponsored trials), and country of trial origin (United States versus non-United States). Data were analyzed using STATA version 12 (College Station, TX).
A flowchart for the study is presented in Figure 1. Two hundred one articles were included in the final analysis. Among the RCTs published in the evaluated journals, 107 of 201 (53%; 95% CI, 46%–60%) had available registration information. Seventy-one of 107 (66%; 95% CI, 56%–75%) of these trials were prospectively registered. Consequently, 130 of 201 (64%; 95% CI, 57%–71%) published clinical trials were not prospectively registered as recommended by the ICMJE.
Registration rates were significantly lower among studies performed in the United States, 15 of 40 (37%), compared with studies not performed in the United States, 92 of 161 (57%), P = 0.03. Registration rates significantly differed by journal, with 53 of 70 (75%) studies registered in articles published in the British Journal of Anaesthesia, 20 of 34 (57%) in Anesthesiology, 8 of 14 (57%) in Regional Anesthesia and Pain Medicine, 16 of 51 (31%) in Anesthesia & Analgesia, and 10 of 32 (31%) in Anaesthesia, P < 0.001. Among the industry-sponsored studies, 23 of 34 (68%) studies had registration compared with 84 of 167 (50%) non–industry-sponsored studies, P = 0.09.
Fifty-two of 107 (48%; 95% CI, 39%–58%) registered trials had a major discrepancy when the clinical trial registration was compared with the published manuscript. Discrepancy rates were not different with respect to the journals evaluated (P = 0.68), country of study origin (P = 0.18), time of registration (P = 0.31), and industry funding (P = 0.82). Types of discrepancies in study intervention, targeted sample size, and primary outcome are presented in Table 2. Thirty-one of the 46 (67%; 95% CI, 51%–80%) primary outcome discrepancies had changes for the outcome with potential characteristics of reporting bias. Discrepancies on secondary outcomes, inclusion and exclusion criteria, and study group design were also common (Table 2). Only 2 manuscripts discussed the discrepancies between the manuscript content and the information in the trial registry.
The most important finding of our investigation was the high rate of major discrepancies between registered clinical trials and reported manuscripts in anesthesiology journals. Only 4% (2 of 52) manuscripts with major discrepancies alluded to the presence of discrepancies in their discussion sections. More than half of the discrepancies represented potential reporting bias in the primary outcome (i.e., adding/omitting or downgrading/upgrading the original primary outcome). Our results suggest a high prevalence of outcome reporting bias in the anesthesiology literature.
Our results are clinically important because outcome reporting bias can significantly affect clinical practice.5 The selective reporting of only positive findings alters the findings of quantitative systematic reviews that are often used to guide clinical practice.2,14,15 Based on our findings, future systematic reviews should attempt to account for the high prevalence of reporting bias in the anesthesia literature. Correcting for reporting bias has the overall effect of moving estimated treatment effects toward the null, leading to a more conservative assessment of significance.16
Another important finding of the current investigation is that 64% of published clinical trials were not prospectively registered, despite a clear policy from the ICMJE requiring researchers to provide information about randomized controlled trials (RCTs) before participant enrollment as necessary condition for publication in member journals.8,9 Of the 5 journals evaluated, only the British Journal of Anaesthesia enforced clinical trial registration before 2013. All 5 journals now enforce this policy, and reporting rates will likely be higher for subsequent years.
There have been considerable problems with fraud in the anesthesia literature.10,17–19 The use of registries to verify study information may be a helpful way for reviewers and editors to screen for potential cases of misconduct, but this is only possible if the study is prospectively registered. At a minimum, discrepancies noted during the peer-reviewed process should be fully discussed in the manuscripts.
Previous studies evaluating rates of discrepancies among clinical trial registration and published manuscripts in other medical specialties have generated varying rates of discrepancies according to the field examined. Similar to our current study, Rosenthal and Dwan20 found outcome discrepancies in 46% of published articles in 3 high-impact surgical journals. In contrast, Mathieu et al.12 detected only a 31% rate of outcome reporting discrepancy in 3 medical areas (cardiology, rheumatology, and gastroenterology). Different rates between specialties of prospectively versus retrospectively registered trials may explain the observed differences in the rates of discrepancies.
Our study should be interpreted only in the context of its limitations. We cannot evaluate potential causes for a discrepancy observed in the outcomes reported in manuscripts published when compared with trial registries. For the purpose of the current study, we did not consider less specific definitions of outcomes as a discrepancy. For example, if the primary outcome of a manuscript was heart rate >100 beats/minute but the outcome recorded at the trial registry was tachycardia, we did not considered it a discrepancy. Nevertheless, it is possible that a more strict rule to define a discrepancy might result in even higher rates of discrepancies than the ones observed in the current investigation.
In summary, we detected a high rate of major discrepancies between published manuscripts and the original registration of respective clinical trial protocols in high-impact anesthesiology journals. In addition to primary outcome discrepancies, concepts of study design, such as trial intervention, inclusion/exclusion criteria, and sample size estimations, frequently diverged from registered protocols. The vast majority of manuscripts with discrepancies did not discuss the presence of discrepancies. Authors, reviewers, and editors need to diligently enforce clinical trial registration requirements and transparently report and explain deviations between the trial registration and the published results to reduce the negative impact of reporting bias in anesthesiology.
Name: Gildasio S. De Oliveira, Jr., MD, MSCI.
Contribution: This author helped to design and conduct the study, statistical analysis, and manuscript preparation.
Attestation: Gildasio S. De Oliveira, Jr., approved the final manuscript and attests to the integrity of the original data and the analysis reported in this manuscript.
Name: Michael J. Jung, BA.
Contribution: This author helped conduct the study and prepare the manuscript.
Attestation: Michael J. Jung approved the final manuscript and attests to the integrity of the original data.
Name: Robert J. McCarthy, PharmD.
Contribution: This author helped with the study design, statistical analysis, and manuscript preparation.
Attestation: Robert J. McCarthy approved the final manuscript and attests to the integrity of the original data and the analysis reported in this manuscript.
This manuscript was handled by: Steven L. Shafer, MD.
1. Dubben HH, Beck-Bornholdt HP. Systematic review of publication bias in studies on publication bias. BMJ. 2005;331:433–4
2. De Oliveira GS Jr., Castro-Alves LJ, Ahmad S, Kendall MC, McCarthy RJ. Dexamethasone to prevent postoperative nausea and vomiting: an updated meta-analysis of randomized controlled trials. Anesth Analg. 2013;116:58–74
3. Shafer SL, Dexter F. Publication bias, retrospective bias, and reproducibility of significant results in observational studies. Anesth Analg. 2012;114:931–2
4. Møller AM. How to map the evidence: the development of the systematic review in anaesthesia. Br J Anaesth. 2012;109:32–4
5. Chan AW, Krleza-Jerić K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171:735–40
6. Chan AW, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330:753
7. Myles PS. II. Trial registration for anaesthesia studies. Br J Anaesth. 2013;110:2–3
8. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MBInternational Committee of Medical Journal Editors. International Committee of Medical Journal Editors. . Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250–1
9. DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJ, Schroeder TV, Sox HC, Van Der Weyden MBInternational Committee of Medical Journal Editors. International Committee of Medical Journal Editors. . Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA. 2004;292:1363–4
10. Sessler DI, Kurz A. Departmental and institutional strategies for reducing fraud in clinical research. Anesth Analg. 2012;115:474–6
11. Hannink G, Gooszen HG, Rovers MM. Comparison of registered and published primary outcomes in randomized clinical trials of surgical interventions. Ann Surg. 2013;257:818–23
12. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302:977–84
13. Dann RS, Koch GG. Review and evaluation of methods for computing confidence intervals for the ratio of two proportions and considerations for non-inferiority clinical trials. J Biopharm Stat. 2005;15:85–107
14. Liu ZQ, Chen XB, Li HB, Qiu MT, Duan T. A comparison of remifentanil parturient-controlled intravenous analgesia with epidural analgesia: a meta-analysis of randomized controlled trials. Anesth Analg. 2014;118:598–603
15. De Oliveira GS Jr, Castro-Alves LJ, Nader A, Kendall MC, McCarthy RJ. Transversus abdominis plane block to ameliorate postoperative pain outcomes after laparoscopic surgery: a meta-analysis of randomized controlled trials. Anesth Analg. 2014;118:454–63
16. Copas J, Dwan K, Kirkham J, Williamson P. A model-based correction for outcome reporting bias in meta-analysis. Biostatistics. 2014;15:370–83
17. Shafer SL. Tattered threads. Anesth Analg. 2009;108:1361–3
18. Shafer SL. Shadow of doubt. Anesth Analg. 2011;112:498–500
19. Shafer SL. Editor’s note: notices of retractions. Anesth Analg. 2013;116:739
20. Rosenthal R, Dwan K. Comparison of randomized controlled trial registry entries and content of reports in surgery journals. Ann Surg. 2013;257:1007–15