Finally, on 29 August 2016, BND searched ClinicalTrials.gov for completed clinical trials using the keywords ‘anaesthesia’ and ‘anesthesia’. The search was limited to the same date range as our searches in PubMed and the Cochrane Database of Systematic Reviews (1 January 2007 to 31 December 2015). BND then conducted the same search, limited to completed trials with results.
To estimate the compliance rate of results reporting in anaesthesia trials, we compared the total number of completed trials with the numbers of trials with reported results. In addition, we examined the reporting of trial results to clinical trial registries by country.
We attempted to eliminate bias in several ways. First, in efforts to minimise selection bias, we conducted our MEDLINE search using a validated search strategy found to be sensitive to the identification of systematic reviews. Second, standardised extraction forms were additional methods used to reduce bias and improve the validity and reliability of study data.
Our primary endpoint was the frequency of clinical trial registry searches reported in systematic reviews. We also measured the number of trials catalogued in clinical trial registries that were possible candidates for inclusion in a subset of systematic reviews from our sample.
Study size and statistical analysis
Our sample was composed of systematic reviews published between 2007 and 2015. This date range was selected because of the 2007 passage of the food and drug administration (FDA) Amendments Act requiring applicable clinical trials involving humans be registered on ClinicalTrials.gov prior to patient enrollment.8
Descriptive statistics were used to summarise data. A χ2 test was used to examine percentage differences between Cochrane systematic reviews and journal-published systematic reviews reporting having searched for unpublished trials. All analyses were conducted using STATA 13.1 (College Station, Texas, USA).
Included systematic reviews
The initial search of PubMed and the Cochrane Database of Systematic Reviews returned 507 and 73 systematic reviews, respectively. Figure 1 shows the subsequent screening and fate of these 580 systematic reviews to leave 415 from the PubMed search and 73 from the Cochrane database.
Of the 415 systematic reviews included from PubMed, 49 (11.8%) reported a search of at least one clinical trial registry for unpublished studies with ClinicalTrials.gov being searched in the majority of these cases (n = 40). In total, 14 reviews searched the international standard registered clinical/social study number (ISRCTN) Registry (controlled-trials.com). The WHO ITCRP was searched in six reviews. Two reviews cited a search of both CenterWatch Clinical Trials Listing Service and controlled-trials.com in addition to a search of ClinicalTrials.gov. Another systematic review reported a search of the UMIN Clinical Trials Registry, and another searched the National Research Register. These four (CenterWatch, controlled-trials.com, UMIN and the National Research Register) are not listed within the WHO registry platform. The British Journal of Anaesthesia had the highest percentage of systematic reviews that included trial registry searches (18.9%, 17/90), followed by the Canadian Journal of Anaesthesia (17.6%, 6/34) and Acta Anaesthesiologica Scandinavica (17.3%, 9/52). Complete results are displayed in Table 2.
Within the sample of 49 PubMed reviews that included a search of clinical trials registries, 12 found potentially usable unpublished data but only five of the reviews included this unpublished data in the final data synthesis. For 19 systematic reviews, we could not determine the extent to which unpublished data was found or incorporated.
We also investigated the frequency of trial registry searches by country. In the PubMed sample, reviews from the United Kingdom contained the largest number of registry searches (31.0%, 15/47). Only 5.6% (4/71) of the reviews from the United States and 9.2% (6/65) of the reviews from Canada reported registry searches. Both reviews from Taiwan and one of the two reviews from Korea reported registry searches. None of the reviews from Germany (22), the Netherlands (14), South Africa (10) and India (5) reported clinical trial registry searches (Fig. 2).
Cochrane Anaesthesia, Critical and Emergency Care Group care group
Reviews conducted by the Cochrane ACE Group featured clinical trial registry searches with greater consistency than reviews published in journals. Of the 73 Cochrane reviews, 43 (58.9%) reported searching registries (Table 2). Of these, 26 found unpublished data and 20 used the unpublished data (Table 2). Most (36 of the 43) searched ClinicalTrials.gov, and 29 searched multiple registries. In total, 29 reported searches of ISRCTN (controlled-trials.com), and eight reported searches of WHO ICTRP. The Cochrane reviews from the United Kingdom and the United States had higher rates of registry searches (10/17 and 5/7, respectively) than reviews published in journals from these countries. χ2 analysis indicated a significant difference between Cochrane reviews and journal-published reviews on the use of trials registries during the search process [P < 0.0001, % difference (95% confidence interval) = 47.1 (34.5 to 58.9)%].
Studies identified via ClinicalTrials.gov
Searches of ClinicalTrials.gov using search terms and date limiters from 30 randomly selected reviews that did not report clinical trial registry searches indicated that many could have located additional studies for consideration had these searches been performed. Although three of these searches did not return any relevant clinical trials, 11 (36.7%) returned more than 100 search results each, with one returning 938.51 These search results were screened and many appeared to be directly applicable to the reviews whose respective search terms retrieved the record (Table 1). The number of potentially relevant studies which could have been considered ranged from 0 to 43 with a median of 3 (IQR: 1 to 8).
ClinicalTrials.gov registry data reporting for anaesthesia
The search of ClinicalTrials.gov for completed clinical trials in anaesthesia from 1 January 2007 to 31 December 2015, yielded 4712 studies. Of these, 727 (15.4%) reported trial results. North American (USA, Canada, Mexico) anaesthesia trials as a whole had results with a reporting rate of 29.4% (459/1563) in ClinicalTrials.gov, with the USA in isolation achieving a rate of 34.5% (439/1274). Trial results reporting, was much lower in other regions: Africa 15% (16/107), East Asia 13.9% (84/603), The Middle East 8.0% (21/263) and Europe 7.3% (111/1524).
Historically, some high-impact factor journals have marginalised studies with nonsignificant or unimportant findings as a credible source of evidence by omitting them from publication. This issue has also been perpetuated by investigators who are more likely to abandon such studies. As the vast physician readership and journal subscribers tend to incorporate published study findings into their decision-making process,3 these omissions may misdirect clinical decision-making and patient care. Such omissions also limit the availability of primary studies for systematic reviews. For the clinical implications of a specific intervention to be accurately assessed, trials with sound methodology should be published regardless of the statistical significance of results.
The real-life consequences of publication bias have been noted to have severe and possibly lethal outcomes for patients. One example discussed by Sutton,54 involved a trial in 1980 in which lorcainide (a Class 1C antiarrhythmic drug) was compared with a placebo in patients with myocardial infarction. The pharmaceutical company that produced the drug halted the trial after nine deaths occurred from the 49 patients in the treatment group and the trial data was never published. Almost a decade later, two trials conducted for related compounds, encainide and flecainide, also resulted in excess patient deaths in the treatment groups. Had the lorcainide trial data been published, these deaths may have been prevented. Although other examples may have less serious implications, the role publication bias plays in understanding the effectiveness of clinical interventions is nonetheless concerning. Systematic reviews, which summarise an intervention's effectiveness, should make use of all available resources to mitigate this form of bias.
Clinical trial registries are a promising source of unpublished data that, placed in the context of a body of literature, may yield a more accurate representation of clinical interventions. The majority of the 415 anaesthesiology systematic reviews neglected to search clinical trial registries. Specifically, only 11.8% of the reviews published in anaesthesiology journals contained searches of clinical trial registries. This proportion is higher than in the clinical neurology literature (6%) reported by Sinnett et al.20, but lower than in the emergency medicine literature (19%).21 The variability of registry searches ranges from 6 in clinical neurology to 35% in general medicine.22 Compared to anaesthesiology journals, systematic reviews reporting the use of a clinical trial registry search in their methodology, as published in high-impact factor general medical journals, was nearly three times greater.12 Among systematic reviews that reported registry searches, many found potentially usable data to consider for inclusion. However, few reviewers used data from registries in the final analysis. Nonetheless, we see promise in searching these registries because other studies have reported inclusion of registry data in systematic reviews.20 Our sample of Cochrane ACE reviews showed a much higher reported rate of trial registry searches than other anaesthesiology reviews. Many sources indicate that Cochrane reviews are held to stricter methodology and are thus better quality,55–58 a finding supported by our results.
We investigated whether it was likely that systematic reviews which did not report trial registry searches would have located additional studies for consideration had such searches been performed. Overall, we found many possible studies that could have been considered for inclusion. The Cochrane Collaboration59 and others60 advocate trial registry searching, yet our results suggest that despite the suggestions from these authoritative groups, authors of systematic reviews often do not perform registry searches as a means to limit publication bias in anaesthesiology reviews.
The compliance rate for reporting results of anaesthesia studies to clinical trials registries was only 15.4%. This finding is consistent with previous research: a 2011 investigation of ClinicalTrials.gov registrations found that 22% of clinical trials had reported results within the 1-year timeline mandated by the US Food and Drug Administration Amendments Act. Phase II trials had lower compliance rates compared with phase III and phase IV trials. Industry-funded trials were more likely to report results than trials funded by other sources.61 A second study of trials completed by September 2008 found that 7.6% of trials posted results on ClinicalTrials.gov by 31 May 2010, with industry-funded trials being three times more likely to postresults.62 A cumulative proportion analysis by days following study completion suggested that 1 year after completion, only 12% of all trials and 14.1% of phase IV trials had reported results.62 More recent evidence suggests that only 13.4% of trials reported summary results within 12 months following trial completion, and of the trials reporting results, median time for reporting trial results was 17 months.63
Given that each year thousands of patients agree to participate in trials that place them at risk, such data matters. It is imperative that, regardless of outcomes, data from trials contribute to the body of evidence. Trial registration promotes greater transparency, which likely improves attitudes toward clinical research. Ultimately, positive perceptions may lead to higher rates of patient participation in trials.64 Clinical trial registration is on the rise globally, increasing exponentially in countries with established registries.65 Recommendations have been made to increase registrations through the development of more geographically available registries and incorporation of these within the WHOs ICTRP.66 However, trial registration must be accompanied by results reporting,62 which we find to be lacking in many countries. Anaesthesia trial results reporting in ClinicalTrials.gov is low: United States (34.4%), Africa (15.0%), East Asia (13.9%), the Middle East (8.0%) and Europe (7.3%).
Several proposals to improve the compliance of results reporting have been discussed.67 The International Committee of Medical Journal Editors has considered mandating data sharing as a precondition for publication among member journals.68 Others have suggested linking mandatory results reporting to funding requirements.69 Doernberg and Wendler67 make a persuasive argument for Institutional Review Boards to take an active role in ensuring trialists comply with reporting results. They recommend that during initial protocol submission, trialists submit a data-sharing plan that includes the nature of results, location or registry where results will be posted, timeline for sharing results and the party responsible for reporting results. Such proposals, if implemented, are likely to have a positive effect on results reporting.
Our study has limitations in both design and execution. First, although we took care in developing a search strategy for systematic reviews, it is possible that some systematic reviews were not retrieved during the search. It is also possible that the inclusion of other databases (e.g. Embase) during the search might have generated additional returns. Our results are also limited to the sample of the included journals and the timeframe of our study, and should not be generalised more broadly. In our screening of potentially eligible trials from registry searches, we may have identified trials that, even if discovered by the authors of original systematic reviews, would not have been included in the data analyses for various reasons; inclusion of such data is a judgement call by the authors of systematic reviews. In some cases, it would have been necessary to contact trialists for additional study details to make more sound determinations. Therefore, our results should be regarded as tentative more than absolute, as the intent was to show the possibility of missed trials rather than the exact number of them. We only searched ClinicalTrials.gov, and it is possible that the WHO ICTRP may have yielded additional studies. Finally, in our analysis of results reporting on ClinicalTrials.gov, we did not take into account the FDA Amendments Act definition of ‘applicable clinical trial’. At present, only applicable clinical trials are required by law to report trial data to ClinicalTrials.gov 1 year after completion. Generally defined, applicable clinical trials include phase II, III or IV trials which used an FDA approved drug, device or biologic, and involved at least one study site in the United States. Ambiguity in the Act, such as statutory terms not being fully defined, has made it difficult to determine whether a trial falls within this definition.70 To address this limitation, the Department of Health and Human Services issued the Notice of Proposed Rulemaking in November 2014 for public comment and reviewed almost 900 comments from individuals and organisations. The Final Rule was made available in September 2016 in the hope of increasing results reporting to ClinicalTrials.gov.70 A more thorough investigation of compliance of applicable trials reporting results to ClinicalTrials.gov would be a welcomed addition to the anaesthesiology literature.
Many choices are available when conducting literature searches for reviews. As well as traditional databases other resources, including trials registries, should be considered to improve the likelihood of identifying all pertinent studies. Current search practices in anaesthesiology reviews should be reevaluated. A better understanding of the extent to which reviewers make use of other possibilities for limiting publication bias would be helpful. Within the anaesthesiology literature, future research could be conducted to determine the extent to which summary effect sizes from systematic reviews are altered when unpublished studies are included. It has been argued that publication bias should not only be evaluated in systematic reviews but also the degree of publication bias in such reviews.71 If only negligible publication bias is found, such that inclusion of unpublished data would not considerably alter outcomes, it would be reasonable to conclude that publication bias is not a cause for concern. However, if the inclusion of unpublished data suggests the intervention is not effective then publication bias should be taken seriously. Such a lack of effect might suggest a reversal of any clinical decisions based on the previously incomplete data analysis. Our results should be interpreted in the context of the specific journals and time frame used in this study, and results may not generalise to all anaesthesia journals.
Acknowledgements relating to this article
Assistance with the analyses: none.
Financial support and sponsorship: none.
Conflicts of interest: none.
1. American Society of Anesthesiologists Task Force on Acute Pain Management. Practice guidelines for acute pain management in the perioperative setting: an updated report by the American Society of Anesthesiologists Task Force on Acute Pain Management. Anesthesiology
2. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA
3. De Oliveira GS, Chang R, Kendall MC. Publication bias in the anesthesiology literature. Anesth Analg
4. Turner EH. Publication bias, with a focus on psychiatry: causes and solutions. CNS Drugs
7. De Angelis C, Drazen JM, Frizelle FA, et al. International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med
8. Food and Drug Administration Amendments Act of 2007. US Public Law 2007:110-185, 21 USC 301.
11. Myles PS. Trial registration for anaesthesia studies. Br J Anaesth
12. Hedin RJ, Umberham BA, Detweiler BN, et al. Publication bias and nonreporting found in majority of systematic reviews and meta-analyses in anesthesiology journals. Anesth Analg
13. Zhang J, Ho K-Y, Wang Y. Efficacy of pregabalin in acute postoperative pain: a meta-analysis. Br J Anaesth
14. Singla NK, Chelly JE, Lionberger DR, et al. Pregabalin for the treatment of postoperative pain: results from three controlled trials using different surgical models. J Pain Res
15. Kim SY, Song JW, Park B, et al. Pregabalin reduces post-operative pain after mastectomy: a double-blind, randomized, placebo-controlled study. Acta Anaesthesiol Scand
16. von Elm E, Altman DG, Egger M, et al. STROBE Initiative. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiology
17. John Wiley & Sons, Lang TA, Altman DG. Moher D, Altman DG, Schulz KF, Simera I, Wager E. Statistical analyses and methods in the published literature: the SAMPL guidelines. Guidelines for reporting health research: a user's manual
18. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ
19. Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol
20. Sinnett PM, Carr B, Cook G, et al. Systematic reviewers in clinical neurology do not routinely search clinical trials registries. PLoS One
21. Keil LG, Platts-Mills TF, Jones CW. Systematic reviews published in emergency medicine journals do not routinely search clinical trials registries: a cross-sectional analysis. Ann Emerg Med
22. Jones CW, Keil LG, Weaver MA, Platts-Mills TF. Clinical trials registries are under-utilized in the conduct of systematic reviews: a cross-sectional analysis. Syst Rev
24. Achuthan S, Singh I, Varthya SB, et al. Gabapentin prophylaxis for postoperative nausea and vomiting in abdominal surgeries: a quantitative analysis of evidence from randomized controlled clinical trials. Br J Anaesth
25. Arulkumaran N, Corredor C, Hamilton M, et al. Treatment-related cardiac complications associated with goal-directed therapy in high-risk surgical patients: a meta-analysis. Crit Care
2013; 17 (Suppl 2):195.
26. Arzola C, Wieczorek PM. Efficacy of low-dose bupivacaine in spinal anaesthesia for Caesarean delivery: systematic review and meta-analysis. Br J Anaesth
27. Bennett MH, Lehm JP, Mitchell SJ, Wasiak J. Recompression and adjunctive therapy for decompression illness: a systematic review of randomized controlled trials. Anesth Analg
28. Bonnet MP, Marret E, Josserand J, Mercier FJ. Effect of prophylactic 5-HT3 receptor antagonists on pruritus induced by neuraxial opioids: a quantitative systematic review. Br J Anaesth
29. Cabrini L, Monti G, Landoni G, et al. Percutaneous tracheostomy, a systematic review. Acta Anaesthesiol Scand
30. Chiumello D, Esquinas AM, Moerer O, Terzi N. A systematic technical review of the systems for the continuous positive airway pressure. Minerva Anestesiol
31. Clark K, Lam LT, Gibson S, Currow D. The effect of ranitidine versus proton pump inhibitors on gastric secretions: a meta-analysis of randomised control trials. Anaesthesia
32. da Silva PS, Fonseca MC. Unplanned endotracheal extubations in the intensive care unit: systematic review, critical appraisal, and evidence-based recommendations. Anesth Analg
33. Engelman E, Marsala C. Efficacy of adding clonidine to intrathecal morphine in acute postoperative pain: meta-analysis. Br J Anaesth
34. Herrera FJ, Wong J, Chung F. A systematic review of postoperative recovery outcomes measurements after ambulatory surgery. Anesth Analg
35. Jennings PA, Cameron P, Bernard S. Ketamine as an analgesic in the pre-hospital setting: a systematic review. Acta Anaesthesiol Scand
36. Katz NP, Paillard FC, Edwards RR. Review of the performance of quantitative sensory testing methods to detect hyperalgesia in chronic pain patients on long-term opioids. Anesthesiology
37. Kossowsky J, Donado C, Berde CB. Immediate rescue designs in pediatric analgesic trials: a systematic review and meta-analysis. Anesthesiology
38. Kumar G, Stendall C, Mistry R, et al. A comparison of total intravenous anaesthesia using propofol with sevoflurane or desflurane in ambulatory surgery: systematic review and meta-analysis. Anaesthesia
39. Martin C, Jacob M, Vicaut E, et al. Effect of waxy maize-derived hydroxyethyl starch 130/0.4 on renal function in surgical patients. Anesthesiology
40. McQuay HJ, Poon KH, Derry S, Moore RA. Acute pain: combination treatments and how we measure their efficacy. Br J Anaesth
41. Mishriky BM, Waldron NH, Habib AS. Impact of pregabalin on acute and persistent postoperative pain: a systematic review and meta-analysis. Br J Anaesth
42. Ong CK, Seymour RA, Lirk P, Merry AF. Combining paracetamol (acetaminophen) with nonsteroidal antiinflammatory drugs: a qualitative systematic review of analgesic efficacy for acute postoperative pain. Anesth Analg
43. Potter LJ, Doleman B, Moppett IK. A systematic review of pre-operative anaemia and blood transfusion in patients with fractured hips. Anaesthesia
44. Schnabel A, Poepping DM, Kranke P, et al. Efficacy and adverse effects of ketamine as an additive for paediatric caudal anaesthesia: a quantitative systematic review of randomized controlled trials. Br J Anaesth
45. Schnabel A, Reichl SU, Kranke P, et al. Efficacy and safety of paravertebral blocks in breast surgery: a meta-analysis of randomized controlled trials. Br J Anaesth
46. Seymour N, Sawh SC. Mega-dose intravenous octreotide for the treatment of carcinoid crisis: a systematic review. Can J Anaesth
47. Spahn DR. Anemia and patient blood management in hip and knee surgery: a systematic review of the literature. Anesthesiology
48. Suppan L, Tramèr MR, Niquille M, et al. Alternative intubation techniques vs Macintosh laryngoscopy in patients with cervical spine immobilization: systematic review and meta-analysis of randomized controlled trials. Br J Anaesth
49. Svircevic V, van Dijk D, Nierich AP, et al. Meta-analysis of thoracic epidural anesthesia versus general anesthesia for cardiac surgery. Anesthesiology
50. Van Der Linden P, James M, Mythen M, Weiskopf RB. Safety of modern starches used during surgery. Anesth Analg
51. Werner MU, Mjöbo HN, Nielsen PR, Rudin A. Prediction of postoperative pain: a systematic review of predictive experimental pain studies. Anesthesiology
52. Wijayasinghe N, Andersen KG, Kehlet H. Neural blockade for persistent pain after breast cancer surgery. Reg Anesth Pain Med
53. Wong J, Lam DP, Abrishami A, et al. Short-term preoperative smoking cessation and postoperative complications: a systematic review and meta-analysis. Can J Anaesth
54. Sutton AJ. Rothstein HR, Sutton AJ, Borenstein M. Evidence concerning the consequences of publication and related biases. Publication bias in meta-analysis: prevention, assessment and adjustments
. Chichester: John Wiley & Sons; 2005. 175–192.
55. Al Faleh K, Al-Omran M. Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews. BMC Pediatr
56. Lundh A, Gøtzsche PC. Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol
57. Moseley AM, Elkins MR, Herbert RD, et al. Cochrane reviews used more rigorous methods than non-Cochrane reviews: survey of systematic reviews in physiotherapy. J Clin Epidemiol
58. Delaney A, Bagshaw SM, Ferland A, et al. The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Crit Care Med
59. Series CB. Cochrane Handbook for Systematic Reviews. John Wiley & Sons, Ltd; 2007: 5.
61. Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ
62. Law MR, Kawasumi Y, Morgan SG. Despite law, fewer than one in eight completed studies of drugs and biologics are reported on time on ClinicalTrials.gov. Health Aff (Millwood)
63. Anderson ML, Chiswell K, Peterson ED, et al. Compliance with results reporting at ClinicalTrials.gov. N Engl J Med
64. John LJ, Muttappallymyalil J. Clinical trial registration in the eastern mediterranean region. Oman Med J
65. Pandey A, Aggarwal A, Maulik M, Seth SD. Clinical trial registration gains momentum in India. Indian J Med Res
2009; 130:85–86. http://www.ncbi.nlm.nih.gov/pubmed/19700807
66. Zeeneldin AA, Taha FM. The Egyptian clinical trials’ registry profile: analysis of three trial registries (International Clinical Trials Registry Platform, Pan-African Clinical Trials Registry and ClinicalTrials.gov). J Adv Res
67. Doernberg SN, Wendler D. Ensuring respect for human research participants: institutional review boards and sharing results from research. JAMA
68. Taichman DB, Backus J, Baethge C. Sharing clinical trial data: a proposal from the International Committee of Medical Journal Editors. Lancet
69. Hudson KL, Collins FS. Sharing and reporting the results of clinical trials. JAMA
70. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov: the final rule. N England J Med
71. Kepes S, Banks GC, McDaniel M, Whetzel DL. Publication bias in the organizational sciences. Organ Res Methods
Supplemental Digital Content
© 2017 European Society of Anaesthesiology