Although the trim and fill method showed evidence of PB in 36 of 45 analyses (80.0%), the Egger regression results indicated PB in 51.1% of analyses (n = 23). Five analyses from 4 different systematic reviews showed no evidence of PB with either the trim and fill method or Egger regression. Four analyses had significant PB with Egger regression but not the trim and fill method. Conversely, 17 analyses showed signs of PB with the trim and fill method, but not Egger regression. The mean absolute percent difference between adjusted and observed point estimates was 15.5%, the median was 6.2%, and the range was 0% to 85.5%.
Our study investigated the methods by which systematic reviewers evaluate and attempt to mitigate PB. In general, we found that roughly half of reviews published in the top 5 anesthesiology journals discussed PB, and only a small percentage made efforts to mitigate PB through the search process.
An example of how PB influences study effect sizes is shown by our analysis of the meta-analysis in Figure 4 of Zhang et al’s46 study. They compared postoperative pain intensity in patients receiving 300 mg pregabalin versus a control group. Their results found a mean difference of −2.39 between the 2 groups. They did not assess PB. When we corrected for funnel plot asymmetry using the trim and fill analysis, the mean difference changed to −0.35. The ratio of the original effect to the adjusted effect was −2.39/−0.35 or 6.8286, and therefore, the original effect overestimates the effectiveness of pregabalin to treat postoperative pain by 582.9%. A similar situation was reported by Onishi and Furukawa8 when they found an 839.2% overestimate of the reduction in hemoglobin A1c levels in response to disease management programs.
Although 83.1% of articles included hand-searched reference lists of relevant articles, only 20.8% searched gray literature sources such as clinicaltrials.gov. Searching gray literature databases, such as the European Association for Grey Literature Exploitation and Open System for Information on Grey Literature, as well as dissertations, theses, and conference proceedings, may lead to more data and a smaller risk of PB.
One key strategy for countering the PB problem in the published literature is perhaps to do as the Journal of Cerebral Blood Flow & Metabolism has done and provide a section of the journal titled “Negative Results.” Here, authors can submit data that do not agree with their alternative hypothesis and/or did not reproduce published results.50 Clinicians can then consider those data while making clinical decisions. The International Journal of Radiation Oncology Biology Physics is currently pilot testing an adapted peer review process by which authors submit an introduction and methods section for an initial round of review. This portion of the manuscript is peer reviewed before data collection based on methodologic quality and design. A second peer review occurs after completion of the study, and both reviews are weighed in the final decision to publish. Efforts such as these may encourage authors to submit their research for publication, regardless of the strength or direction of their outcome.
Adherence to reporting guidelines may also increase the likelihood of discussing and evaluating PB when appropriate. Half of the reviews in this sample discussed PB. Many of these reviews appropriately followed published guidelines such as the PRISMA,51 MOOSE,52 or QUOROM53 statements; however, barely more than half of those studies evaluated PB. Item 15 of the PRISMA statement notes, “specify any assessment of risk of bias that may affect the cumulative evidence (eg, publication bias).”51 For the reviews that reported adhering to the PRISMA statement, only 75% discussed PB. Ideally, all reviews conducted with the PRISMA statement as a guide would at a minimum discuss PB and evaluate PB when appropriate.
Even though the PRISMA statement was published years ago, many researchers continue to use its predecessor, the QUORUM statement.51,53 An analysis of the journals included in this study demonstrated the lack of specificity and perhaps importance placed on adhering to the PRISMA guidelines. For example, Anesthesiology merely provides word count and abstract guidelines for systematic reviews.54British Journal of Anaesthesia has specific requirements for reporting RCTs and animal studies, but for systematic reviews, the journal provides 1 nondescript sentence about methods.55Regional Anesthesia and Pain Medicine has limited guidelines for systematic reviews.56 The other 2 journals emphasized following PRISMA for systematic review. Anaesthesia says systematic reviews should ideally be presented according to the PRISMA statement.57Anesthesia & Analgesia requires systematic reviews to be presented in accordance with the PRISMA statement.58 If journals do not require following reporting guidelines such as PRISMA, study quality may be decreased.59
As may be expected, in most of the PB analyses we conducted, the effect sizes decreased, although in some studies the effect size increased. Because so many of the analyses we conducted turned out to show significant PB, it stands that authors need to make PB evaluation a standard practice during the systematic review process when appropriate. Doing so will provide reliable and inclusive information for clinical situations and resource allocation situations to decision makers in anesthesiology and across all fields of medicine.
a There are many reasons a study’s results may not be published, such as (a) discovery mid-study of a flawed approach, (b) lack of diligence on the part of the investigators, (c) upstaging by other research that investigated the same issue in the same manner, (d) upstaging by the introduction of better drugs or methodologies, (e) reports of treatment side effects, (f) difficulty recruiting patients, and (g) other reasons. Our study is referring to results not being published to due statistically nonsignificant results.
b NIH Health and Education. Available at: http://www.nimh.nih.gov/health/topics/child-and-adolescent-mental-health/antidepressant-medications-for-children-and-adolescents-information-for-parents-and-caregivers.shtml. Accessed November 4, 2015.
c Covidence. Available at: https://www.covidence.org. Accessed November 4, 2015.
1. Murad MH, Montori VM, Ioannidis JP, et al. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312:171–179.
2. Manchikanti L. Evidence-based medicine, systematic reviews, and guidelines in interventional pain management, part I: introduction and general considerations. Pain Physician. 2008;11:161–186.
3. Jefferson T, Jones MA, Doshi P, et al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012;1:CD008965.
4. Souza JP, Pileggi C, Cecatti JG. Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey. Reprod Health. 2007;4:3.
5. American Society of Anesthesiologists. Practice guidelines for the perioperative management of patients with obstructive sleep apnea: An updated report by the American Society of Anesthesiologists Task Force on Perioperative Management of Patients with Obstructive Sleep Apnea. Anesthesiology. 2014;120:268–286.
6. Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ. 2012;344:d7762.
7. Svircevic V, van Dijk D, Nierich AP, et al. Meta-analysis of thoracic epidural anesthesia versus general anesthesia for cardiac surgery. Anesthesiology. 2011;114:271–282.
8. Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol. 2014;67:1320–1326.
9. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081.
10. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986;4:1529–1541.
11. Sterne JA, Egger M, Smith GD. Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;323:101–105.
12. Sutton AJ, Duval SJ, Tweedie RL, et al. Empirical assessment of effect of publication bias on meta-analyses. BMJ. 2000;320:1574–1577.
13. Rothstein HR, Sutton AJ, Borenstein M. Publication Bias in Meta-Analysis
. Hoboken, NJ: Wiley; 2005.
14. De Oliveira GS Jr, Chang R, Kendall MC, et al. Publication bias in the anesthesiology literature. Anesth Analg. 2012;114:1042–1048.
15. Littner Y, Mimouni FB, Dollberg S, et al. Negative results and impact factor: a lesson from neonatology. Arch Pediatr Adolesc Med. 2005;159:1036–1037.
16. Thaler K, Kien C, Nussbaumer B, et al; UNCOVER Project CONSORTIUM. Inadequate use and regulation of interventions against publication bias decreases their effectiveness: a systematic review. J Clin Epidemiol. 2015;68:792–802.
17. Whittington CJ, Kendall T, Fonagy P, et al. Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data. Lancet. 2004;363:1341–1345.
18. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. 2007;356:2457–2471.
19. Psaty BM, Kronmal RA. Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: a case study based on documents from rofecoxib litigation. JAMA. 2008;299:1813–1817.
20. Montori VM, Wilczynski NL, Morgan D, et al; Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330:68.
21. Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0
. 2011 The Cochrane Collaboration; Available at: www.cochrane-handbook.org
22. Sterne JA, Sutton AJ, Ioannidis JP, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.
23. Duval S, Tweedie R. Practical estimates of the effect of publication bias in meta-analysis. Australas Epidemiologist. 1998;5:14–17.
24. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455–463.
25. Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629–634.
26. Komatsu R, Turan AM, Orhan-Sungur M, et al. Remifentanil for general anaesthesia: a systematic review. Anaesthesia. 2007;62:1266–1280.
27. Potter LJ, Doleman B, Moppett IK. A systematic review of pre-operative anaemia and blood transfusion in patients with fractured hips. Anaesthesia. 2015;70:483–500.
28. Andersen LP, Werner MU, Rosenberg J, et al. A systematic review of peri-operative melatonin. Anaesthesia. 2014;69:1163–1171.
29. Carlisle JB. A meta-analysis of prevention of postoperative nausea and vomiting: randomised controlled trials by Fujii et al
. compared with other authors. Anaesthesia. 2012;67:1076–1090.
30. Yin JY, Ho KM. Use of plethysmographic variability index derived from the Massimo(®) pulse oximeter to predict fluid or preload responsiveness: a systematic review and meta-analysis. Anaesthesia. 2012;67:777–783.
31. Pikwer A, Åkeson J, Lindgren S. Complications associated with peripheral or central routes for central venous cannulation. Anaesthesia. 2012;67:65–71.
32. Gattas DJ, Dan A, Myburgh J, et al; CHEST Management Committee. Fluid resuscitation with 6% hydroxyethyl starch (130/0.4) in acutely ill patients: an updated systematic review and meta-analysis. Anesth Analg. 2012;114:159–169.
33. Gurgel ST, do Nascimento P Jr. Maintaining tissue perfusion in high-risk surgical patients: a systematic review of randomized clinical trials. Anesth Analg. 2011;112:1384–1391.
34. Hamilton MA, Cecconi M, Rhodes A. A systematic review and meta-analysis on the use of preemptive hemodynamic intervention to improve postoperative outcomes in moderate and high-risk surgical patients. Anesth Analg. 2011;112:1392–1402.
35. Yu SK, Tait G, Karkouti K, et al. The safety of perioperative esmolol: a systematic review and meta-analysis of randomized controlled trials. Anesth Analg. 2011;112:267–281.
36. Orhan-Sungur M, Kranke P, Sessler D, et al. Does supplemental oxygen reduce postoperative nausea and vomiting? A meta-analysis of randomized controlled trials. Anesth Analg. 2008;106:1733–1738.
37. Beattie WS, Wijeysundera DN, Karkouti K, et al. Does tight heart rate control improve beta-blocker efficacy? An updated analysis of the noncardiac surgical randomized trials. Anesth Analg. 2008;106:1039–1048.
38. Tiippana EM, Hamunen K, Kontinen VK, et al. Do surgical patients benefit from perioperative gabapentin/pregabalin? A systematic review of efficacy and safety. Anesth Analg. 2007;104:1545–1556.
39. Peyton PJ, Wu CY. Nitrous oxide-related postoperative nausea and vomiting depends on duration of exposure. Anesthesiology. 2014;120:1137–1145.
40. Levy M, Heels-Ansdell D, Hiralal R, et al. Prognostic value of troponin and creatine kinase muscle and brain isoenzyme measurement after noncardiac surgery: a systematic review and meta-analysis. Anesthesiology. 2011;114:796–806.
41. Arulkumaran N, Corredor C, Hamilton MA, et al. Cardiac complications associated with goal-directed therapy in high-risk surgical patients: a meta-analysis. Br J Anaesth. 2014;112:648–659.
42. Abdallah FW, Laffey JG, Halpern SH, et al. Duration of analgesic effectiveness after the posterior and lateral transversus abdominis plane block techniques for transverse lower abdominal incisions: a meta-analysis. Br J Anaesth. 2013;111:721–735.
43. Glossop AJ, Shephard N, Shepherd N, et al. Non-invasive ventilation for weaning, avoiding reintubation after extubation and in the postoperative period: a meta-analysis. Br J Anaesth. 2012;109:305–314.
44. Lundstrøm LH, Vester-Andersen M, Møller AM, et al; Danish Anaesthesia Database. Poor prognostic value of the modified Mallampati score: a meta-analysis involving 177 088 patients. Br J Anaesth. 2011;107:659–667.
45. McNicol ED, Tzortzopoulou A, Cepeda MS, et al. Single-dose intravenous paracetamol or propacetamol for prevention or treatment of postoperative pain: a systematic review and meta-analysis. Br J Anaesth. 2011;106:764–775.
46. Zhang J, Ho KY, Wang Y. Efficacy of pregabalin in acute postoperative pain: a meta-analysis. Br J Anaesth. 2011;106:454–462.
47. Ho KM, Tan JA. Use of L’Abbé and pooled calibration plots to assess the relationship between severity of illness and effectiveness in studies of corticosteroids for severe sepsis. Br J Anaesth. 2011;106:528–536.
48. Giglio MT, Marucci M, Testini M, et al. Goal-directed haemodynamic therapy and gastrointestinal complications in major surgery: a meta-analysis of randomized controlled trials. Br J Anaesth. 2009;103:637–646.
49. Hanna MN, Elhassan A, Veloso PM, et al. Efficacy of bicarbonate in decreasing pain on intradermal injection of local anesthetics: a meta-analysis. Reg Anesth Pain Med. 2009;34:122–125.
50. Dirnagl U, Lauritzen M. Fighting publication bias: introducing the Negative Results section. J Cereb Blood Flow Metab. 2010;30:1263–1264.
51. Moher D, Liberati A, Tetzlaff J, et al; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.
52. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012.
53. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999;354:1896–1900.
56. Regional Anesthesia and Pain Medicine. Instructions for Authors. Regional Anesthesia and Pain Medicine Online Submission and Review System Website. 2016. Available at: http://edmgr.ovid.com/rapm/accounts/ifauth.htm
. Accessed March 2, 2016.
59. Panic N, Leoncini E, de Belvis G, et al. Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One. 2013;8:e83138.