Share this article on:

Systematic Reviews and Meta-Analyses of Randomized Controlled Trials on Perioperative Outcomes: An Urgent Need for Critical Reappraisal

Kehlet, Henrik MD, PhD*; Joshi, Girish P. MBBS, MD, FFARCSI

doi: 10.1213/ANE.0000000000000687
The Open Mind: The Open Mind

From the *Department for Surgical Pathophysiology, Rigshospitalet Copenhagen University, Copenhagen, Denmark; and Department of Anesthesiology and Pain Management, University of Texas Southwestern Medical Center, Dallas, Texas.

Accepted for publication November 18, 2014.

Funding: None.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Henrik Kehlet, MD, PhD, Department for Surgical Pathophysiology, 4074, Rigshospitalet, Blegdamsvej 9, 2100 Copenhagen, Denmark. Address e-mail to

The use of systematic reviews and meta-analyses based on randomized controlled trials (RCTs) has in many circumstances helped clinicians in their medical decision making.1 General guidelines to assist in performance and interpretation of these reviews have been available, including appropriate literature search strategies, risk of bias (including publication bias), RCT quality scoring and study details, statistical heterogeneity, and gaps in the literature, to justify conclusions.2–4 However, in assessing studies on perioperative outcomes associated with varied anesthetic, analgesic, and surgical aspects (technical or care principles), the pathophysiologic mechanisms and etiologies of investigated outcomes are much more complex than, for instance, assessing studies on the effects of lidocaine on arrhythmias after myocardial infarction. Despite the value and the need for proper systematic reviews and meta-analyses of RCTs assessing perioperative outcomes,4 we have, during the past decade, seen an increasing number of such reviews, often outnumbering the actual original studies, along with widely varying inclusion criteria and interpretation of pooled findings. Likely, the most important problem in recent systematic reviews and meta-analyses of perioperative outcomes has been the lack of details regarding individual elements or components of the perioperative care program within the scope of the single model care being studied. Here we want to give a few examples of this inadequacy of the methodology of the systematic reviews that is hindering sufficient information to clinicians.

Back to Top | Article Outline


That multimodal/multidisciplinary enhanced recovery after surgery programs (ERAS) programs improve perioperative outcomes has been well documented, mostly through detailed prospective cohort trials from many international centers. However, almost all of the RCTs that form the basis of the available systematic reviews5 have uniformly had inadequate compliance with the essential care principles of these programs.6 The systematic reviews have shown an average reduction of about 2.5 days of hospital length of stay, down from about 8 to 5.5 days, and approximately 50% reductions in medical complications.5,7 There are now >12 systematic reviews on ERAS programs after colorectal surgery; however, they are based on a smaller total number of RCTs. Although these systematic reviews applied the conventional generic quality criteria for interpreting the RCTs, it is troubling that not even one of these reviews considered the lack of compliance and more so that none has assessed whether the essential components of ERAS programs were implemented.6 Furthermore, it is surprising that these systematic reviews not only do not compare their findings with the observations of the many prospective, detailed cohort studies of a fully implemented fast-track program in open colonic surgery but also do not attempt to explain the reasons for a 40% shorter length of stay in these studies8 than the RCTs, which have incomplete and thus insufficient implementation of the essential care principles. When it comes to comparison between laparoscopic and open colonic resection, with or without ERAS programs, the situation is similar or even worse.9,10 This is again attributed to suboptimal compliance with the evidence-based components of the care programs. Importantly, such reviews neglect to discuss the prospective cohort studies with fully implemented enhanced recovery programs and superior outcomes when compared with the included RCTs.8,11 In summary, systematic reviews in ERAS colorectal surgery may have been insufficient because of the lack of critical analyses of the literature regarding compliance with important evidence-based care principles, thereby hindering interpretation for the nonexpert clinician.12

Back to Top | Article Outline


There is general agreement that continuous epidural analgesia with a sufficient dose of local anesthetics has many favorable physiological effects in the postoperative period. These include optimal dynamic analgesia (with mobility), reduction of the catabolic injury response, reduction of paralytic ileus, and opioid sparing,13 all of which are essential to enhancing recovery after major open abdominal surgery. Nevertheless, we have seen a steady series of systematic reviews during the past 2 decades, including a recent one from 201414 which concluded that mortality may be reduced and that the analysis provided helpful information to clinicians treating surgical patients. Disappointingly, the overall observed beneficial effects on organ dysfunctions and complications have been minor. Unfortunately, all such meta-analyses continue to include RCTs beginning from the 1980s, and all include studies that have very little or no information about implementation of updated and integrated perioperative care principles (e.g., early mobilization, oral feeding, and fluid management), thereby limiting interpretation regarding the benefits of epidural analgesia in modern evidence-based perioperative care.15 These meta-analyses have also included widely different surgical populations, including those in which there is now general agreement that epidural analgesia is not necessary. For example, multimodal analgesic techniques, such as regional analgesia, including the use of extended release or continuous infusion of local anesthetics combined with nonopioid analgesics, are adequate for a variety of less invasive surgical procedures, such as video-assisted thorascopic surgery, lower limb arthroplasty, prostatectomy, hysterectomy, and laparoscopic procedures.16 Also, for open thoracic surgery, paravertebral blocks may be preferred.17 Therefore, future RCTs and systematic reviews/meta-analyses focusing on epidural analgesia and related postoperative outcomes should not include previous data with insufficient information about perioperative care but should instead concentrate on currently relevant procedures with a fully integrated enhanced recovery program.

Back to Top | Article Outline


The use of transversus abdominis plane blocks has recently become more common in patients undergoing abdominal surgery.13,18 A recent systematic review and meta-analysis concluded that transversus abdominis plane block is an effective strategy to improve early pain relief after laparoscopic procedures.19 The analysis included different types of surgical procedures, which, per se, may be problematic.18 However, considering the 3 included laparoscopic cholecystectomy studies,20–22 it is apparent that such meta-analyses, performed without consideration of study details, are not useful or may even be invalid. Although the conventional criteria of the quality of the studies included in this meta-analysis were acceptable, on closer inspection, patients in all 3 studies did not receive evidence-based basic analgesic treatment, such as low-dose dexamethasone, acetaminophen, a nonsteroidal anti-inflammatory drug, and, more importantly, the well-documented, simple, and inexpensive approach of port-sided local anesthetic infiltration.13,18 This is a typical example of inappropriate assessment of a novel invasive analgesic technique without appropriate comparison to other well-established, evidence-based, simpler, and less costly analgesic regimens, thereby hindering interpretation and relevance and clinical guidance of the systematic review.19

Back to Top | Article Outline


Several systematic reviews have supported the claim that goal-directed fluid management improves postoperative outcomes, including morbidity and hospital stay.23 However, as with the studies evaluating epidural analgesia mentioned above, almost all goal-directed fluid management studies have failed to implement other evidence-based care principles shown to improve outcomes, thereby hampering their interpretation and limiting clinical applicability. Furthermore, most systematic reviews have included older studies that used general hemodynamic optimization criteria and not the rational idea of individualized goal-directed fluid management to enhance stroke volume. Also, the RCTs vary in the choice of goals (e.g., maximal stroke volume, predetermined oxygen delivery index, specific hemodynamic algorithm, or their combinations). The hemodynamic therapy protocols varied such that some studies used only fluid administration, whereas others used combinations of fluids and inotropic drugs. In addition, the complications are ill defined, and definitions vary significantly. It is noteworthy that the reported mortality and morbidity rates in the control group of the included studies range from 0% to 50%. Similarly, studies were included that did not follow current evidence-based perioperative care principles as well as studies that consider length of hospital stays significantly longer than the current evidence.23 Most importantly, these limitations are not discussed in the systematic reviews.23 Thus, a recent critical reassessment of studies evaluating enhanced recovery programs questioned the magnitude of the value of goal-directed fluid therapy.24 Overall, although goal-directed fluid therapy is rational, it is necessary to be careful when interpreting the systematic reviews as well as to be critical about the exact methodology in the individual studies, including the types of patients, types of surgical procedures, and the use of evidence-based care principles, when applying the information in clinical practice.

Back to Top | Article Outline


Because of concerns that some flawed systematic reviews and meta-analysis may make the evidence appear stronger than it really is, which may negatively influence patient care, there is an urgent need for a modified approach to performing (for authors) and publishing (for editors and reviewers of journals) systematic reviews and meta-analyses. A recent guideline provides an approach to evaluating systematic reviews and meta-analyses and to applying the results to patient care, which itself is based on the well-accepted process of conducting a systematic review.4 This includes formulating a valid clinical question, defining eligibility criteria for studies, performing an exhaustive literature search, appropriately assessing and selecting studies, and presenting results in a clinically applicable manner. In this context, one of the most important aspects of any systematic review is determining criteria for inclusion and exclusion of studies. Thus, appropriate selection will ensure that the systematic review remains clinically valid. For example, it is necessary to exclude RCTs with methodologies that do not reflect current clinical practice or in which practice may have influenced the outcomes more than the intervention in question. An example could be a systematic review of RCTs evaluating 2 different fluid administration regimens in colonic surgery with inclusion of a study that conforms to the principles of enhanced recovery and a study that has not followed these principles (i.e., inappropriate use of a gastric tube with delayed oral feeding, predominantly opioid-based analgesia, and limited mobilization). Thus, a critical appraisal of the perioperative care used in the included studies is necessary. Furthermore, in perioperative outcome studies, it is necessary to confirm that the methodologies of the included studies are similar enough to be pooled together for meta-analysis.

For example, studies evaluating the same analgesic intervention may require detailed assessment with respect to differences in supplemental analgesics used to allow sufficiently valid and clinically relevant information to the clinician. In addition to refining the inclusion/exclusion criteria and appropriate evaluation of the methodology of included studies, we propose that the following approach should be used by authors of systematic reviews and meta-analyses:

  1. Discuss whether the currently available systematic reviews/meta-analyses on the subject in question have appropriate methodology and sufficient uniformity in perioperative care to allow relevant clinical conclusions.
  2. Explain why the authors performed a duplicate systematic review/meta-analysis when a previous systematic review on the subject already exists.25
  3. Discuss, in detail, how and why the new systematic review/meta-analysis differs from the previous ones.25
Back to Top | Article Outline


The pathophysiologic mechanisms and etiologies of perioperative outcomes are multifactorial and thus the value of a particular single intervention may be difficult to assess, unless all other important factors influencing outcomes are well controlled and congruent with current evidence. Unfortunately, as has been discussed above, this has often not been the case. In most published systematic reviews, the authors have simply performed statistical analyses of reported complication rates without critical appraisal of the methodology and current clinical applicability of the included studies. Therefore, we submit that the conclusions of such meta-analyses may be questionable. In fact, the flawed conclusions may lead to inappropriate patient care and subsequent negative unintended consequences. Hopefully, in future, we will see an improvement in systematic reviews of studies in complex perioperative care settings, such that not only the usual criteria for performing a systematic review4 are fulfilled but also that a critical appraisal is performed of the methodology of perioperative care used in the individual RCTs.

Finally, in complex perioperative care settings, detailed prospective multicenter cohort studies may be equally important or preferable26 and provide guidance for the design of future RCTs of unimodal interventions. This also applies to the currently popular large RCTs,27 in which the potential effect of a unimodal intervention may be missed because of there being too many outcome variables (i.e., noise factors), widely different surgical procedures, no recognition of, or adjustment for, suboptimal perioperative care programs, and/or lack of a complete understanding of the pathogenesis of the specific postoperative morbidity in question.

Back to Top | Article Outline


Name: Henrik Kehlet, MD, PhD.

Contribution: This author helped write the manuscript.

Attestation: Henrik Kehlet approved the final manuscript.

Name: Girish P. Joshi, MBBS, MD, FFARCSI.

Contribution: This author helped write the manuscript.

Attestation: Girish P. Joshi approved the final manuscript.

This manuscript was handled by: Steven L. Shafer, MD.

Back to Top | Article Outline


1. Greenhalgh T, Howick J, Maskrey NEvidence Based Medicine Renaissance Group. Evidence Based Medicine Renaissance Group. . Evidence based medicine: a movement in crisis? BMJ. 2014;348:g3725
2. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJGRADE Working Group. GRADE Working Group. . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924–6
3. Moher D, Liberati A, Tetzlaff J, Altman DGPRISMA Group. PRISMA Group. . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009;6:e1000097
4. Murad MH, Montori VM, Ioannidis JP, Jaeschke R, Devereaux PJ, Prasad K, Neumann I, Carrasco-Labra A, Agoritsas T, Hatala R, Meade MO, Wyer P, Cook DJ, Guyatt G. How to read a systematic review and meta-analysis and apply the results to patient care: users’ guides to the medical literature. JAMA. 2014;312:171–9
5. Chambers D, Paton F, Wilson P, Eastwood A, Craig D, Fox D, Jayne D, McGinnes E. An overview and methodological assessment of systematic reviews and meta-analyses of enhanced recovery programmes in colorectal surgery. BMJ Open. 2014;4:e005014
6. Slim K, Kehlet H. Commentary: fast track surgery: the need for improved study design. Colorectal Dis. 2012;14:1013–4
7. Varadhan KK, Neal KR, Dejong CH, Fearon KC, Ljungqvist O, Lobo DN. The enhanced recovery after surgery (ERAS) pathway for patients undergoing major elective open colorectal surgery: a meta-analysis of randomized controlled trials. Clin Nutr. 2010;29:434–40
8. Kehlet H. Fast-track colorectal surgery. Lancet. 2008;371:791–3
9. Li MZ, Xiao LB, Wu WH, Yang SB, Li SZ. Meta-analysis of laparoscopic versus open colorectal surgery within fast-track perioperative care. Dis Colon Rectum. 2012;55:821–7
10. Kehlet H. Interpretation of meta-analyses of laparoscopic versus open colorectal surgery? Dis Colon Rectum. 2013;56:e364–5
11. Keller DS, Stulberg JJ, Lawrence JK, Delaney CP. Process control to measure process improvement in colorectal surgery: modifications to an established enhanced recovery pathway. Dis Colon Rectum. 2014;57:194–200
12. Paton F, Chambers D, Wilson P, Eastwood A, Craig D, Fox D, Jayne D, McGinnes E. Effectiveness and implementation of enhanced recovery after surgery programmes: a rapid evidence synthesis. BMJ Open. 2014;4:e005015
13. Joshi GP, Schug SA, Kehlet H. Procedure-specific pain management and outcome strategies. Best Pract Res Clin Anaesthesiol. 2014;28:191–201
14. Pöpping DM, Elia N, Van Aken HK, Marret E, Schug SA, Kranke P, Wenk M, Tramèr MR. Impact of epidural analgesia on mortality and morbidity after surgery: systematic review and meta-analysis of randomized controlled trials. Ann Surg. 2014;259:1056–67
15. Kehlet H, Wilmore DW. Evidence-based surgical care and the evolution of fast-track surgery. Ann Surg. 2008;248:189–98
16. Joshi GP, Bonnet F, Kehlet HPROSPECT Collaboration. PROSPECT Collaboration. . Evidence-based postoperative pain management after laparoscopic colorectal surgery. Colorectal Dis. 2013;15:146–55
17. Joshi GP, Bonnet F, Shah R, Wilkinson RC, Camu F, Fischer B, Neugebauer EA, Rawal N, Schug SA, Simanski C, Kehlet H. A systematic review of randomized trials evaluating regional techniques for postthoracotomy analgesia. Anesth Analg. 2008;107:1026–40
18. Joshi GP, Kehlet H. Procedure-specific pain management: the road to improve postsurgical pain management? Anesthesiology. 2013;118:780–2
19. De Oliveira GS Jr, Castro-Alves LJ, Nader A, Kendall MC, McCarthy RJ. Transversus abdominis plane block to ameliorate postoperative pain outcomes after laparoscopic surgery: a meta-analysis of randomized controlled trials. Anesth Analg. 2014;118:454–63
20. El-Dawlatly AA, Turkistani A, Kettner SC, Machata AM, Delvi MB, Thallaj A, Kapral S, Marhofer P. Ultrasound-guided transversus abdominis plane block: description of a new technique and comparison with conventional systemic analgesia during laparoscopic cholecystectomy. Br J Anaesth. 2009;102:763–7
21. Petersen PL, Stjernholm P, Kristiansen VB, Torup H, Hansen EG, Mitchell AU, Moeller A, Rosenberg J, Dahl JB, Mathiesen O. The beneficial effect of transversus abdominis plane block after laparoscopic cholecystectomy in day-case surgery: a randomized clinical trial. Anesth Analg. 2012;115:527–33
22. Ra YS, Kim CH, Lee GY, Han JI. The analgesic effect of the ultrasound-guided transverse abdominis plane block after laparoscopic cholecystectomy. Korean J Anesthesiol. 2010;58:362–8
23. Pearse RM, Harrison DA, MacDonald N, Gillies MA, Blunt M, Ackland G, Grocott MP, Ahern A, Griggs K, Scott R, Hinds C, Rowan KOPTIMISE Study Group. OPTIMISE Study Group. . Effect of a perioperative, cardiac output-guided hemodynamic therapy algorithm on outcomes following major gastrointestinal surgery: a randomized clinical trial and systematic review. JAMA. 2014;311:2181–90
24. Srinivasa S, Lemanu DP, Singh PP, Taylor MH, Hill AG. Systematic review and meta-analysis of oesophageal Doppler-guided fluid management in colorectal surgery. Br J Surg. 2013;100:1701–8
25. Kahn R, Gale EA. Gridlocked guidelines for diabetes. Lancet. 2010;375:2203–4
26. Berwick DM. The science of improvement. JAMA. 2008;299:1182–4
27. Devereaux PJ, Chan MT, Eisenach J, Schricker T, Sessler DI. The need for large clinical studies in perioperative medicine. Anesthesiology. 2012;116:1169–75
© 2015 International Anesthesia Research Society