Because the results of a meta-analysis are used to formulate the highest level recommendation in clinical practice guidelines, clinicians should be mindful of problems inherent in this technique. Rather than reviewing meta-analysis in abstract, general terms, we believe readers can gain a more concrete understanding of the problems through a detailed examination of one meta-analysis. The meta-analysis on which we focus is that conducted by an American College of Chest Physicians/American Association for Respiratory Care/American College of Critical Care Medicine Task Force on ventilator weaning.
Two authors extracted data from all studies included in the Task Force’s meta-analysis.
The major obstacle to reliable internal validity and, thus, reliable external validity (generalizability) in biological research is systematic error, not random error. If systematic errors are present, averaging (as with a meta-analysis) does not decrease them—instead, it reinforces them, producing artifact. The Task Force’s meta-analysis commits several examples of the three main types of systematic error: selection bias (test-referral bias, spectrum bias), misclassification bias (categorizing reintubation as weaning failure, etc.), and confounding (pressure support treated as unassisted breathing). Several additional interpretative errors are present.
An increase in study size, as achieved through the pooling of data in a meta-analysis, is mistakenly thought to increase external validity. On the contrary, combining heterogeneous studies poses considerable risk of systematic error, which impairs internal validity and, thus, external validity. The strength of recommendations in clinical practice guidelines is based on a misperception of the relative importance of systematic vs. random error in science.
From the Division of Pulmonary and Critical Care Medicine, Edward Hines Jr. Veterans Affairs Hospital, Hines, IL; and Loyola University of Chicago Stritch School of Medicine, Hines, IL.
Dr. Tobin receives royalties for two books on critical care published by McGraw-Hill, New York, NY. Dr. Jubran has not disclosed any potential conflicts of interest.
Supported, in part, by a Merit Review grant from the Veterans Administration Research Service and by grant RO1 NR008782 from the National Institutes of Health.
For information regarding this article, E-mail: email@example.com