Secondary Logo

Journal Logo

Institutional members access full text with Ovid®

Can We Trust Observational Studies Using Propensity Scores in the Critical Care Literature? A Systematic Comparison With Randomized Clinical Trials*

Kitsios, Georgios D. MD, PhD1; Dahabreh, Issa J. MD, MS2,3; Callahan, Sean MD4; Paulus, Jessica K. PhD5; Campagna, Anthony C. MD6; Dargin, James M. MD6

doi: 10.1097/CCM.0000000000001135
Clinical Investigations

Objective: To assess the degree of agreement between propensity score studies and randomized clinical trials in critical care research.

Data Sources: Propensity score studies published in highly cited critical care or general medicine journals or included in a previous systematic review; corresponding randomized clinical trials included in Cochrane Systematic Reviews or published in PubMed.

Study Selection: We identified propensity score studies of the effects of therapeutic interventions on short- or long-term mortality. We systematically matched propensity score studies to randomized clinical trials based on patient selection criteria, interventions, and outcomes.

Data Extraction: We appraised the methods of included studies and extracted treatment effect estimates to compare the results of propensity score studies and randomized clinical trials. When multiple studies were identified for the same topic, we performed meta-analyses to obtain summary treatment effect estimates.

Data Synthesis: We matched 21 propensity score studies with 58 randomized clinical trials in 18 distinct comparisons (median, one propensity score study and two randomized clinical trials per comparison), for short- and long-term mortality. We found one statistically significant difference between designs (hyperoncotic albumin vs crystalloid fluids) among these 18 comparisons. Propensity score studies did not produce systematically higher (or lower) treatment effect estimates compared with randomized clinical trials, but estimates from the two designs differed by more than 30% in one third of the comparisons examined. Observational studies in critical care met widely accepted methodological standards for propensity score analyses.

Conclusions: Across diverse critical care topics, propensity score studies published in high-impact journals produced results that were generally consistent with the findings of randomized clinical trials. However, caution is needed when interpreting propensity score studies because occasionally their results contradict those of randomized clinical trials and there is no reliable way to predict disagreements.

1Division of Pulmonary, Allergy, and Critical Care Medicine, University of Pittsburgh Medical Center, Pittsburgh, PA.

2Center for Evidence-Based Medicine, School of Public Health, Brown University, Providence, RI.

3Department of Health Services, Policy & Practice, School of Public Health, Brown University, Providence, RI.

4Division Pulmonary, Critical Care, Allergy, and Sleep, Medical University of South Carolina, Charleston, SC.

5Predictive Analytics and Comparative Effectiveness Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA.

6Department of Pulmonary and Critical Care Medicine, Lahey Hospital and Medical Center, Burlington, MA.

*See also p. 2024.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (

Dr. Dahabreh received contract funding from the Patient-Centered Outcomes Research Institute (PCORI) Methods Research Award (ME-1306–03758; Principal Investigator). Dr. Callahan is employed by Medical University of South Carolina (Pulmonary Critical Care Fellowship Program). Dr. Paulus received support for article research from the National Institutes of Health (NIH). Her institution received grant support from the NIH. All statements in this article, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the PCORI, its Board of Governors, or the Methodology Committee. The remaining authors have disclosed that they do not have any potential conflicts of interest.

For information regarding this article, E-mail:

Copyright © by 2015 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.