Secondary Logo

Journal Logo

Institutional members access full text with Ovid®

Comparing Observed and Predicted Mortality Among ICUs Using Different Prognostic Systems: Why Do Performance Assessments Differ?*

Kramer, Andrew A. PhD1,2; Higgins, Thomas L. MD, MBA, MCCM3,4; Zimmerman, Jack E. MD, FCCM1,5

doi: 10.1097/CCM.0000000000000694
Feature Articles
Buy
SDC

Objectives: To compare ICU performance using standardized mortality ratios generated by the Acute Physiology and Chronic Health Evaluation IVa and a National Quality Forum–endorsed methodology and examine potential reasons for model-based standardized mortality ratio differences.

Design: Retrospective analysis of day 1 hospital mortality predictions at the ICU level using Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum models on the same patient cohort.

Setting: Forty-seven ICUs at 36 U.S. hospitals from January 2008 to May 2013.

Patients: Eighty-nine thousand three hundred fifty-three consecutive unselected ICU admissions.

Interventions: None.

Measurements and Main Results: We assessed standardized mortality ratios for each ICU using data for patients eligible for Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum predictions in order to compare unit-level model performance, differences in ICU rankings, and how case-mix adjustment might explain standardized mortality ratio differences. Hospital mortality was 11.5%. Overall standardized mortality ratio was 0.89 using Acute Physiology and Chronic Health Evaluation IVa and 1.07 using National Quality Forum, the latter having a widely dispersed and multimodal standardized mortality ratio distribution. Model exclusion criteria eliminated mortality predictions for 10.6% of patients for Acute Physiology and Chronic Health Evaluation IVa and 27.9% for National Quality Forum. The two models agreed on the significance and direction of standardized mortality ratio only 45% of the time. Four ICUs had standardized mortality ratios significantly less than 1.0 using Acute Physiology and Chronic Health Evaluation IVa, but significantly greater than 1.0 using National Quality Forum. Two ICUs had standardized mortality ratios exceeding 1.75 using National Quality Forum, but nonsignificant performance using Acute Physiology and Chronic Health Evaluation IVa. Stratification by patient and institutional characteristics indicated that units caring for more severely ill patients and those with a higher percentage of patients on mechanical ventilation had the most discordant standardized mortality ratios between the two predictive models.

Conclusions: Acute Physiology and Chronic Health Evaluation IVa and National Quality Forum models yield different ICU performance assessments due to differences in case-mix adjustment. Given the growing role of outcomes in driving prospective payment patient referral and public reporting, performance should be assessed by models with fewer exclusions, superior accuracy, and better case-mix adjustment.

Supplemental Digital Content is available in the text.

1Cerner Corporation, Vienna, VA.

2Department of Biostatistics, Kansas University Medical Center, Kansas City, MO.

3Critical Care Division, Department of Medicine, Baystate Medical Center, Springfield, MA.

4Tufts University School of Medicine, Boston, MA.

5Department of Anesthesiology and Critical Care Medicine, George Washington University, Washington, DC.

* See also p. 473.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ccmjournal).

Supported, in part, by Cerner Corporation, which owns the registered trademark for Acute Physiology and Chronic Health Evaluation (APACHE) and holds the marketing rights to APACHE and Mortality Probability Model (MPM0-III).

Dr. Kramer is employed by and has stock options with Cerner Corporation (Cerner Corporation owns the marketing rights to Acute Physiology and Chronic Health Evaluation [APACHE]). Dr. Higgins served as Chair of the Project IMPACT research committee (2003–2007) and had access to Project IMPACT data used to develop MPM0-III during that time. He owns Cerner stock, has received travel support to speak at the Cerner Critical Care Outcomes forum, and has previously collaborated with Drs. Zimmerman and Kramer on other MPM and APACHE articles. Dr. Higgins received support for travel from and lectured for Cerner (travel support at Cerner Critical Care Conference January 2014 and in previous years) and has stock options with Cerner (not related to the article or any work with Cerner in the past). Dr. Zimmerman consulted for, received support for travel from, lectured for, and received support for article preparation from Cerner Corporation.

For information regarding this article, E-mail: akramer@cerner.com

Copyright © by 2015 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.