Late Breaker Articles

Multicenter Comparison of Machine Learning Methods and Conventional Regression for Predicting Clinical Deterioration on the Wards

Churpek, Matthew M. MD, MPH, PhD¹; Yuen, Trevor C. MS¹; Winslow, Christopher MD²; Meltzer, David O. MD, PhD¹; Kattan, Michael W. MBA, PhD³; Edelson, Dana P. MD, MS¹

Author Information

¹Department of Medicine, University of Chicago, Chicago, IL.

²Department of Medicine, NorthShore University HealthSystem, Evanston, IL.

³Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH.

Author Contributions: Study concept and design: M.C., D.P.E.; acquisition of data: C.W., T.Y., D.P.E.; analysis and interpretation of data: all authors; first drafting of the manuscript: M.C.; critical revision of the manuscript for important intellectual content: all authors; statistical analysis: M.C.; obtained funding: M.C., C.W., D.P.E.; administrative, technical, and material support: T.Y., D.P.E.; study supervision: D.P.E, M.K.

Dr. Churpek had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ccmjournal).

This research was funded in part by an institutional Clinical and Translational Science Award grant (UL1 RR024999; PI: Dr. Julian Solway). Drs. Churpek and Edelson have a patent pending (ARCD. P0535US.P2) for risk stratification algorithms for hospitalized patients. Dr. Churpek is supported by a career development award from the National Heart, Lung, and Blood Institute (K08 HL121080) and has received honoraria from Chest for invited speaking engagements. Dr. Meltzer is supported by a career development award from the National Institute of Aging (1 K24 AG031326-01). In addition, Dr. Edelson has received research support and honoraria from Philips Healthcare (Andover, MA) and an honorarium from Early Sense (Tel Aviv, Israel). She has ownership interest in Quant HC (Chicago, IL), which is developing products for risk stratification of hospitalized patients.

Preliminary versions of these data were presented at the 2015 meeting of the American Thoracic Society (May 20, 2015; Denver, CO).

Dr. Churpek received funding (honoraria from Chest for invited speaking engagements), has a patent pending (Drs. Churpek and Edelson have a patent pending [ARCD. P0535US.P2] for risk stratification algorithms for hospitalized patients), and received support for article research from the NIH. His institution received funding from the NHLBI K08 HL121080 and from institutional Clinical and Translational Science Award grant UL1 RR024999. Dr. Meltzer’s institution received support for article research from the NIH. Dr. Edelson has a patent pending (Drs. Churpek and Edelson have a patent pending [ARCD. P0535US.P2] for risk stratification algorithms for hospitalized patients), received support for article research from the NIH, and received funding from a honorarium from EarlySense and from ownership interest in Quant HC ([Chicago, IL] which is developing products for risk stratification of hospitalized patients). Her institution received funding from research support from Philips Healthcare. The remaining authors have disclosed that they do not have any potential conflicts of interest.

For information regarding this article, Email: [email protected]

Critical Care Medicine 44(2):p 368-374, February 2016. | DOI: 10.1097/CCM.0000000000001571

Abstract

Objective:

Machine learning methods are flexible prediction algorithms that may be more accurate than conventional regression. We compared the accuracy of different techniques for detecting clinical deterioration on the wards in a large, multicenter database.

Design:

Observational cohort study.

Setting:

Five hospitals, from November 2008 until January 2013.

Patients:

Hospitalized ward patients

Interventions:

None

Measurements And Main Results:

Demographic variables, laboratory values, and vital signs were utilized in a discrete-time survival analysis framework to predict the combined outcome of cardiac arrest, intensive care unit transfer, or death. Two logistic regression models (one using linear predictor terms and a second utilizing restricted cubic splines) were compared to several different machine learning methods. The models were derived in the first 60% of the data by date and then validated in the next 40%. For model derivation, each event time window was matched to a non-event window. All models were compared to each other and to the Modified Early Warning score, a commonly cited early warning score, using the area under the receiver operating characteristic curve (AUC). A total of 269,999 patients were admitted, and 424 cardiac arrests, 13,188 intensive care unit transfers, and 2,840 deaths occurred in the study. In the validation dataset, the random forest model was the most accurate model (AUC, 0.80 [95% CI, 0.80-0.80]). The logistic regression model with spline predictors was more accurate than the model utilizing linear predictors (AUC, 0.77 vs 0.74; p < 0.01), and all models were more accurate than the MEWS (AUC, 0.70 [95% CI, 0.70-0.70]).

Conclusions:

In this multicenter study, we found that several machine learning methods more accurately predicted clinical deterioration than logistic regression. Use of detection algorithms derived from these techniques may result in improved identification of critically ill patients on the wards.