High-performance detection and early prediction of septic shock for alcohol-use disorder patients : Annals of Medicine and Surgery

Secondary Logo

Journal Logo


High-performance detection and early prediction of septic shock for alcohol-use disorder patients

Calvert, Jacoba; Desautels, Thomasa; Chettipally, Ulib,c; Barton, Christopherc; Hoffman, Janaa; Jay, Melissaa; Mao, Qingqinga; Mohamadlou, Hamida; Das, Ritankara,*

Author Information
Annals of Medicine & Surgery 8():p 50-55, June 2016. | DOI: 10.1016/j.amsu.2016.04.023
  • Free


1. Introduction

Alcohol Use Disorder (AUD) encompasses alcohol dependency, abuse, and addiction [1]. In the United States, AUD affects over 18 million people, and can lead to increased severity of illness for a variety of conditions [2,3]. AUD is estimated to be present in between 10% and 33% of patients in the Intensive Care Unit (ICU) [2]. AUD patients have increased hospital stays by 2.4 days on average, and are up to 8% more likely to experience unplanned rehospitalization within 30-days of discharge [4,5]. According to the World Health Organization, “In 2012, about 3.3 million net deaths, or 5.9% of all global deaths, were attributable to alcohol consumption. 139 million net DALYs (disability-adjusted life years), or 5.1% of the global burden of disease and injury, were attributable to alcohol consumption.” [6] Through increased complications [7] and longer length of stays, AUD increases costs and burdens on the health care system [8,9].

Sepsis has been one of the leading causes of death in the United States for over a decade [10,11]. It is a major public health concern, costing over $20 billion per year in the U.S. alone [12]. New definitions for sepsis and septic shock have recently been introduced, in an effort to simplify and streamline the clinical diagnoses of sepsis [13]. While these new definitions may prove useful and eventually find wide adoption, they are currently still under debate. Therefore for the purposes of this manuscript we have utilized the standard definitions of sepsis, severe sepsis, and septic shock, which are summarized in Table 1.

Table 1:
Classification of sepsis, severe sepsis, and septic shock.

Patients with AUD are 1.7 times more likely to develop any healthcare associated infection, including sepsis, than patients who do not have AUD [18]. In particular, AUD is known to complicate and exacerbate infections and sepsis in hospitalized patients [19,20]. Although the relationships between AUD, septic shock, and infections are still being explored, increased sepsis mortality in patients with AUD may be impacted by the effects of AUD on cortisol and cytokine production [21]. The exact pathophysiologic mechanisms for the increased risk of sepsis and adverse outcomes in the AUD patient have not clearly been elucidated but a number of potential mechanisms (specific and general) have been suggested. These include the compromise of cellular immune function [22] and the alteration in the ratio of T1 helper cells to T2 helper cells [23]. Abuse of alcohol also directly affects the functioning of macrophages [24]. Complicating the recognition of emerging septic shock, AUD patients often suffer from chronic hypertension [25]. Therefore hypotension, which correlates with septic shock [17], may be difficult to identify among patients in the AUD subpopulation. Additionally, lactate, a common biomarker test which is used in the recently proposed updated septic shock definition [13,26], may be inaccurate for AUD patients [27] because patients with AUD often suffer from chronic lactic acidosis [28,29]. While this study was not meant to explore or explain the pathophysiologic mechanisms of sepsis and failure in AUD patients, it does recognize and attempt to correlate the clinical presentation of these patients and propose methods to identify those at risk for sepsis and septic shock before they have fully manifested themselves.

The higher costs and increased risks from sepsis and septic shock in the AUD population, in conjunction with suboptimal existing septic shock diagnostic screening performance, demonstrate the need for improved risk scoring systems for septic shock in AUD patients. Here, we analyze the performance of a risk scoring system, InSight, when detecting and predicting septic shock onset for AUD patients. We have determined to use septic shock as the gold standard for the InSight program because accurate identification and prediction of septic shock is crucial for the timely administration of antibiotics and supportive treatments to reduce mortality [30]. Additionally, the onset time of septic shock is well defined, and thus provides a clear time point for predictive assessment. We will demonstrate that InSight outperforms existing methods in discriminating between septic shock and non-septic shock patients, as well as providing early warning of impending septic shock onset.

2. Methods

2.1. Data set

The Multiparameter Intelligent Monitoring in Intensive Care (MIMIC) III database [31] was queried to obtain the 29,083 patients used in InSight training and testing. MIMIC III contains de-identified patient records collected from Beth Israel Deaconess Medical Center during the years 2001–2012. We filtered a total of 61,532 MIMIC III ICU stays to obtain patients aged 15 or more years with admission to any of the intensive care units (the age filter primarily excludes neonatal ICU patients and a handful of pediatric cases), and with at least one observation of the following measurements: blood oxygen saturation, heart rate, pH, pulse pressure, respiration rate, systolic blood pressure, temperature, and white blood cell count. We also recorded the presence of ICD-9 codes for septic shock (785.52) and of alcohol abuse and related conditions (291.X, 291.XX, 303.XX, 305.XX, 357.5, 425.5, 535.3X, 571.2, and 571.3, where X denotes a wildcard).

2.2. Gold standard

Patients were assigned outcomes of septic shock upon meeting the following, hierarchical definition. Septic shock was identified using the following criteria: (1) SIRS criteria score ≥ 2, [16] (2) presence of an infection-related ICD-9 code, (3) organ dysfunction, (4) systolic blood pressure below 90 mmHg for at least 1 hour, and (5) total fluid replacement ≥1200 mL or ≥20 mL/kg for 24 hours. Combined with the requirement that patients have an AUD-related ICD-9 code, a total of 270 ICU stays were associated with AUD patients who also contracted septic shock, giving a prevalence of 0.9%. This prevalence is reasonable since septic shock and AUD prevalences are roughly 10% each and, assuming independence, a 1% net prevalence would be expected.

2.3. InSight training, score assignment, and comparison with MEWS and SIRS

InSight performs multidimensional analysis on streams of patient measurements. When working with these patient time-series data, we used a standard time resolution of one hour. We used the most recent value of measurements that were not updated by the end of each hour period. For singlet measurements, we fit a continuous function approximating the measurement value-conditioned probability distribution of the gold standard outcome. Doublets and triplets of measurement trends were binned according to heuristic tables. These tables associate a bin's empirical septic shock risk with ranges of measurement values, similar to the calculation of Modified Early Warning Score (MEWS) [32]. In the next step, we estimated the correlation between each feature and septic shock for AUD patients. The features were weighted by these correlations, then all of the measurements, singlet, doublet, and triplet trends were summed. Finally, these aggregates were combined through logistic regression, in order to assign risk scores which best reflect training data. This process and an associated equation were detailed in a previous publication, though a different clinical use-case was modeled [33,40]. The testing results that follow were generated from assigning risk scores to patient data not included in the training.

The MEWS scores for each patient were calculated by applying the corresponding heuristic table [32]. The Simplified Acute Physiology Score II (SAPS II) [34] was calculated for each patient using the open source SQL queries for MIMIC III [35]. The SIRS criteria were also applied at each time, with the number of criteria met equaling the score assigned at that time. These scores provide comparison in the following results.

3. Results

Septic shock onset detection and prediction performance of InSight, MEWS, SAPS II, and SIRS is summarized by the Receiver Operating Characteristic (ROC) curves in Fig. 1. Each curve represents the average performance under 4-fold cross validation. At the time of onset, InSight had an average Area Under the ROC (AUROC) value of 0.965, compared to 0.7397 for MEWS, 0.6220 for SAPS II, and 0.5998 for SIRS. Four hours prior to onset, InSight, MEWS, SAPS II, and SIRS had average AUROCs of 0.8149, 0.6355, 0.6220, and 0.6128, respectively. This analysis was repeated across the one, two, and three hours prior to septic shock to demonstrate the improvement in each tool's prediction quality nearer the time of onset (Fig. 2). For any of these prediction times, InSight outperformed the comparison methods.

Fig. 1.:
Receiver Operating Characteristic (ROC) curves for InSight, Modified Early Warning Score (MEWS), Simplified Acute Physiology Score (SAPS II), and the Systemic Inflammatory Response Syndrome (SIRS) criteria at septic shock onset and four hours prior. Each curve was generated by averaging the results from 4-fold cross validation.
Fig. 2.:
Area Under the ROC curve (AUROC) metric for InSight, MEWS, SAPS II, and SIRS, at septic shock onset and the four hours prior. Each curve is the average result under 4-fold cross validation.

The score thresholds which determine the patient classifications of septic shock and non-septic shock can be tuned to optimize performance metrics of interest. This is illustrated for septic shock detection in Table 2. InSight, MEWS, SAPS II, and SIRS score thresholds were chosen to yield sensitivities similar to 0.90, which highlights InSight’s improvement across a variety of metrics, including a 6-fold improvement in positive predictive value (PPV), 33-fold improvement in diagnostic odds ratio (DOR), and 5-fold improvement in F1 score, which is particularly relevant given the low prevalence of AUD patient septic shock. InSight manages to catch nearly all septic shock cases while maintaining a specificity more than double than that of SIRS. Over the comparison method with highest specificity, InSight reduces false alarms by 82%.

Table 2:
Performance metric comparison of InSight with other scoring systems at the time of septic shock onset (with score cutoffs chosen to produce sensitivities near 0.90). Diagnostic odds ratio (DOR) and the likelihood ratios (LR) are abbreviated.

A second comparison is displayed in Table 3 for the time four hours prior to septic shock onset, with sensitivities fixed near 0.75. Similar to the time-of-onset results, InSight predictions improve upon those made by MEWS, SAPS II, and SIRS, across all metrics. Compared with MEWS, the most specific of the other methods, InSight gives a 48% reduction in false alarms.

Table 3:
Performance metric comparison of InSight with other scoring systems four hours prior to the time of septic shock onset (with score cutoffs chosen to produce sensitivities near 0.75). Diagnostic odds ratio (DOR) and the likelihood ratios (LR) are abbreviated.

In Fig. 3, we illustrate the evolution of the septic shock risk score distribution, at the time of onset and four hours prior. At the time of onset, the distribution of scores assigned to septic shock positive and negative patients are highly separated, leading to reliable classifications. Four hours before septic shock onset, it becomes more difficult to discriminate between the populations. Note that, because the prevalence of AUD patients with septic shock in this dataset is small, shifting the score cutoff to include as positive even small portions of the negative septic shock population can lead to a large false positive rate.

Fig. 3.:
Scaled InSight score distributions for the time of septic shock onset and four hours prior.

4. Discussion

We have described a risk scoring system, InSight, for early septic shock detection in the AUD population. Vital signs, lab tests, patient demographics, and their changes over time were processed into dimensionless features. These features captured the complicated, higher-order correlations between distinct measurements, enabling their aggregation into septic shock risk scores. Our results demonstrated that this approach is capable of high quality septic shock detection and prediction several hours prior to onset, with sensitivity and specificity that exceed those of existing diagnostic tools.

This was demonstrated in Figs. 1 and 2, where InSight maintains a relatively high AUROC for the time of onset and the four hours prior, across the trials of 4-fold cross validation. However, there are significant decreases in performance for each hour earlier the prediction is made. An explanation for this was shown in Fig. 3, where the score distributions of the two populations become more distinct closer to the onset time. Considering the low prevalence, the consequence of this overlap is a large increase in the absolute number of false alarms that are set off in order to obtain more true alarms. Despite this, InSight performance was quantifiably strong, as highlighted by Tables 2 and 3. Note the bolded rows of the tables, which emphasize diagnostic odds ratio and F1 score, in addition to sensitivity and specificity. The diagnostic odds ratio and F1 score are particularly important metrics for predictions made on small subpopulations and which, unlike accuracy, PPV, and NPV, are independent of prevalence. InSight performs particularly well on these metrics in comparison with SIRS, SAPS II, and MEWS. Because these results were generated through cross validation, they demonstrate that InSight is a robust alternative to the SIRS criteria, SAPS II, and MEWS in predicting septic shock onset.

Homeostasis is the process of maintaining balanced biological health through positive and negative feedback loops. In cases of septic shock, the host response to infectious insult is dysregulated. This host dysregulation involves the loss of coupling of biological processes involved in the homeostatic mechanism. By examining the degree of interrelations of vital sign measurements over time, our goal is to detect the dysregulation of biological processes before individual vital sign analysis would indicate a deterioration of homeostasis.

A simple example of this is the shock index (the ratio of heart rate to systolic blood pressure). This ratio is used because independently heart rate and blood pressure are often poor predictors of perfusion status. However, their ratio provides a much better estimate of patient condition. We extend this concept to the correlations of doublets and triplets between eight common clinical measurements (blood oxygen saturation, heart rate, pH, pulse pressure, respiration rate, systolic blood pressure, temperature, and white blood cell count) over time. Through analysis of patient trends the algorithm is able to predict which correlations correspond most closely with the risk of impending septic shock. While these correlations and trends are meaningful, it does not provide relevant clinical information to deconstruct these measurements to single values of the component vital signs.

The InSight algorithm is designed to integrate with a hospital's existing EHR system, and to be trained on the data set and patient population available at the site of implementation. InSight’s function is to continuously monitor patient vital signs and autonomously provide status indication of patient risk for septic shock. Alert levels can be set at the sensitivity and specificity desired by the supervising clinician to provide the desired sensitivity while minimizing alarm fatigue.

A challenge in developing predictive algorithms for clinical settings is the identification of patients who are at risk for adverse outcomes either before the outcome occurs or the underlying diagnosis has been made. In particular, while the AUD inpatient subpopulation is particularly vulnerable to infection and sepsis, the identification of AUD patients has been problematic, especially in trauma centers where patients may present with acute intoxication and acute medical/surgical conditions [36]. There have been several strategies suggested for detecting AUD patients that rely on medical record review or interviews with the patient [37–39] - strategies that in many cases may not be practical in a busy critical care setting. However, the robust results of InSight demonstrate accurate predictions of developing septic shock for AUD patients, even if their AUD status is unrecognized. Application of the InSight algorithm will identify those AUD patients most at risk of sepsis prior to onset, despite confounding diagnostic elements, and may provide the opportunity for meaningful clinical intervention to prevent adverse outcomes.

5. Conclusion

We have developed a risk scoring system that predicts the onset of septic shock. Vital signs, lab tests, patient demographics, and time series of these measurements were converted to dimensionless indicators; a combination of singlets, doublets, and triplets of these measurements were used by InSight to develop a risk score. The results indicate that InSight can predict the onset of septic shock better than MEWS, SAPS II, and SIRS, and continues to outperform these systems even four hours prior to the onset of septic shock. The strong performance of InSight in this study suggests that it will be efficacious in predicting septic shock in patients in spite of an unrecognized AUD status.

Ethical approval

Not applicable.

Sources of funding

This material is based upon work supported by the National Science Foundation under Grant No. 1549867. The funder had no role in the conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, and approval of the manuscript; and decision to submit the manuscript for publication.

Author contribution

Jake Calvert, Thomas Desauutels: Study design, data collection, data analysis, writing.

Uli Chettipally, Christopher Barton: Study design, review.

Jana Hoffman: Study design, writing.

Melissa Jay, Qingqing Mao, Hamid Mohamadlou: Data collection, data analysis.

Ritankar Das: Study design, writing.

Conflicts of interest

We declare no conflicts of interest.


Ritankar Das.


We wish to thank Samson Mataraso, Nima Shajarian, Allen Chen, Hana Ghanim, Adrian Gunawan, Lauren Song, and Jasmine Jan for valuable contributions to the writing of this manuscript.


[1] DSM-5 American Psychiatric Association., 2013. Diagnostic and Statistical Manual of Mental Disorders, American Psychiatric Publishing, Arlington.
[2] de Wit M, Jones DG, Sessler CN, Zilberberg MD, Weaver MF. Alcohol-use disorders in the critically ill patient. CHEST J. 2010 Oct 1;138(4):994-1003.
[3] Rehm J, Baliunas D, Borges GL, Graham K, Irving H, Kehoe T, Parry CD, Patra J, Popova S, Poznyak V, Roerecke M. The relation between different dimensions of alcohol consumption and burden of disease: an overview. Addiction. 2010 May 1;105(5):817-843.
[4] Baldwin WA, Rosenfeld BA, Breslow MJ, Buchman TG, Deutschman CS, Moore RD. Substance abuse-related admissions to adult intensive care. CHEST J. 1993 Jan 1;103(1):21-25.
[5] Lindenauer PK, Normand SL, Drye EE, Lin Z, Goodrich K, Desai MM, Bratzler DW, O'Donnell WJ, Metersky ML, Krumholz HM. Development, validation, and results of a measure of 30-day readmission following hospitalization for pneumonia. J. Hosp. Med. 2011 Mar 1;6(3):142-150.
[6] World Health Organization. Management of Substance Abuse: Alcohol. [online]. 2016. http://www.who.int/substance_abuse/facts/alcohol/en/. (accessed 25.04.16).
[7] De Wit M, Best AM, Gennings C, Burnham EL, Moss M. Alcohol use disorders increase the risk for mechanical ventilation in medical patients. Alcohol. Clin. Exp. Res. 2007 Jul 1;31(7):1224-1230.
[8] de Wit M, Zilberberg MD, Boehmler JM, Bearman GM, Edmond MB. Outcomes of patients with alcohol use disorders experiencing healthcare-associated infections. Alcohol. Clin. Exp. Res. 2011 Jul 1;35(7):1368-1373.
[9] Mehta AJ. Alcoholism and critical illness: a review. World J. Crit. Care Med. 2016 Feb 4;5(1):27.
[10] Hoyert DL, Heron MP, Murphy SL, Kung HC. Deaths: Final Data for 2003. Natl. Vital Stat. Rep. 2006 Apr 19;54(13):1-20.
[11] Murphy SL, Xu J, Kochanek KD. Deaths: Final Data for 2010. Natl. Vital Stat. Rep. 2013 May;61(4):1-17. from the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System.
[12] Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit. Care Med. Baltim. 2001 Jul 1;29(7):1303-1310.
[13] Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016 Feb 23;315(8):801-810.
[14] Vincent JL, Opal SM, Marshall JC, Tracey KJ. Sepsis definitions: time for change. Lancet Lond. Engl. 2013 Mar 2;381(9868):774-775.
    [15] Sprung CL, Sakr Y, Vincent JL, et al. An evaluation of systemic inflammatory response syndrome signs in the Sepsis Occurrence in Acutely ill Patients (SOAP) study. Intensive Care Med. 2006 Mar 1;32(3):421-427.
      [16] Levy MM, Fink MP, Marshall JC, et al. 2001 sccm/esicm/accp/ats/sis international sepsis definitions conference. Intensive Care Med. 2003 Apr 1;29(4):530-538.
      [17] Dellinger RP, Levy MM, Rhodes A, et al. Surviving Sepsis Campaign: international guidelines for management of severe sepsis and septic shock. Intensive Care Med. 2012;39(2):165-228. 2013 Feb 1.
      [18] de Wit M, Goldberg S, Hussein E, Neifeld JP. Health care-associated infections in surgical patients undergoing elective surgery: are alcohol use disorders a risk factor? J. Am. Coll. Surg. 2012 Aug 31;215(2):229-236.
      [19] Moss M, Parsons PE, Steinberg KP, et al. Chronic alcohol abuse is associated with an increased incidence of acute respiratory distress syndrome and severity of multiple organ dysfunction in patients with septic shock. Crit. Care Med. 2003 Mar 1;31(3):869-877.
      [20] O'Brien JM Jr, Lu B, Ali NA, et al. Alcohol dependence is independently associated with sepsis, septic shock, and hospital mortality among adult intensive care unit patients*. Crit. Care Med. 2007 Feb 1;35(2):345-350.
      [21] de Wit M, Wiaterek GK, Gray ND, et al. Relationship between alcohol use disorders, cortisol concentrations, and cytokine levels in patients with sepsis. Crit. Care. 2010 Dec 22;14(6):R230.
      [22] Szabo G, Mandrekar P. A recent perspective on alcohol, immunity, and host defense. Alcohol. Clin. Exp. Res. 2009 Feb 1;33(2):220-232.
      [23] Lau A, von Dossow V, Sander M, MacGuill M, Lanzke N, Spies C. Alcohol use disorder and perioperative immune dysfunction. Anesth. Analg. 2009 Mar 1;108(3):916-920.
      [24] Moss M. Epidemiology of sepsis: race, sex, and chronic alcohol abuse. Clin. Infect. Dis. 2005 Nov 15;41(Suppl. 7):S490-S497.
      [25] Beilin LJ. Alcohol and hypertension. Clin. Exp. Pharmacol. Physiol. 1995 Mar 1;22(3):185-188.
      [26] Okorie ON, Dellinger P. Lactate: biomarker and potential therapeutic target. Crit. Care Clin. 2011 Apr 30;27(2):299-326.
      [27] Gustafson Mark L, et al. The effect of ethanol on lactate and base deficit as predictors of morbidity and mortality in trauma. Am. J. Emerg. Med. 2015;33.5:607-613.
      [28] Nguyen HB, Rivers EP, Knoblich BP, et al. Early lactate clearance is associated with improved outcome in severe sepsis and septic shock. Crit. Care Med. 2004 Aug 1;32(8):1637-1642.
      [29] Dezman Zachary DW, et al. Alcohol consumption decreases lactate clearance in acutely injured patients. Injury 2016. pii: S0020-1383(16)30034-1. doi: 10.1016/j.injury.2016.03.007. http://www.ncbi.nlm.nih.gov/pubmed/27025567.
      [30] Kumar, Anand, et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit. Care Med. 2006;34.6:1589-1596.
      [31] Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, Mietus JE, Moody GB, Peng CK, Stanley HE. Physiobank, physiotoolkit, and physionet components of a new research resource for complex physiologic signals. Circulation. 2000 Jun 13;101(23):e215-e220.
      [32] Subbe CP, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. Qjm. 2001 Oct 1;94(10):521-526.
      [33] Calvert JS, Price DA, Barton CW, Chettipally UK, Das R. Discharge recommendation based on a novel technique of homeostatic analysis. J. Am. Med. Inform. Assoc. 2016. ocw014. https://jamia.oxfordjournals.org/content/early/2016/03/28/jamia.ocw014.full.
      [34] Le Gall JR, Lemeshow S, Saulnier F. A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. Jama. 1993 Dec 22;270(24):2957-2963.
      [35] https://github.com/MIT-LCP/mimic-code. Archived at: http://www.webcitation.org/6ffJC48Mj.
      [36] Gentilello LM, Villaveces A, Ries RR, et al. Detection of acute alcohol intoxication and chronic alcohol dependence by trauma center staff. J. Trauma Acute Care Surg. 1999 Dec 1;47(6):1131-1135.
      [37] Escobar F, Espi F, Canteras M. Diagnostic tests for alcoholism in primary health care: compared efficacy of different instruments. Drug Alcohol Depend. 1995 Dec 31;40(2):151-158.
      [38] Schneekloth TD, Morse RM, Herrick LM, Suman VJ, Offord KP, Davis LJ. Point prevalence of alcoholism in hospitalized patients: continuing challenges of detection, assessment, and diagnosis. InMayo Clin. Proc. 2001 May 3;76(5):460-466.
      [39] Ziring DJ, Adler AG. Alcoholism. Are you missing the diagnosis? Postgrad. Med. 1991 Apr;89(5):139-141.
      [40] Calvert JS, Price DA, Chettipally UK, Barton CW, Feldman MD, Hoffman JL, Jay M, Das R. A computational approach to early sepsis detection. Comput. Biol. Med. 2016. doi:10.1016/j.compbiomed.2016.05.003.

      Alcohol use disorder; Clinical decision support systems; Septic shock; Sepsis; Electronic health records

      © 2016 by Lippincott Williams & Wilkins, Inc.