Original ArticlesPostimplementation Evaluation of a Machine Learning–Based Deterioration Risk Alert to Enhance Sepsis Outcome ImprovementsLinnen, Daniel T. PhD, MS, RN, RN-BC; Hu, Xiao PhD; Stephens, Caroline E. PhD, RN, GNP-BC, FAANAuthor Information Kaiser Permanente Northern California, Kaiser Foundation Hospitals, Inc, Regional Offices, Oakland, California (Dr Linnen); Duke University, School of Nursing Durham, North Carolina (Dr Hu); and School of Nursing, University of Utah, Salt Lake City (Dr Stephens). Correspondence: Daniel T. Linnen, PhD, MS, RN, RN-BC, Kaiser Permanente Northern California, Regional Offices, 1950 Franklin St, 17th floor, Oakland, CA 94612 ([email protected]). Authors' Contribution: Dr Linnen led study conception, design, data acquisition, performed clinical chart reviews, conducted data abstraction and data analysis, interpreted results, and prepared the manuscript. Dr Hu assisted with the development of study conception and design, data acquisition and analysis, interpretation of results, and manuscript preparation. Dr Stephens is senior author and guided the development of study conception and design, data acquisition and analysis, interpretation of results, and manuscript preparation. The Maribelle & Stephen Leavitt Scholarship and Jonas Nurse Scholars Scholarship at the University of California, San Francisco, and the Predoctoral Research Fellowship of the Nurse Scholars Academy at Kaiser Permanente Northern California supported this research study during Daniel Linnen's PhD training at the University of California, San Francisco. Daniel Linnen thanks Dr Gabriel Escobar, MD, and Dr Vincent Liu, MD, at the Kaiser Permanente Northern California Division of Research for their mentorship and content expertise; John Greene, Senior Programmer at the Kaiser Permanente Northern California Division of Research, for data acquisition; and Ms Jill Pope, Academic Editor at the Kaiser Permanente Center for Health Research in Portland, Oregon, for her assistance with manuscript preparation. Conflicts of Interests: None. Nursing Administration Quarterly: October/December 2020 - Volume 44 - Issue 4 - p 336-346 doi: 10.1097/NAQ.0000000000000438 Buy Metrics Abstract Machine learning–based early warning systems (EWSs) can detect clinical deterioration more accurately than point-score tools. In patients with sepsis, however, the timing and scope of sepsis interventions relative to an advanced EWS alert are not well understood. The objectives of this study were to evaluate the timing and frequency of fluid bolus therapy, new antibiotics, and Do Not Resuscitate (DNR) status relative to the time of an advanced EWS alert. We conducted 2 rounds of chart reviews of patients with an EWS alert admitted to community hospitals of a large integrated health system in Northern California (round 1: n = 21; round 2: n = 47). We abstracted patient characteristics and process measures of sepsis intervention and performed summary statistics. Sepsis decedents were older and sicker at admission and alert time. Most EWS alerts occurred near admission, and most sepsis interventions occurred before the first alert. Of 14 decedents, 12 (86%) had a DNR order before death. Fluid bolus therapy and new intravenous antibiotics frequently occurred before the alert, suggesting a potential overlap between sepsis care in the emergency department and the first alert following admission. Two tactics to minimize alerts that may not motivate new sepsis interventions are (1) locking out the alert during the immediate time after hospital admission; and (2) triaging and reviewing patients with alerts outside of the unit before activating a bedside response. Some decedents may have been on a palliative/end-of-life trajectory, because DNR orders were very common among decedents. Nurse leaders sponsoring or leading machine learning projects should consider tactics to reduce false-positive and clinically meaningless alerts dispatched to clinical staff. © 2020 Wolters Kluwer Health, Inc. All rights reserved.