Sepsis is a leading cause of mortality and morbidity globally.[1–4] Early diagnosis of sepsis and early initiation of evidence-based bundle care can greatly improve the outcome of sepsis. Unfortunately, early and accurate diagnosis of sepsis is difficult. Sepsis is a complex clinical syndrome with a wide range of manifestations. Although the systemic inflammatory response syndrome (SIRS) criteria were part of the prior definition of sepsis, it has been shown to be unable to differentiate severe from uncomplicated infections. The latest Sepsis-3 definition describes sepsis as a life-threatening organ dysfunction caused by a dysregulated host response to infection,[5,6] with life-threatening organ dysfunction defined as a change in the sequential organ failure assessment (SOFA) score of more than 2 points in intensive care unit (ICU) patients.[7–9] In clinical settings outside the ICU where calculating the SOFA score is not routine, a simplified score—quick sepsis-related organ failure assessment (qSOFA)—was introduced as a screening tool for patients with sepsis.
Since the introduction of qSOFA, concerns have been raised. None of the elements in qSOFA are specific for the detection of infection, and subsequent validation studies showed suboptimal overall discrimination and sensitivity (reported sensitivity 32%) under the recommended cut-off. In addition, diagnosing sepsis relies on the accuracy of the physician's clinical suspicion of infection. Infectious disease biomarkers such as C-reactive protein (CRP) and procalcitonin (PCT), on the contrary, have been shown to accurately predict infection and mortality. These 2 markers were accepted as part of the diagnostic criteria in Sepsis-2, but were not included in the Sepsis-3 definition.[13,14] Thus, combining biomarker information with the qSOFA score would potentially enhance its ability to predict the mortality risk from sepsis. In this study, we sought to evaluate whether adding either CRP or PCT to the qSOFA score would improve its ability to predict inhospital mortality in a multicenter cohort of patients who presented with clinical symptoms of systemic infection.
2.1 Study design and locations
This was a multicenter retrospective cohort study performed at the Sichuan Provincial People Hospital (SCPH) in Chengdu City, First People's Hospital of Foshan (FSFPH) in Guangdong province of China, and National Taiwan University Hospital Yunlin Branch (NTUH) in Douliou city. All 3 hospitals are tertiary-care urban medical centers.
2.2 Study population
Patients were enrolled retrospectively using consecutive sampling of cases from each hospital from January 1, 2015 to December 31, 2016. All adult patients (≥18 years old) who presented to the emergency department or were admitted to the hospital floor were eligible for inclusion. Patients were included if they had symptoms that indicated systemic infection; and PCT (VIDAS BRAHMS PCT) and blood culture tests within 24 hours of admission. Exclusion criteria were missing data, transfer from other hospitals, leucopenia, do-not-resuscitate (DNR) orders, lost to follow-up, or history of pre-existing thyroid disease that may affect procalcitonin levels. Participating investigators from the 3 sites independently reviewed all retrieved medical records to confirm the presence of infection as a reason for admission. The study focused on qSOFA score, which targeted the sepsis patients outside the ICU. Therefore, we excluded patients who developed sepsis in ICU. A cohort that prospectively collected 493 consecutive emergency department (ED) patients with presumed sepsis was used for independent validation. Patients with presumed sepsis was defined patients who fulfilled at least 2 of the 3 SIRS criteria (temperature >38°C or <36°C, pulse rate ≥90 beats per minute, and respiratory rate ≥20/min) upon ED admission with a presumed diagnosis of systemic infection by treating physicians. This study was approved by the Research Committees and Institutional Review Boards for all institutions, and it met criteria for exemption from informed consent.
2.3 Data collection
The hospitalization course of all patients was followed from admission to discharge. Using a standardized data collection instrument, data were collected on patients (Table 1). The worst physiological and laboratory measurements within the initial 24 h of admission were noted. The endpoint, all-cause inhospital mortality, was determined by systematically reviewing hospital records resulting from each patient's index hospitalization. SIRS criteria variables include abnormal body temperature, tachycardia, tachypnea, and abnormal white blood cell count. The criteria for qSOFA include altered mental status, hypotension, or tachypnea. In addition, we defined septic shock as a systolic blood pressure <100 mm Hg requiring vasopressor therapy, and respiratory difficulty as a respiratory rate >22 breaths/min, an oxygen saturation <90%, or need for supplemental oxygen by either face mask or 100% nonrebreather.
2.4 Measurement of PCT and CRP
Blood samples were collected within 24 hours of admission. PCT concentrations were measured using an immunoluminometric assay with a detection limit of 0.06 ng/mL (VIDAS PCT; bioMerieux, Marcy, France). CRP was measured using an immunonephelometric assay (Olympus, Osaka, Japan) with a detection limit of 0.2 mg/L.
2.5 Statistical analysis
Chi-square tests for categorical variables and Mann–Whitney U tests for continuous variables were performed for univariate comparison. The correlation between serum levels of biomarkers and clinical severity (defined by SIRS or qSOFA score) was examined with box plots. To evaluate whether PCT or CRP has incremental prognostic value independent of qSOFA variables, we employed a logistic regression model. The values of CRP and PCT were entered into the model as predefined ordinal scales to ease clinical application. We categorized the biomarker values into 3 ordinal classes (0, 1, and 2): 0 was assigned for PCT levels <0.25 and CRP <60, 1 for PCT levels from 0.25 to 2 and CRP from 60 to 120, and 3 for PCT >2 and CRP >120. We then compared the predictive values of SIRS, qSOFA, qSOFA plus ordinal CRP class (qSOFA_CRP), and qSOFA plus ordinal PCT class (qSOFA_PCT). The predictive value of each model was first calculated by the c-statistic, or area under the receiver-operating characteristic (ROC) curve (AUC), for which larger values indicate better discrimination. We compared the predicted probabilities and observed risk in each risk category.
Next, we divided all patients into 4 predicted inhospital mortality risk groups: very low risk (0% to less than 5%), low risk (5% to less than 15%), high risk (15% to less than 30%), and very high risk (30% or greater). We then assessed whether the biomarker enhanced qSOFA models significantly reclassified patients into more appropriate mortality risk categories in comparison with the qSOFA model. The overall reclassification improvement was evaluated by the net reclassification improvement (NRI) and the integrated discrimination improvement (IDI). The NRI was calculated by summing the proportion of participants across risk categories whose estimated risk shifts in the correct direction minus the proportion of participants whose risk shifts in the incorrect direction.[16–18] Therefore, NRI was used as a measure to estimate any overall improvement in reclassification with the new model. Rather than using discrete risk categories, the IDI calculates the difference in discrimination slopes between the 2 models. Consequently, the IDI index demonstrates the improvement in both discrimination and reclassification.
Lastly, we calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of SIRS, qSOFA, and qSOFA_PCT in predicting inhospital mortality, and we calibrated each model to mortality rate with bar graphs. We validate the accuracy of qSOFA and qSOFA_PCT in an independent historical sample by calculating the sensitivity, specificity, PPV, and NPV. All analyses were performed with SAS Version 9.4 (Cary, NC), except for the NRI and IDI statistics, which were computed with R Statistical Software (Foundation for Statistical Computing, Vienna, Austria). A 2-sided P value <.05 was viewed as significant.
2.6 Ethical approval
This study was approved by the institutional review board of National Taiwan University Hospital.
3.1 Patient characteristics and outcome
During the study period, 604 patients from NTUH, 503 from SCPH, and 515from FSFPH fulfilled the inclusion criteria. After the exclusion of 304 ICU patients, the final cohort included a total of 1318 patients (Supplementary eFig. 1, http://links.lww.com/MD/D28). In all, 867 (65.8%), 208 (15.8%), and 752 (53.1%) patients were diagnosed with sepsis according to the SIRS criteria and qSOFA score, respectively. In all, 205 (15.6%) patients had clinical significant bacteremia, with a higher prevalence of gram-negative bacteremia (7.1%) than gram-positive bacteremia (6.1%). The median age of the study sample was 64.0 (interquartile range 47–75) years old. The overall inhospital mortality rate was 13.5%. The mortality of patients with clinical significant bacteremia was 19.5%.
Table 1 summarizes the baseline characteristic of patients. In general, nonsurvivors were older, had a higher burden of comorbidities and organ dysfunction, had more lower respiratory tract infections, had more bloodstream infection and polymicrobial infection, and had a greater proportion of patients with PCT, SIRS, or qSOFA scores higher than reference levels.
3.2 Comparison of correlation of laboratory markers and qSOFA
We examined the correlation between 3 laboratory markers (white blood cell count, CRP, and PCT) and the qSOFA score. The scatter plots of the 3 markers stratified by the 4 qSOFA classes are shown in Fig. 1. PCT has the highest correlation with qSOFA class, followed by CRP and white blood cell (WBC) count.
3.3 Comparison of model performance-discrimination, model fit, and reclassification
Table 2 also presents comparisons of discrimination, overall fit, and reclassification statistics for 4 different models. For discrimination, defined as the AUC, qSOFA_PCT model had the largest AUC, followed by qSOFA_CRP, qSOFA, and SIRS. For overall fit, using Bayes information criteria, a lower score indicates better goodness of model fit with a parsimonious model. qSOFA_PCT had the best performance, followed by qSOFA, qSOFA + CRP, and SIRS. For reclassification, we made 2 stage comparisons. First, when comparing qSOFA to SIRS, both IDI and NRI showed qSOFA reclassified patients to a more appropriate mortality risk category than SIRS. Comparing qSOFA_CRP or qSOFA_PCT with qSOFA, we found that qSOFA_PCT could better reclassify patients as indicated by significantly higher NRI and IDI. qSOFA_CRP, however, failed to demonstrate a higher IDI than qSOFA, although the combined score still had better NRI than qSOFA alone. Using the qSOFA_PCT score, in all, 752 (53.1%) patients could be diagnosed with sepsis, in contrast to 208 (15.8%) patients by qSOFA score alone.
3.4 Model calibration
To visually compare model performance, we constructed bar graphs in Fig. 2 to present the mortality rate in each severity class for the 3 models (SIRS, qSOFA, and qSOFA_PCT). qSOFA_PCT showed the best calibration, followed by qSOFA. The mortality rate did not correlate with the SIRS classification. We further calculated the sensitivity, specificity, PPV, and NPV for SIRS, qSOFA, and qSOFA_PCT for inhospital mortality prediction (Table 3). We used a cut-off score of ≥2 for each criterion. Despite the poor performance in model fit and calibration, SIRS had the best sensitivity at the cost of the lowest specificity. qSOFA had the highest specificity, but lowest sensitivity. qSOFA_PCT ≥2 greatly enhances the sensitivity of qSOFA to 86.5% with a compromised specificity of 47.5%.
In the validation cohort, we confirmed qSOFA had a suboptimal sensitivity (39.1%) and high specificity (94.9%) in predicting 30-day mortality. Incorporation of PCT into qSOFA could improve the sensitivity to 90.9% at the cost of low specificity (50.3%).
In our retrospective validation of the Sepsis-3 definitions using multicenter cohorts, we confirmed that the qSOFA score is superior in mortality prediction compared with SIRS in terms of discrimination, model fit, reclassification, and calibration statistics. We also demonstrated that PCT has better correlation with clinical severity than WBC count or CRP. Furthermore, we found that combining PCT and the qSOFA score by simply adding the ordinal scale of PCT to the qSOFA score can significantly enhance its mortality prediction capability in all dimensions of model performance indicators. Clinically, PCT enhanced qSOFA, or qSOFA_PCT, has the best sensitivity (86.5%) and can be served as a screening tool to quickly identify patients with sepsis who may benefit from early intervention. qSOFA alone has the best specificity (87.0%) and can subsequently serve as a quick confirmation tool to aid in the decision to pursue more invasive treatment.
The original Sepsis-3 definitions proposed using the simple qSOFA score as the initial screening tool, followed by the comprehensive SOFA score as the confirmation tool for sepsis. In the original work, the sensitivity of a qSOFA score ≥2 was reported to be low at 55%, albeit with a high specificity (84%), whereas a change of SOFA score of ≥2 had a higher sensitivity (68%), but a lower specificity (67%). The sensitivity and specificity profiles of qSOFA and SOFA, however, are contradictory to their proposed clinical use. A screening tool requires high sensitivity, whereas a confirmation tool requires high specificity. Our study, like other external validation studies, confirmed the low sensitivity and high specificity of the qSOFA score.
Thus, we propose incorporating PCT levels into qSOFA to correct for its low sensitivity. The high sensitivity (86.5%) and high NPV (95.8%) of qSOFA_PCT ≥2 justify the combined score as an initial screening tool. Nonetheless, we would like to preserve the qSOFA as a confirmation tool for 2 reasons. First, it is validated as a simple but highly specific tool with a specificity (84%) higher than the proposed delta SOFA greater than 2 (specificity 67%). Second, in clinical settings outside the ICU such as the ED, the comprehensive SOFA score may not be easily obtained. In summary, we believe the newly proposed algorithm, using high-sensitivity qSOFA_PCT as a screening tool and the high-specificity qSOFA score as the confirmation tool, is the most optimal use of biomarker information and the best clinical decision rule in clinical settings outside of the ICU.
In addition to the contradiction between the sensitivity and specificity profiles of qSOFA and SOFA, another major concern regarding replacing SIRS with qSOFA is the absence of clinical indicators of infection in the qSOFA score. PCT has been shown to be a reasonably sensitive marker in differentiating sepsis from sterile SIRS. In a large meta-analysis with 3244 patients, PCT alone had a sensitivity of 77% and a specificity of 79% in the diagnosis of sepsis. The accuracy of PCT has been shown to be valid for various sites of infection and for different populations such as the elderly, patients with renal impairment, and patients with autoimmune disease.[22–25] Therefore, the addition of PCT to the qSOFA score may complement its lack of infection indicators. However, the accuracy of PCT may be compromised in patients with neutropenia, hyperfunctioning thyroid cancer, and in patients with certain subacute infectious diseases such as infective endocarditis.[26–29] The qSOFA_PCT should be used cautiously in these specific populations.
In clinical settings where testing for PCT is not readily available, the SIRS criteria would retain its value as a simple and low-cost screening tool. In multiple validation studies, SIRS ≥2 consistently demonstrates a higher sensitivity than qSOFA. We thus propose that when PCT cannot be obtained, SIRS should remain as a screening tool with qSOFA as the confirmation tool.
Although we showed that the AUC of qSOFA_PCT is higher than qSOFA alone, the confidence intervals overlapped. The limitations of the c-statistic, or AUC, as a measure of clinical model performance, have been discussed extensively in the literature. AUC is based exclusively on ranks, and it only measures how well the predicted values can rank order the responses.[30,31] It may not be as sensitive as the likelihood function in choosing between models, and it is less clinically relevant as a calibration measure that directly cross-classifies the predicted risk categories with the observed risk. The magnitude of IDI, defined as the difference in discrimination slopes, has a direct interpretation. In our study, qSOFA_PCT, compared with qSOFA, had an IDI of 0.0097. This indicates that the new model increases the mean difference of predicted probabilities for death and nondeath by 0.97%. In the reclassification analysis, NRI is calculated as the sum of the net percentages of correctly reclassified patients with and without the event of interest. NRI is the favored metric when assessing the true discriminatory potential of a new predictor compared with other predictors. It captures the incremental strength of the new predictor after accounting for correlations with variables included in the baseline model. NRI values above 0.6 are considered strong, 0.4 intermediate, and below 0.2 weak. In our case, the NRI comparing qSOFA_PCT with qSOFA is 0.35, which suggests a medium number of patients were reclassified to more appropriate risk categories. This change was found to be significant (P = .00011).
Since its publication of Sepsis-3 criteria and qSOFA in 2016, numerous attempts have been made to assess the performance of qSOFA. A recent meta-analysis collecting 45 studies showed low sensitivity and good specificity when the qSOFA is used as a screening tool for sepsis (pooled sensitivity was 61%, and specificity was 72%). In contrast, SIRS criteria resulted in a pooled sensitivity of 88%, but with only 26% specificity. Our study demonstrates that combing PCT and qSOFA, rather than PCT and SIRS, can achieve the best sensitivity. The simple qSOFA_PCT score will help clinicians identify at-risk patients and those with high likelihood for deterioration. A prospective external validation of this simple score is needed to verify the generalizability of this modified score.
Our work has both strengths and limitations. We were the first in the literature to prove the added prognostic value of PCT to the qSOFA score by a rigorous statistical analysis. We further proposed a practical strategy for clinical use: the highly sensitive qSOFA_PCT score used as a screening tool, followed by the highly specific qSOFA score used as the confirmation tool. In addition, this multicenter design has a relatively large population, which increases the generalizability of our work. A key limitation of our study was the retrospective nature of this work. However, the patient characteristics of our study cohort are comparable with prospectively collected continuous samples. In addition, as these sepsis patients were not enrolled in ICU, common severity scores such as SAPS, APACHE II, or SOFA score were not available.
Our work confirmed that qSOFA has low sensitivity and high specificity in predicting sepsis mortality. The incorporation of the ordinal scale of PCT to the qSOFA model could enhance sensitivity and reclassify patients into risk groups that better reflect their actual short-term mortality risk. We propose using qSOFA_PCT as a screening tool followed by qSOFA alone as a confirmation tool for the identification of patients with sepsis in settings outside of the ICU.
We thank the staff of the Core Labs, the Department of Medical Research, and National Taiwan University Hospital for technical support, and Medical Wisdom Consulting Group for technical assistance in statistical analysis.
C.-C.L. designed the study, obtained funding, drafted the analytical plan, guided the statistical analysis, interpreted the data, and wrote the draft. H.Y., L.N., A.L., and K.L. helped to collect data in this study. D.Y. and M.-T.L. helped with language editing and provided critical comments. T.-C.H. conducted statistical analysis. Y.-C.H. analyzed the data, provided critical feedback, and authorized the final manuscript.
Data curation: Lu Nie.
Funding acquisition: Yenh-Chen Hsein.
Investigation: Aibo Liu.
Methodology: Chien-Chang Lee.
Project administration: Chien-Chang Lee.
Resources: Kuihai Wu.
Supervision: Yenh-Chen Hsein, Chien-Chang Lee.
Writing – original draft: Hua Yu, Chien-Chang Lee.
Writing – review & editing: Debra W Yen, Meng-tse G. Lee.
. Fleischmann C, Scherag A, Adhikari NK, et al. Assessment of global incidence and mortality of hospital-treated sepsis
. Current estimates and limitations. Am J Respir Crit Care Med 2016;193:259–72.
. Gary T, Mingle D, Yenamandra A. The Evolving Definition of Sepsis
. arXiv preprint arXiv:160907214 2016.
. Stevenson EK, Rubenstein AR, Radin GT, et al. Two decades of mortality trends among patients with severe sepsis
: a comparative meta-analysis. Crit Care Med 2014;42:625.
. Martin GS. Sepsis
, severe sepsis
and septic shock: changes in incidence, pathogens and outcomes. Expert Rev Anti Infect Ther 2012;10:701–6.
. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis
and septic shock (sepsis
-3). JAMA 2016;315:801–10.
. Shankar-Hari M, Phillips GS, Levy ML, et al. Developing a new definition and assessing new clinical criteria for septic shock: for the Third International Consensus Definitions for Sepsis
and Septic Shock (Sepsis
-3). JAMA 2016;315:775–87.
. Jones AE, Trzeciak S, Kline JA. The Sequential Organ Failure Assessment score for predicting outcome in patients with severe sepsis
and evidence of hypoperfusion at the time of emergency department presentation. Crit Care Med 2009;37:1649.
. Ferreira FL, Bota DP, Bross A, et al. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA 2001;286:1754–8.
. Freund Y, Lemachatti N, Krastinova E, et al. Prognostic accuracy of sepsis
-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA 2017;317:301–8.
. Finkelsztein EJ, Jones DS, Ma KC, et al. Comparison of qSOFA
and SIRS for predicting adverse outcomes of patients with suspicion of sepsis
outside the intensive care unit. Crit Care 2017;21:73.
. Jaimes F, Leon A, Ascuntar J, et al. 1414: Prospective validation of qSOFA
in emergency services: a useless bedside clinical score. Crit Care Med 2016;44:429.
. Klouwenberg PMK, Cremer OL, van Vught LA, et al. Likelihood of infection in patients with presumed sepsis
at the time of intensive care unit admission: a cohort study. Crit Care 2015;19:319.
. Lee C-C, Chen S-Y, Tsai C-L, et al. Prognostic value of mortality in emergency department sepsis
, and C-reactive protein in patients with sepsis
at the emergency department. Shock 2008;29:322–7.
. Kushimoto S, Shibata Y, Koido Y, et al. The clinical usefulness of procalcitonin
measurement for assessing the severity of bacterial infection in critically ill patients requiring corticosteroid therapy. J Nippon Med Sch 2007;74:236–40.
. Hosmer DW, Hosmer T, Le Cessie S, et al. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 1997;16:965–80.
. Cook NR. Assessing the Incremental Role of Novel and Emerging Risk Factors. Curr Cardiovasc Risk Rep 2010;4:112–9.
. Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med 2009;150:795–802.
. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157–72. [discussion 207-112].
. Williams JM, Greenslade JH, McKenzie JV, et al. Systemic inflammatory response syndrome, quick sequential organ function assessment, and organ dysfunction: insights from a prospective database of ED patients with infection. Chest 2017;151:586–96.
. Vincent J-L, Martin GS. Levy MM. qSOFA
does not replace SIRS in the definition of sepsis
. Crit Care 2016;20:210.
. Wacker C, Prkno A, Brunkhorst FM, et al. Procalcitonin
as a diagnostic marker for sepsis
: a systematic review and meta-analysis. Lancet Infect Dis 2013;13:426–35.
. Wu JY, Lee SH, Shen CJ, et al. Use of serum procalcitonin
to detect bacterial infection in patients with autoimmune diseases: A systematic review and meta-analysis. Arthritis Rheum 2012;64:3034–42.
. Lu X-L, Xiao Z-H, Yang M-Y, et al. Diagnostic value of serum procalcitonin
in patients with chronic renal insufficiency: a systematic review and meta-analysis. Nephrol Dial Transplant 2012;28:122–9.
. Lee SH, Chan RC, Wu JY, et al. Diagnostic value of procalcitonin
for bacterial infection in elderly patients–a systemic review and meta-analysis. Int J Clin Pract 2013;67:1350–7.
. Lai CC, Chen SY, Wang CY, et al. Diagnostic value of procalcitonin
for bacterial infection in elderly patients in the emergency department. J Am Geriatr Soc 2010;58:518–22.
. Yu C-W, Juan L-I, Hsu S-C, et al. Role of procalcitonin
in the diagnosis of infective endocarditis: a meta-analysis. Am J Emerg Med 2013;31:935–41.
. Wu C-W, Wu J-Y, Chen C-K, et al. Does procalcitonin
, C-reactive protein, or interleukin-6 test have a role in the diagnosis of severe infection in patients with febrile neutropenia? A systematic review and meta-analysis. Support Care Cancer 2015;23:2863–72.
. Kratzsch J, Petzold A, Raue F, et al. Basal and stimulated calcitonin and procalcitonin
by various assays in patients with and without medullary thyroid cancer. Clin Chem 2011;57:467–74.
. Becker KL, Snider R, Nylen ES. Procalcitonin
assay in systemic inflammation, infection, and sepsis
: clinical utility and limitations. Crit Care Med 2008;36:941–52.
. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 2007;115:928–35.
. Cook NR. Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 2008;54:17–23.
. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology 2010;21:128–38.
. Fernando SM, Tran A, Taljaard M, et al. Prognostic accuracy of the quick sequential organ failure assessment for mortality in patients with suspected infection: a systematic review and meta-analysis. Ann Intern Med 2018;168:266–75.
procalcitonin; qSOFA; quick SOFA score; sepsis
Supplemental Digital Content
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc. All rights reserved.