Subgroup analyses confirmed that qSOFA outperformed SIRS across different settings and patient populations (Table 3). The difference between qSOFA and SIRS remained the same irrespective of study design (prospective vs retrospective), patient populations (ED vs ICU), countries (developed vs resource-limited), or study quality (all studies vs low-quality studies excluded) (Table 3). In these analyses, an adjusted statistical significance level (p < 0.005) was used to correct for multiple comparisons. Despite an increased stringency level, the difference between qSOFA and SIRS was statistically significant in most subgroups, except in one subgroup, namely, the ICU studies (Table 3).
For primary outcome, qSOFA showed a lower sensitivity (56.39%) but a higher specificity (78.84%), whereas SIRS showed a higher sensitivity (74.58%) but a lower specificity (35.24%) (Table 2). For secondary outcome, a similar trend was observed; the qSOFA showed a lower sensitivity (54.86%) but a higher specificity (74.22%), whereas SIRS showed a higher sensitivity (73.02%) but a lower specificity (39.64%). Thus, in predicting both primary and secondary outcomes, qSOFA was more specific, whereas SIRS was more sensitive.
As expected, the included studies were highly heterogenous, with I2 over 85% (indicating high heterogeneity) in most analyses. An extensive search for the source of heterogeneity across predefined subgroups was performed. In this analysis, pooled AUCs were compared across prospective and retrospective studies, ED patients and ICU patients, developed countries and resource-limited countries, and between all studies and low-quality studies excluded. This analysis did not identify any obvious source of heterogeneity in either primary outcome or secondary outcomes (Supplementary File 3, Supplemental Digital Content 3, http://links.lww.com/CCX/A96; and Supplementary File 4, Supplemental Digital Content 4, http://links.lww.com/CCX/A97).
Sensitivity analysis was performed to examine the robustness of the findings of the meta-analysis. Because the primary outcome was reported as hospital mortality (67%) or 28- or 30-day mortality (35%), we analyzed separately the studies with hospital mortality and 28- or 30-day mortality (Table 3). This analysis showed that the pooled estimates were likely similar among the hospital mortality (AUCqSOFA, 0.704; 95% CI, 0.683–0.725), 28- or 30-day mortality (AUCqSOFA, 0.694; 95% CI, 0.669–0.720), and all mortality (AUCqSOFA, 0.702; 95% CI, 0.685–0.718). Because a small number of studies did not fully meet the quality assessment criteria (Supplementary File 5, Supplemental Digital Content 5, http://links.lww.com/CCX/A98), we repeated the analysis after excluding these studies. The findings of the repeated analysis did not differ significantly from the original findings (Table 3).
Two additional analyses were performed to assess the effect of potential bias on our findings. First, an influence analysis was performed on both SIRS and qSOFA, across all metrics (AUC, OR, sensitivity, and specificity) and in both primary and secondary outcomes. This analysis showed that no individual study exerted a dominant effect on the pooled estimates (Supplementary File 6, Supplemental Digital Content 6, http://links.lww.com/CCX/A99; legend: influence meta-analysis on AUC [qSOFA for mortality prediction]). Second, a cumulative analysis was performed across the same parameters (AUC, OR, sensitivity, and specificity) and in both primary and secondary outcomes. It showed that the evidence was consistent over time, with the pooled estimates and their CIs stabilized as evidence accumulated over time. Importantly, the pool estimates remained unchanged even as more studies were added (Supplementary File 7, Supplemental Digital Content 7, http://links.lww.com/CCX/A100; legend: cumulative meta-analysis on AUC [qSOFA for mortality prediction]), suggesting that adding more studies is unlikely to change the existing body of evidence.
Finally, we compared our meta-analysis to other recently published meta-analyses (Supplementary File 8, Supplemental Digital Content 8, http://links.lww.com/CCX/A101). In terms of sensitivity and specificity, these meta-analyses showed similar findings to our findings (qSOFA is more specific, whereas SIRS is more sensitive). However, these meta-analyses did not perform analysis on AUC or OR, with the exception of one study by Song et al (11), which had a limited analysis on AUC and was performed on a smaller sample size including 23 studies (n = 146,551). Overall, these meta-analyses did not conclusively show whether qSOFA outperforms SIRS. On July 1, 2019, we updated our literature search and identified four additional studies that performed a head-to-head comparison of qSOFA versus SIRS. All four studies showed qSOFA outperforms SIRS (Supplementary File 9, Supplemental Digital Content 9, http://links.lww.com/CCX/A102), which is consistent with our original finding.
Sepsis is characterized by life-threatening organ dysfunction triggered by infection. It induces a myriad of host response patterns, resulting in highly varied clinical courses/manifestations in different patients. This vast heterogeneity makes it challenging to develop a prognostic tool that can reliably forecast disease progression for each and every sepsis patient. The recently introduced qSOFA faces such a challenge; its acceptance by clinicians is predicated on the evidence that qSOFA can predict outcome, consistently and reliably, across different patient populations and clinical settings. This analysis, based on 121 studies (n = 1,716,017), is the largest meta-analysis to date that evaluates the ability of qSOFA to predict sepsis outcomes. Its findings confirmed that qSOFA score predicts sepsis outcome and this predictive performance is consistent across different patient populations and clinical settings. This analysis also revealed the performance of qSOFA to be modest (e.g., AUC 0.702 for mortality); nevertheless, qSOFA consistently outperformed SIRS criteria irrespective of study design (prospective vs retrospective), patient populations (ED vs ICU), or geography (developed vs resource-limited). Collectively, these findings provide a comprehensive evidence base to inform the current understanding of the qSOFA score in predicting sepsis outcome.
This meta-analysis differs from previous meta-analyses on three major aspects. First, our meta-analysis is the largest study conducted so far; it has included 121 studies with over 1.7 million study participants. All previous meta-analyses included much smaller sample sizes. Second, analyses of AUC and ORs provide important information regarding the predictive performance of qSOFA. This approach (using AUC and ORs) was adopted by the original qSOFA authors (3). Our meta-analysis provides the most extensive analyses, to date, on AUC and ORs of qSOFA score. In contrast, previous meta-analyses did not provide AUC/OR data except for one study (Song et al ), which performed a limited analysis on AUC. Third, by providing extensive analyses on AUC and ORs, our findings provide strong evidence that qSOFA outperforms SIRS. Previous meta-analyses do not provide such high-level evidence because they did not perform an extensive AUC and odd ratios analysis. In summary, our meta-analysis differs from other meta-analyses by having the largest sample size, a similar methodologic approach with the original qSOFA study and compelling evidence that qSOFA outperforms SIRS.
qSOFA was designed as a clinical prompt to assist clinicians to identify high-risk patients (3). In this aspect, qSOFA share some common characteristics with SIRS. As was recently shown by an extensive analysis of a large dataset (n = 1,171,797), SIRS also predicts sepsis outcomes (e.g., mortality) (9). However, qSOFA and SIRS use different parameters; qSOFA includes blood pressure, respiratory rate (> 22 breath/min), and changes in mental state, whereas SIRS includes respiratory rate (> 20 breaths/min), temperature, white cell count, and heart rate. Unsurprisingly, the two clinical scores display different sensitivity and specificity (qSOFA more specific and SIRS more sensitive), as demonstrated by our findings. Neither score is perfect because each score has its own limitations—qSOFA has a lower sensitivity (i.e., it may miss those individuals with subclinical organ dysfunction), whereas SIRS has a lower specificity (i.e., it may cause unnecessary testing in low-risk patients). In practice, the choice (qSOFA vs SIRS) is based on an optimal balance between sensitivity and specificity. The original qSOFA authors (the Sepsis-3 Task Force) had addressed this issue by calculating global performance parameters (e.g., AUC) and they compared the relative performance of qSOFA and SIRS by using these global parameters (3). In this meta-analysis, we used a similar approach—it showed that the combined analysis by AUC, OR, and sensitivity/specificity provided a more accurate evaluation of the clinical scores’ performance than using sensitivity/specificity alone.
This study has several strengths, including a large sample size, an inclusion of all relevant metrics (AUC, OR, sensitivity, and specificity) and the consistency of its findings. We used an exhaustive search strategy to identify both published and unpublished studies. This results in a large sample size (121 studies), providing us with an increased statistical power to detect a difference between qSOFA and SIRS. By analyzing global metrics (such as AUC and OR), we were able to demonstrate that the overall performance of qSOFA was better than SIRS, a finding not shown by previous meta-analyses (10–16). This difference in AUC between qSOFA and SIRS seems consistent because the same difference was observed across different clinical settings, patient populations, or study design.
This study focused on criterion validity of qSOFA; it does not address other important aspects of sepsis diagnosis, such as content validity and construct validity (20, 21). As stated by the original authors, qSOFA is not intended to be used as diagnostic criteria for sepsis; rather, qSOFA was designed as a clinical prompt to alert clinicians to consider the diagnosis of sepsis (22, 23). The diagnosis of sepsis requires meeting a different set of criteria, namely the international consensus definition of sepsis (Sepsis-3), which has a higher content validity and construct validity than qSOFA score (20, 21). The more formal Sepsis-3 criteria lack flexibility in enabling early recognition of sepsis (it requires laboratory tests to be performed, which can be time consuming and costly). qSOFA addresses this limitation by possessing three desirable characteristics of a less formal bedside tool, namely, 1) low measurement burden; 2) reproducibility; and 3) timeliness (20).
Like all clinical prediction tools, there is an inherent risk in using qSOFA score in practice. Some qSOFA-negative patients do develop organ failure, and these false-negative cases can result in serious consequence (i.e., missed treatment opportunity). Thus, a negative qSOFA requires clinicians to continue to look elsewhere for evidence of sepsis. This raises the question of whether combining qSOFA with other tools, such as SIRS, may reduce false-negatives cases. Our findings suggested that qSOFA and SIRS do have complementary strengths; qSOFA is more specific, whereas SIRS is more sensitive. An intriguing next question is, therefore, whether a combined qSOFA/SIRS score may improve the overall prediction accuracy. Such questions should be addressed in future studies.
The qSOFA was designed to be used in the non-ICU setting. In this meta-analysis, a majority of the studies were performed in this setting, including 67 studies performed in ED and 12 studies performed in other non-ICU settings. The remaining studies included 28 studies performed in ED/ICU, and 13 studies performed exclusively in ICU. Overall, the relative proportions of distribution were 56% (ED), 10% (other non-ICU settings), 23% (ED/ICU), and 11% (ICU only).
As expected, heterogeneity was evident across the entire dataset. Despite an extensive search, the sources of heterogeneity could not be identified. There are several explanations for this. First, the subgroup analyses may have excluded key factors that had contributed to heterogeneity (e.g., timing of qSOFA measurement or stage of illness); however, information on these additional variables was not available in many studies, thereby precluding their analyses. Second, there were low number of studies in some subgroups, making them underpowered to detect a statistically significant difference across subgroups. Third, traditional metrics to define heterogeneity (e.g., patient populations, study design, and settings) may have been inadequate. Emerging evidence from “omics” studies has revealed that sepsis subtypes (“endotypes”) are present, but they are usually undetectable by routine clinical evaluation or conventional laboratory tests (24). These sepsis subtypes may have contributed to the heterogeneity observed in this meta-analysis.
It is expected that new qSOFA studies will continue to emerge, given the ease of qSOFA measurement and the low cost of performing such studies. A recent search in PROSPERO (a registry for meta-analyses) indicates that there are at least 12 meta-analyses on qSOFA, with some published but a large majority are still in progress. Therefore, an important question is whether adding new findings or future studies may change our findings. In our opinion, the additional studies are unlikely to change our findings for two reasons. First, our meta-analysis has a large sample size (121 studies consisting of 1,716,017 patients)—this generates a point estimate (AUROCqSOFA, 0.70) with a very narrow CI (0.69–0.72). Thus, adding more studies to the dataset is unlikely to narrow this CI any further. Second, we find that the point estimate stabilizes over time (as shown by our cumulative meta-analysis), and thus, adding more studies is unlikely to change the final point estimate. We expect that future studies are likely to gravitate toward this final point estimate, as predicted by the well-established regression to mean principle (25).
This study has several limitations. First, it did not assess the incremental predictive validity of qSOFA. Our analyses were limited to analyzing the effect of having two or more qSOFA criteria fulfilled; the effect of having only one qSOFA criterion fulfilled remains unknown. Second, our analysis did not consider the effect of the timing of measurement. This needs to be addressed in future studies. Third, most included studies did not provide data on the component variable of either qSOFA or SIRS. Thus, the contribution of individual component (e.g., respiratory rate) to the overall predictive performance is unclear.
In conclusion, we found that qSOFA score has a modest ability to predict sepsis outcomes, but its predictive performance is better than SIRS. The higher performance of qSOFA over SIRS is consistent in different patient populations and across a diverse range of settings. However, our findings are limited by the presence of significant heterogeneity, which cannot be adequately explained by subgroup analyses.
1. Reinhart K, Daniels R, Kissoon N, et al. Recognizing sepsis
as a global health priority - A WHO resolution.N Engl J Med2017377414–417
2. Seymour CW, Gesten F, Prescott HC, et al. Time to treatment and mortality
during mandated emergency care for sepsis
.N Engl J Med20173762235–2244
3. Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis
: For the third international consensus definitions for sepsis
and septic shock (Sepsis
4. Singer M, Deutschman CS, Seymour CW, et al. The third international consensus definitions for sepsis
and septic shock (Sepsis
5. Askim Å, Moser F, Gustad LT, et al. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis
- A prospective study of patients admitted with infection to the emergency department.Scand J Trauma Resusc Emerg Med20172556
6. Churpek MM, Snyder A, Han X, et al. Quick sepsis
-related organ failure assessment, systemic inflammatory response syndrome
, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit.Am J Respir Crit Care Med2017195906–911
7. Dorsett M, Kroll M, Smith CS, et al. qSOFA has poor sensitivity for prehospital identification of severe sepsis
and septic shock.Prehosp Emerg Care201721489–497
8. Bone RC, Balk RA, Cerra FB, et al. Definitions for sepsis
and organ failure and guidelines for the use of innovative therapies in sepsis
. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine.Chest19921011644–1655
9. Kaukonen KM, Bailey M, Pilcher D, et al. Systemic inflammatory response syndrome
criteria in defining severe sepsis
.N Engl J Med20153721629–1638
10. Fernando SM, Tran A, Taljaard M, et al. Prognostic accuracy of the quick Sequential Organ Failure Assessment
in patients with suspected infection: A systematic review and meta-analysis
.Ann Intern Med2018168266–275
11. Song JU, Sin CK, Park HK, et al. Performance of the quick sequential (sepsis
-related) organ failure assessment score as a prognostic tool in infected patients outside the intensive care unit: A systematic review and meta-analysis
12. Serafim R, Gomes JA, Povoa P. A Comparison of the quick-SOFA and systemic inflammatory response syndrome
criteria for the diagnosis of sepsis
and prediction of mortality
13. Maitra S, Som A, Bhattacharjee S. Accuracy of quick Sequential Organ Failure Assessment
(qSOFA) score and Systemic Inflammatory Response Syndrome
(SIRS) criteria for predicting mortality
in hospitalized patients with suspected infection: A meta-analysis
of observational studies.Clin Microbiol Infect2018241123–1129
14. Jiang J, Yang J, Mei J, et al. Head-to-head comparison of qSOFA and SIRS criteria in predicting the mortality
of infected patients in the emergency department: A meta-analysis
.Scand J Trauma Resusc Emerg Med20182656
15. Franchini S, Scarallo L, Carlucci M, et al. SIRS or qSOFA? Is that the question? Clinical and methodological observations from a meta-analysis
and critical review on the prognostication of patients with suspected sepsis
outside the ICU.Intern Emerg Med201914593–602
16. Liu YC, Luo YY, Zhang X, et al. Quick Sequential Organ Failure Assessment
as a prognostic factor for infected patients outside the intensive care unit: A systematic review and meta-analysis
.Intern Emerg Med201914603–615
17. Moher D, Liberati A, Tetzlaff J, et al; PRISMA GroupPreferred reporting items for systematic reviews and meta-analyses: The PRISMA statement.BMJ2009339b2535
18. Whiting P, Rutjes AW, Reitsma JB, et al. The development of QUADAS: A tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews.BMC Med Res Methodol2003325
20. Angus DC, Seymour CW, Coopersmith CM, et al. A framework for the development and interpretation of different sepsis
definitions and clinical criteria.Crit Care Med201644e113–e121
21. Seymour CW, Coopersmith CM, Deutschman CS, et al. Application of a framework to assess the usefulness of alternative sepsis
criteria.Crit Care Med201644e122–e130
22. Singer M, Shankar-Hari M. qSOFA, cue confusion.Ann Intern Med2018168293–295
23. Vincent JL, Martin GS, Levy MM. qSOFA does not replace SIRS in the definition of sepsis
24. Scicluna BP, van Vught LA, Zwinderman AH, et al. Classification of patients with sepsis
according to blood genomic endotype: A prospective cohort study.Lancet Respir Med20175816–826
25. Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: What it is and how to deal with it.Int J Epidemiol200534215–220
meta-analysis; mortality; quick Sequential Organ Failure Assessment; sepsis; systemic inflammatory response syndrome
Supplemental Digital Content
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine