Acute infections are among the most frequent reasons patients present to emergency departments (EDs). These can manifest with a wide range of severities ranging from mild infections to sepsis, a life-threatening condition (1).
A critical step in the management of patients with suspected infection is: 1) to quickly and correctly identify whether an infection is truly causing the patient’s symptoms, 2) to identify the pathogen causing the infection, and 3) to anticipate the potential clinical outcome. The most common pathogens are bacteria and viruses; however, fast and accurate differentiation between the two remains a clinical challenge. For a bacterial infection, the initial management should include cultures for pathogen identification and a timely administered broad-spectrum antibiotic with regard to sepsis due to a 4–7% increase in mortality odds per hour of delayed antibiotics (2,3). For viral or noninfectious conditions, unnecessary antibiotic treatment can result in adverse effects and an increase in antimicrobial resistance and healthcare costs. Additionally, a patient with a viral infection may require further isolation precautions and, if applicable, an antiviral agent. Furthermore, there are no established biomarkers to diagnose viral infections with certainty. The gold standard to prove an infection remains the identification of pathogens by culture or polymerase chain reaction (PCR)-based methods. However, these methods have limited utility due to rather slow turnaround times, and partly difficult interpretation. Currently, there are no highly accurate tests to diagnose and/or distinguish among bacterial infection, viral infection, and noninfectious inflammation with turnaround times that would allow rapid decision-making in the ED (4,5).
The Inflammatix Bacterial Viral Non-Infected version 2 (IMX-BVN-2) classifier evaluated here (to be launched under the name InSep; Inflammatix, Burlingame, CA) is a new host-response assay and neural network–based classifier, which has the potential to meet these needs. The assay measures 29 host mRNAs from peripheral blood and incorporates advanced machine learning to calculate three scores for predicting: 1) the likelihood of bacterial infection, 2) the likelihood of viral infection, and 3) the risk for 30-day mortality (4,6–9). In this study, we investigate its accuracy for predicting the presence of bacterial and viral infections in a final cohort of 312 prospectively enrolled patients in the ED with clinically suspected acute infections and/or sepsis with at least one vital sign change.
MATERIALS AND METHODS
Study Design and Population
We performed a prospective, observational study at the ED of the Charité—Universitätsmedizin Berlin, Campus Benjamin Franklin, Germany. The study was approved by the institutional review board and registered at the German Clinical Trials Register (ID 0017395). ED triage nurses were trained to notify the study team in cases of suspected infection. We enrolled a convenience sample of adult patients (≥ 18 yr) presenting to the ED with clinically suspected acute infections and at least one of the following: temperature greater than 38°C or less than 36°C, respiratory rate greater than 20/min, heart rate greater than 90/min, systolic blood pressure less than 100 mm Hg, or altered mental state.
Patients meeting the criteria were enrolled at the point of presentation. Those unable to consent due to altered mental state or the severity of their condition were enrolled on an interim basis, until they, or their legal guardian, were able to retroactively consent. Upon enrollment, PAXgene blood RNA tubes (PreAnalytiX, Hombrechtikon, Switzerland) were collected along with samples required for extended standard-of-care diagnostics. Patients were then examined, diagnosed, and treated according to standard-of-care.
For full diagnostics and sample processing, see Supplementary Table S1 (Supplemental Digital Content 1, http://links.lww.com/CCM/G491).
We performed a chart review to establish the adjudicated status for the presence of bacterial infection and the presence of viral infection at the time of blood sampling for each case. An expert panel consisting of two attending physicians trained in Internal Medicine and Emergency Medicine used a medical chart comprised of the medical history, physical examination findings, all available laboratory, microbiological and radiological data, and the final medical report or treatment data. Each case was adjudicated into one of four predefined categories (Ruled Out, Unlikely, Probable, and Proven) for the presence of a bacterial infection and again, separately, for a viral infection (for detailed definitions, see Supplementary Table S2, Supplemental Digital Content 2, http://links.lww.com/CCM/G492).
The two adjudicators independently made their assessments and then compared results. If they differed, the case was discussed until an agreement was made. Both adjudicators were blinded to the IMX-BVN-2 results, but not to other biomarkers, such as C-reactive protein (CRP) or procalcitonin (PCT).
Two methods were used for converting the four original assessment categories into a binary (i.e., present or absent) classification for bacterial and viral infection: “consensus adjudication” (CA) and “forced adjudication” (FA) (Supplementary Tables S3 and S4, Supplemental Digital Content 3, http://links.lww.com/CCM/G493). The more stringent CA method only considers proven (bacterial/viral) assessments as “(bacterial/viral) infection present” cases and ruled out assessments as “(bacterial/viral) infection absent” cases. The remaining cases (unlikely or probable assessments) were considered “(bacterial/viral) infection inconclusive” and removed from downstream analyses for the bacterial or viral IMX-BVN-2 score. The more lenient FA method “forced” every case into a binary classification for the presence of bacterial infection and presence of viral infection, at the risk of introducing error into the adjudicated infection status, due to ambiguous clinical presentation; all cases assessed as probable or proven became “(bacterial/viral) infection present” cases and those assessed as unlikely or ruled out became “(bacterial/viral) infection absent” cases. It is noted that (bacterial/viral) infection present cases may have been bacterialviral coinfections.
A machine-learning classifier that integrates 29 host mRNAs to diagnose bacterial and viral infections and noninfectious inflammation (IMX-BVN-1) was described previously (9). To develop IMX-BVN-1, data were conormalized from multiple sources to match a targeted diagnostic platform (the NanoString nCounter; NanoString Technologies, Seattle, WA) in order to lock the classifier and apply it in a blinded fashion. Here, an advanced machine learning approach to training classifiers on the same 29 host mRNAs was used, using a vastly expanded development dataset, split into training and validation sets (Supplementary Table S5a–c, Supplemental Digital Content 4, http://links.lww.com/CCM/G494). The resulting best classifier was named IMX-BVN-2.
Additionally, three thresholds were set in the training data such that IMX-BVN-2 scores would be segmented into four bands, targeting prespecified likelihood ratios for the rule-in and rule-out bands. The three thresholds were then locked in for validation. We next applied IMX-BVN-2 to this study’s data to calculate predicted probabilities of bacterial and viral infections for each patient. The three preestablished score thresholds separated the viral and bacterial scores into four bands each: very unlikely, unlikely, possibly, and very likely.
IMX-BVN-2 bacterial and viral scores were plotted to visually demonstrate the algorithm’s ability to separate bacterial, viral, and noninfected patients in the training set, validation set, and in this study’s patient cohort (Supplementary Fig. S1, Supplemental Digital Content 5, http://links.lww.com/CCM/G495). The training set probabilities were computed using cross-validation.
Continuous variables are presented with the median and interquartile range (IQR) and were compared using Mann-Whitney U test. Nominal variables are presented as n and column percentages and were compared using Fisher exact test. For further details regarding statistical analysis, see Supplementary Methods (Supplemental Digital Content 6, http://links.lww.com/CCM/G496), and Clinical Variables (Supplemental Digital Content 7, http://links.lww.com/CCM/G497).
We enrolled 317 consecutive adult patients presenting with signs of acute infection from February until December 2019, with five patients excluded and removed from analysis (Fig. 1). Table 1 summarizes clinical characteristics in our patient cohort. The median age was 73 years (IQR, 57–80 yr). Our cohort included a substantial proportion of older patients with comorbidities and severe illness, with 76 patients (24.4%) having a quick Sequential Organ Failure Assessment (qSOFA) score of 2 or more, 86 patients (27.7%) with malignancies, and 65 patients (20.8%) with compromised immune systems. Detailed baseline characteristics, including by CA, are available in Supplementary Tables S6–S8 (Supplemental Digital Content 8, http://links.lww.com/CCM/G498). Detailed data on disease severity and the prognostic performance of the InSep test are currently being analyzed. Briefly, 22 patients (7.1%) died in hospital and 58 (18.6%) suffered from multiorgan failure within 72 hours of admission. In total, 22 patients (7.1%) required vasopressors, whereas 17 (5.4%) were mechanically ventilated.
TABLE 1. -
Patient Demographics Segmented by Adjudicated Infection Status Established Using Forced Adjudication
||All, n = 312 (100%)
||Noninfected, n = 32 (10.3%)
||Only Bacterial, n = 194 (62.2%)
||Only Viral, n = 41 (13.1%)
||Coinfected, n = 45 (14.4%)
| Age, yr, median (IQR)
| Sex, female, n (%)
| Quick Sequential Organ Failure Assessment ≥2, n (%)
| Immunocompromised, n (%)
| Previous antibiotics, n (%)
| WBC, 10e9 cells/L, median (IQR)
| Neutropenic, n (%)
| C-reactive protein, mg/L, median (IQR)
| Procalcitonin, µg/L, median (IQR)
| IMX-BVN-2 bacterial, median (IQR)
| IMX-BVN-2 viral, median (IQR)
| IMX-BVN-2 noninfected, median (IQR)
IMX-BVN-2 = Inflammatix Bacterial Viral Non-Infected version 2, IQR = interquartile range.
For the purposes of this table, “only bacterial” does not mean the presence of a bacterial infection, but rather the presence of a bacterial infection and the absence of a viral infection. Similarly, “only viral” does not mean the presence of a viral infection, but rather the presence of a viral infection and the absence of a bacterial infection.
Results of the expert panel chart review assessments and subsequent classifications for the presence of bacterial and viral infections under FA and CA are summarized in Figure 1. Under FA, 239 patients (76.6%) had a bacterial infection present, and 86 patients (27.5%) had a viral infection present. Under CA, which excludes patients with “unlikely” and “probable” expert panel assessments, 174 patients (55.8%) had a bacterial infection present, and 75 patients (24%) had a viral infection present. A distribution of expert panel assessments is provided in Supplementary Figure S2 (Supplemental Digital Content 9, http://links.lww.com/CCM/G499) and reveals the number of noninfected patients with both bacterial and viral infections absent (32 [10.3%] under FA and 8 [2.6%] under CA) and the number of patients with bacterial-viral coinfections (45 [14.4%] under FA and 23 [7.3%] under CA).
Performance of IMX-BVN-2 by CA and FA in Comparison With Routine Biomarkers
The distribution of IMX-BVN-2 scores segmented by CA and FA adjudicated infection statuses is provided in Figure 2. To assess the performance of IMX-BVN-2, we calculated area under the receiver operating curves (AUROCs) for distinguishing the presence of bacterial or viral infection and compared the results to the routine laboratory parameters and biomarkers PCT, CRP, and WBC (Supplementary Fig. S3, Supplemental Digital Content 10, http://links.lww.com/CCM/G500). Under CA, the IMX-BVN-2 bacterial score performed with an AUROC of 0.90 (95% CI, 0.85–0.95) for distinguishing patients with proven versus ruled out bacterial infection. In comparison, the CA AUROCs for PCT, CRP, and WBC were 0.89 (95% CI, 0.84–0.94), 0.84 (95% CI, 0.77–0.90), and 0.77 (95%, CI 0.69–0.85), respectively. Under FA, which included patients with uncertain adjudications, the IMX-BVN-2 bacterial score distinguished bacterial-infection patients with an AUROC of 0.82 (95% CI, 0.77–0.86). In comparison, the FA AUROCs for PCT, CRP, and WBC were 0.80 (95% CI, 0.75–0.85), 0.79 (95% CI, 0.74–0.85), and 0.69 (95% CI, 0.63–0.76), respectively.
The IMX-BVN-2 viral score performed with an AUROC of 0.83 (95% CI, 0.77–0.89) for distinguishing patients with proven versus ruled out viral infection under CA, and expectedly, biomarkers including PCT, CRP, or WBC had very low AUROCs of less than 0.39 under CA. Similarly, IMX-BVN-2 demonstrated an AUROC of 0.82 (95% CI, 0.76–0.88) under FA, whereas other biomarkers had a low AUROCs of less than 0.4, further highlighting the potential additional value of IMX-BVN-2 for the identification of patients with viral infections.
Performance of the IMX-BVN-2 Classifier on Specific Subpopulations
Patients with bacterial infections were analyzed separately to determine whether the type of bacterium or focus of the infection affected the IMX-BVN-2 score (Table 2). Scores tended to be higher among patients with intra-abdominal or urogenital infections, compared with those with respiratory infections. Additionally, infections with Gram-negative bacteria tended to result in higher BVN-2 bacterial scores than infections with Gram-positive bacteria. IMX-BVN-2 viral scores segmented by virus-types are provided in Supplementary Table S9 (Supplemental Digital Content 11, http://links.lww.com/CCM/G501).
TABLE 2. -
Effects of Source of Infection and Bacterium on IMX-BVN-2 Bacterial Scores Among Patients With Presence of Bacterial Infection (n
||Patients With Presence of Bacterial Infection by Forced Adjudication
||Patients With Presence of Bacterial Infection by Consensus Adjudication
||IMX-BVN-2 Bacterial Score, Median (IQR)
||PCT, Median (IQR)
||IMX-BVN-2 Bacterial Score, Median (IQR)
||PCT, Median (IQR)
|Source of infection
| Soft tissue/skin/bone
| No clear focus
| Positive Gram staining bacteria
| Negative Gram staining bacteria
IMX-BVN-2 = Inflammatix Bacterial Viral Non-Infected version 2, IQR = interquartile range, PCT = procalcitonin.
IMX-BVN-2 Result Interpretation Bands Provide Clinical Actionability
Figure 3 presents the interpretation bands for IMX-BVN-2 as well as for PCT under CA using PCT cutoffs established in the Procalcitonin Antibiotic Consensus Trial (ProACT) (10). Bands for FA, as well as posttest probabilities for all IMX-BVN-2 and PCT bands, are illustrated in Supplementary Figures S4 and S5 (Supplemental Digital Content 12, http://links.lww.com/CCM/G502).
Approximately half of the patients were included in the most actionable Very Unlikely (rule-out) and Very Likely (rule-in) IMX-BVN-2 bands under CA (Fig. 3). Among 26 patients with bacterial infection absent, only one patient fell into the Very Likely Bacterial band, resulting in a 96% IMX-BVN-2 rule-in band specificity. Similarly, among 174 patients with bacterial infection present, only four fell into the Very Unlikely Bacterial band, resulting in a 98% IMX-BVN-2 rule-out band sensitivity. PCT had a 96% rule-in band specificity and 90% rule-out band sensitivity for predicting bacterial infections.
Finally, the IMX-BVN-2 viral score had a 94% rule-in band specificity and 88% rule-out band sensitivity for predicting viral infections. As there are no well-established (bio)markers for the identification of viral infections, no performance comparisons to other biomarkers are shown.
The rapid diagnosis of acute infections and sepsis in the ED remains an unmet need. In this study, we debut IMX-BVN-2, an updated, blood-based 29-messenger RNA neural network classifier for determining the likelihood of bacterial and viral infections. We prospectively validated IMX-BVN-2 in a German ED with 312 patients with clinically suspected infection with at least one vital sign change. Nearly a quarter of patients were critically ill or septic (24.4% qSOFA greater than or equal to 2). We found that IMX-BVN-2 is characterized by high accuracy for diagnosing bacterial as well as viral infections. With 89.7% of patients having infections, the cohort reflects the high pretest probability of the target population for the test.
There is still debate over whether PCT can provide sufficient accuracy and clinical utility, especially in the ED (10–13). We showed that IMX-BVN-2 seems at least as accurate as PCT for diagnosing bacterial infections when comparing AUROCs (Fig. 3). However, because PCT was used by clinician adjudicators, it is highly unlikely to be outperformed in a direct comparison. The adjudicators relied on PCT to assess the likelihood of bacterial infection in the chart review, especially for patients without positive microbiology. If PCT was false-negative or false-positive but IMX-BVN-2 correctly predicted a bacterial infection in these patients, then the measured IMX-BVN-2 accuracy would be impacted by the misleading impact of PCT on adjudicators, during statistical analysis. Blinding the adjudicators to PCT, though reflecting a nonstandard-of-care procedure, may have allowed for a less biased comparison between PCT and IMX-BVN-2 but would have potentially reduced the accuracy of the clinical adjudications and, as such, the assessment of IMX-BVN-2 performance.
Currently, there is no FDA-approved biomarker for diagnosing viral infection (a particular problem during novel viral pandemics such as severe acute respiratory syndrome coronavirus 2). With AUROCs of 0.82–0.83 for predicting the presence versus absence of viral infection, IMX-BVN-2 showed potential for diagnosing viral infections, thereby demonstrating utility for determining viral isolation precautions and/or, if applicable, antiviral treatments. As currently used biomarkers, such as CRP, are unhelpful and perhaps even misleading regarding viral infection, IMX-BVN-2 appears to be a potential unique novel biomarker for the identification of acute viral infections. Of note, compared with routine PCR-diagnostics, the IMX-BVN-2 classifier is not restricted to a predefined set of viruses but able to detect a host response to a viral infection. Since the expert panelist used the PCR as gold standard, any viral infection not covered by the PCR might have been missed.
We designed this study with as few exclusion criteria as possible to more closely align with real-world clinical practice. Validation studies of other bacterial versus viral host-response biomarkers have excluded more complex populations such as immunocompromised patients (14–17), noninfectious patients (originally suspected of infection), and patients with malignancies (15,17) in performance calculations. Noteworthy, we included all of these relevant patient populations in our cohort and were able to demonstrate high accuracy.
Our study has limitations. First, as with all clinical trials for validating diagnostics for bacterial and/or viral infections, the imperfect adjudicated infection status represents a limitation. In the absence of positive clinically applicable microbiological tests, it is often difficult to rule out bacterial or viral infections due to the limited breadth and sensitivity of most standard-of-care assays. Second, owing to the way IMX-BVN-2 was designed, the sum of the bacterial and viral scores cannot exceed 1, and so, patients with clear coinfections were unable to be classified in the “Very Likely” rule-in bands simultaneously for both the bacterial and viral scores. In this regard, our observations of lower BVN-2 bacterial scores in respiratory infections may have been driven by a higher rate of coinfections among respiratory infection patients and the limitations of IMX-BVN-2 to cover coinfections. Additionally, we showed that Gram-negative infections resulted in higher median IMX-BVN-2 and PCT scores. The comparatively lower rate of Gram-negative pathogens among cases of respiratory infections may partially explain the lower values for IMX-BVN-2 and PCT. Updating IMX-BVN-2 with expanded datasets and refined machine learning techniques may improve its overall performance, including for coinfections.
Although the IMX-BVN-2 data for this study were generated using the reference NanoString nCounter platform, a point-of-care loop-medicated isothermal amplification device currently in development will allow for the faster turnaround times required for immediate clinical decision-making compliant with time-sensitive sepsis guidelines.
In this trial, we debut the 29-messenger RNA blood-based host-response classifier IMX-BVN-2 to identify bacterial and viral infections in a real-life ED setting at a German quaternary care hospital. IMX-BVN-2 showed high accuracy for predicting the presence of both bacterial and viral infections, offering a novel and practical approach for determining the origin of an infection. If used in a point-of-care device with a turnaround time under 30 minutes, emergency physicians will be able to interpret the host’s response to a pathogen, rather than needing to chase the pathogen itself. An advantage in time and accuracy could improve patient outcomes while upholding antimicrobial stewardship (18,19).
We thank all patients participating in the study. We also thank the staff at the emergency department at Campus Benjamin Franklin at Charité—Universitätsmedizin Berlin for their contribution to this study. We also like to thank Sabrina Coyle for expert technical support and Michael Mayhew for his support and expertise regarding classifier development.
1. Álvaro-Meca A, Jiménez-Sousa MA, Micheloud D, et al.; Group of Biomedical Research in Critical Care Medicine (BioCritic): Epidemiological trends of sepsis in the twenty-first century (2000-2013): An analysis of incidence, mortality, and associated costs in Spain. Popul Health Metr. 2018; 16:4
2. Rhodes A, Evans LE, Alhazzani W, et al.: Surviving sepsis campaign: International guidelines for management of sepsis and septic shock: 2016. Crit Care Med. 2017; 45:486–552
3. Seymour CW, Gesten F, Prescott HC, et al.: Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med. 2017; 376:2235–2244
4. Sweeney TE, Shidham A, Wong HR, et al.: A comprehensive time-course-based multicohort analysis of sepsis and sterile inflammation reveals a robust diagnostic gene set. Sci Transl Med. 2015; 7:287ra71
5. Sweeney TE, Liesenfeld O, May L: Diagnosis of bacterial sepsis: Why are tests for bacteremia not sufficient? Expert Rev Mol Diagn. 2019; 19:959–962
6. Sweeney TE, Wong HR, Khatri P: Robust classification of bacterial and viral infections via integrated host gene expression diagnostics. Sci Transl Med. 2016; 8:346ra91
7. Sweeney TE, Perumal TM, Henao R, et al.: Mortality prediction in sepsis via gene expression analysis: A community approach. bioRxiv. 095489
8. Sweeney TE, Khatri P: Benchmarking sepsis gene expression diagnostics using public data. Crit Care Med. 2017; 45:1–10
9. Mayhew MB, Buturovic L, Luethy R, et al.: A generalizable 29-mRNA neural-network classifier for acute bacterial and viral infections. Nat Commun. 2020; 11:1177
10. Huang DT, Yealy DM, Filbin MR, et al.; ProACT Investigators: Procalcitonin-guided use of antibiotics for lower respiratory tract infection. N Engl J Med. 2018; 379:236–249
11. Hamade B, Huang DT: Procalcitonin: Where are we now? Crit Care Clin. 2020; 36:23–40
12. Goodlet KJ, Cameron EA, Nailor MD: Low sensitivity of procalcitonin for bacteremia at an academic medical center: A cautionary tale for antimicrobial stewardship. Open Forum Infect Dis. 2020; 7:ofaa096
13. Schuetz P, Christ-Crain M, Thomann R, et al.; ProHOSP Study Group: Effect of procalcitonin-based guidelines vs standard guidelines on antibiotic use in lower respiratory tract infections: The ProHOSP randomized controlled trial. JAMA. 2009; 302:1059–1066
14. Shapiro NI, Self WH, Rosen J, et al.: A prospective, multi-centre US clinical trial to determine accuracy of FebriDx point-of-care testing for acute upper respiratory infections with and without a confirmed fever. Ann Med. 2018; 50:420–429
15. van Houten CB, de Groot JAH, Klein A, et al.: A host-protein based assay to differentiate between bacterial and viral infections in preschool children (OPPORTUNITY): A double-blind, multicentre, validation study. Lancet Infect Dis. 2017; 17:431–440
16. Self WH, Rosen J, Sharp SC, et al.: Diagnostic accuracy of FebriDx: A rapid test to detect immune responses to viral and bacterial upper respiratory infections. J Clin Med. 2017; 6:E94
17. Srugo I, Klein A, Stein M, et al.: Validation of a novel assay to distinguish bacterial and viral infections. Pediatrics. 2017; 140:e20163453
18. Singer M, Inada-Kim M, Shankar-Hari M: Sepsis hysteria: Excess hype and unrealistic expectations. Lancet. 2019; 394:1513–1514
19. Mi MY, Klompas M, Evans L: Early Administration of antibiotics for suspected sepsis. N Engl J Med. 2019; 380:593–596