Secondary Logo

Journal Logo

The Pediatric Infectious Disease Journal

Refined Lab-score, a Risk Score Predicting Serious Bacterial Infection in Febrile Children Less Than 3 Years of Age

Leroy, Sandrine MD, PhD*,†,‡; Bressan, Silvia MD, PhD§; Lacroix, Laurence MD; Andreola, Barbara MD§; Zamora, Samuel MD; Bailey, Benoit MD; Da Dalt, Liviana MD§; Manzano, Sergio MD; Gervaix, Alain MD; Galetto-Lacour, Annick MD

Author Information
The Pediatric Infectious Disease Journal: May 2018 - Volume 37 - Issue 5 - p 387-393
doi: 10.1097/INF.0000000000001915
  • Free

Abstract

Serious bacterial infection (SBI) is an important cause of morbidity and mortality in children in developed countries and accounted for more than 80% of deaths before 5 years of age.1 These serious illnesses need to be distinguished from self-limited acute illnesses that are very common in children, up to 20% of all visits to pediatric emergency departments.2 However, it is always a challenging even if common situation for physicians to distinguish between SBI whose unique clinical sign may be fever, and non–life-threatening infectious diseases.3

An early and accurate diagnosis of SBI in children is essential to reduce morbidity and mortality. However, diagnosis is rarely straightforward because of the low prevalence of SBIs. Moreover, children with SBI can present at an early stage when the severity of the illness is not apparent. In a primary care setting, less than 1% of assessed children will suffer from SBI.4 The clinician must then reassure anxious parents of healthy children while not missing the diagnosis of seriously affected children.5 Triage might be performed rapidly in a high-pressure environment, or by telephone, sometimes by a staff with limited pediatric experience. Consequently, the diagnosis can be missed at first,6 sometimes with serious consequences.7

The commonly used screening methods to discriminate nontoxic children at risk for SBI often combines clinical evaluation and laboratory variables, such as a white blood cell (WBC) count with differential, and urine analysis.8–11 More recently, the determination of C-reactive protein (CRP) and procalcitonin (PCT) concentrations have been reported to have better diagnostic accuracy,12–15 with an interesting additive value for PCT.16 However, the considerable overlap of these variables in patients with and without SBI limits their discriminative ability when applied as single predictors. A risk index score called the “Lab-score” was derived combining 3 markers—PCT, CRP, and urinary dipstick.17 The derivation yielded a 94% sensitivity (95% confidence interval [CI]: 82–99) and an 81% specificity (95% CI: 72–88) for a positive Lab-score (≥ 3),17 whereas external validations found lower area under the ROC curve (AUC), and sensitivity: AUC of 0.91 (95% CI: 0.87–0.97), with a 86% sensitivity (95% CI: 77–92)18 in one hand, and in another external validation in well-appearing infants < 3 months of age demonstrated a lower area under the ROC curve of 0.83 (95% CI: 0.80–0.86).19 The Lab-score uses dichotomized biomarkers very early in the analysis processing, and weighs the categories according to a linear risk gradient that has not been previously demonstrated. Consequently, this may have lead to a decreased power and then a reduced applicability across different settings.20 Moreover, the time interval from the onset of fever is not taken into account and may unpredictably influence the risk score results. Thus, our aim was to refine the Lab-score to improve its robustness and applicability for a broader use in clinical practice, and then to perform an external validation.

PATIENTS AND METHODS

Study Design

The Lab-score refinement was conducted as a secondary analysis of the cohort studies already conducted in Geneva University Hospital (Switzerland) during 2 time periods (from January to June 2007, and from August 2012 to June 2013), and in Padova hospital (from May to October 2005).17,18,21 Briefly, all children younger than 3 years of age consecutively admitted in emergency department of each center with fever without any identified source of infection after a careful history and a thorough physical examination were considered for inclusion. They underwent blood analysis according to the protocols in use in the centers at the time of the studies (e.g., infants 7 days to 3 months of age with rectal temperature > 38°C and children 3–36 months old, ill/toxic appearing), and urine testing (using a collection technique in accordance with the local protocols). Children with a history of antibiotic use within 48 hours before admission, vaccination during the previous 2 days, known immunodeficiencies, any chronic pathology or fever lasting longer than 5 days had not been included. The external validation was performed on another prospective cohort study conducted in Sainte-Justine Hospital, Montreal, Canada (from November 2006 to November 2007), with similar inclusion and noninclusion criteria.22 Ethical committees in each participating center had approved the respective protocols of the initial studies from which data were collected.

Outcome Definition

Children were classified on the basis of their final diagnosis into 2 groups: patients with SBI, or without SBI. The SBI group included: (1) bacteremia defined by growth of a single bacterial pathogen using standard blood culture techniques; (2) acute pyelonephritis defined by growth of a single bacterial urinary tract pathogen at ≥ 105 colony-forming units/mL and presence of a renal involvement on the dimercaptosuccinic acid (DMSA) scan performed within the first week after admission, or by any bacterial growth on urine obtained by suprapubic aspiration or ≥ 104 colony-forming units/mL of a single pathogen on urine obtained by bladder catheterization; (3) lobar pneumonia diagnosed on chest radiography and then confirmed by a blinded pediatric radiologist; (4) bacterial meningitis with a positive cerebrospinal fluid culture; (5) bone or joint infections defined as local isolation or isolation in blood culture of a microorganism with concomitant arthritis; or lastly (6) sepsis defined according to Levy et al.23 Children with negative cultures or clinical improvement without antibiotic therapy or with detection of a focal infection at follow-up were classified in the non-SBI group.

Data Collection

Clinical information, grading and duration of fever, as well as data from physical examination were recorded. WBC, absolute neutrophil count, quantitative CRP measurement, urine dipstick result and PCT level were determined. In Gevena center, all children underwent blood analysis, including blood culture, and urine culture. In Padova, toxic-appearing children had a full sepsis work-up; infants from 1 week to 3 months of age and ill-appearing children 3–36 months of age were subjected to a blood culture and a urine culture; blood culture was performed in well-appearing children 3–36 months of age only when displaying WBC > 15,000 cells/mm3 or ANC > 10,000 cells/mm3, and urine culture if urine analysis was positive for leucocyte esterase and/or nitrite test. In the Canadian validation set, all included children underwent blood culture, urine analysis and culture. Chest radiograph lumbar puncture, and early DMSA scan were left at the physician discretion.22

Statistical Analysis

We analyzed the relationships between SBI and all predictive variables using a backward, stepwise multilevel logistic-regression model, with fractional polynomial transformation for continuous variables if the model assumption of linearity was violated. General characteristics were compared between centers. In case of significant differences, multilevel models were adjusted on the group-level variable “center.” Gender and age were integrated into the model as adjustment covariables. The discriminative ability of each biomarker for SBI was evaluated by drawing ROC curves, as well as by calculating sensitivity, specificity, predictive values, and likelihood ratios (LRs) after dichotomization. Predictive variables that were independently associated with SBI were combined into a mathematical equation using their coefficients from the multilevel modeling. The discriminative ability of the derived model was estimated using AUC and compared with the initial Lab-score that was also calculated for each patient, and to each biomarker, as well as by calculating sensitivity, specificity, positive and negative predictive values, and positive and negative LR after dichotomization. In addition, we compared biomarker models using decision curve analysis (DCA), a method for evaluating the net clinical benefit of prediction models in which the benefits (true-positives) are added and the harms (false-positives) are subtracted.24 The weights assigned to true-positives and false-positives are derived from the threshold probability, defined as the minimum probability of SBI that would prompt a clinician to opt for a therapeutic action. As the threshold probabilities can vary, the net benefit was calculated across a range of probabilities, resulting in a curve. In a simplified way, net benefit results from a balance between true-positives and false-positives weighted by the “clinical price” of misdiagnosing a false-positive case. We then performed an external validation on the Canadian cohort study by calculating the refined Lab-score for each patient. The predictive abilities obtained were compared with the ones of the derivation set. Statistical analyses used Stata/SE 13 software (StataCorp, College Station, TX).

RESULTS

Patients’ Characteristics

Eight hundred seventy-seven children were considered for the Lab-score refinement study. The mean age was 10.0 months (standard deviation: 9.2, median: 7.5, interquartile range: 2.2–15.0); 272 (31%) were younger than 3 months of age; 428 (49%) were boys. SBI was diagnosed in 211 (24%) children. Among these, the main diagnoses were acute pyelonephritis in 126 (14%) and bacteremia in 56 (6%) children (Table 1). In the non-SBI group of children, the diagnoses were focal bacterial infection (e.g., media otitis) in 97 (11%) and suspected viral infection in 565 (64%).

TABLE 1.
TABLE 1.:
Patients’ Characteristics

The validation set included 347 children; 168 (48%) were male. The mean age of children was 12.0 months (standard deviation: 8.0; median: 11.0, interquartile range: 6.0–17.0); 55 (16%) were younger than 3 months. SBI was demonstrated in 55 (16%), mostly acute pyelonephritis (49; 14%).

Refined Lab-score Derivation

Age, maximal temperature, and fever duration significantly differed between centers: children were younger in Geneva during the second time-period (median: 4.0 months versus 9.0 in Geneva during the first time period and 9.5 months in Padova, P < 0.001), fever lasted significantly longer in Geneva during the first time period (median: 24 versus 3 hours in Padova and 1 day in Geneva during the second time-period; P < 0.001). Models were thus adjusted on the “center” group-level variable to take into account between-center variability. Because the assumption of linearity for PCT as a continuous variable made by the logistic regression model was not achieved (P ≤ 0.001), an attempt was made to transform PCT into a fractional polynomial function. However, the nonconvergence of the model led us to dichotomize PCT into classical categories (≤ 0.5, 0.5–2, > 2 ng/mL). PCT was then significantly associated with SBI, as well as CRP (transformed into a degree 1 polynomial function to verify linearity assumption), duration of fever, maximal temperature, WBC and urine dipstick results (Table 2). Because of colinearity between WBC and neutrophil count, only WBC was kept into the modeling. All variables were entered into the model. Age, PCT, CRP, and urine dipstick remained significantly associated with SBI after a stepwise reduction procedure with a significant contribution to the prediction according to the maximum likelihood ratio estimation. Among the several modeling built with age respectively transformed into a fractional polynomial function, a categorical variable (based on the distribution of the non-SBI children), and binary variables (using 3 months as a threshold). Age was finally kept as the simplest (i.e., binary variable), because of no difference on diagnostic accuracy between models. The fit of the final model was good (P > 0.2). Using the coefficients assigned to each predictor in the logistic regression model, we derived the prediction model,

TABLE 2.
TABLE 2.:
Uni- and Multivariate Analysis

where: (i) PCT was coded (0) if ≤ 0.5 ng/mL, (1) if > 0.5 and ≤ 2 ng/mL, and (2) if > 2 ng/mL;

  • (ii)
  • (iii) urine dipstick was coded (0) if normal, and (1) if leucocytes esterase and/or nitrites were present;
  • (iv) age was coded (0) if > 3 months, and (1) if ≤ 3 months.

The model offered an area under the ROC curve of 0.94 (95% CI: 0.93–0.96—Fig. 1), significantly higher than each biomarker area under the ROC curve ([P < 0.0001], as well as greater than that of the original Lab-score [P < 0.0001]). The DCA demonstrated that the model provided a more statistically robust test than PCT, CRP, WBC count and the original Lab-score for all threshold probabilities (Fig. 1). A threshold analysis was then conducted, to identify an optimal cutoff (Table 3): one of the thresholds studied yielded a valuable tradeoff between sensitivity and specificity: sensitivity was 96% (95% CI: 92–98), specificity 73% (95% CI: 70–77), with a significant association with SBI (odds ratio [OR] = 60.8 [95% CI: 30.9–120]). Using cutoff 2, the refined Lab-score could be turned into a simple rule (Fig. 2).

TABLE 3.
TABLE 3.:
Discriminative Ability of the Initial Lab-score on the Original Derivation and Validation Sets, and of Refined Lab-score and the Original Lab-score in the Current Derivation and Validation Sets
FIGURE 1.
FIGURE 1.:
ROC and DCA curves for the Lab-score, the refined Lab-score, procalcitonin, CRP, and neutrophil count in the derivation (A, B, and E) and validation (C, D, and F) sets. For E and F graphs, one line represents a biomarker (Procalcitonin, CRP, and neutrophil count) or a score (refined or initial Lab-score); the line “all” represents the benefit/harm curve if everyone is considered as a severe bacterial infection (and benefit from all examinations and IV antibiotics), whereas the line “none” represents the corresponding curve if no one undergoes examination nor treatment.
FIGURE 2.
FIGURE 2.:
The refined Lab-score. Legend: the high-risk group is defined based on the diagnostic values obtained with the cutoff 2: 96% sensitivity (95% CI: 92–98), 73% specificity (95% CI: 69–79), 42% positive predictive value (95% CI: 33–51), 100% negative predictive value (95% CI: 98–100).

External Validation

The external validation of the refined Lab-score produced an area under the ROC curve of 0.96 (95% IC: 0.93–0.99), significantly higher than those of any other isolated biomarker (P < 0.0001) as well as the original Lab-score (P < 0.0001—Fig. 1). The area under the DCA was similarly higher for the refined Lab-score than for all isolated biomarkers and initial Lab-score. Sensitivities for the 2 cutoffs identified in the derivation study were close to the ones on the derivation set, whereas specificities were higher (74% and 87% for cutoff 1 and 2, respectively, in the validation set versus 57% and 73% in the derivation set—Table 3).

DISCUSSION

Our study results indicate that refining the Lab-score would be of interest. Indeed, the Lab-score discriminative ability was lower over validations compared with derivation (decreased sensitivity down to 84%, decreased specificity down to 78%), indicating need for refinement. The refined Lab-score was derived, based on a statistical strategy slightly changed for maintaining biomarkers as continuous variables as long as possible, and dichotomizing them only if necessary, based on observed data distribution, instead of initially dichotomizing them.21 The same variables were finally included in the refined Lab-score, leading to an AUC significantly higher than that of the original Lab-score. Its predictive ability was also higher and remained stable during the external validation: the sensitivity was 96% in the derivation set, and 95% in the validation set; the specificity was 73% in the derivation set, and 87% in the validation set. Contrary to what we were expecting, fever duration was not retained in the final model. This may be because fever duration is implicitly related to the temporal variation of PCT and CRP, which evolves very quickly for PCT, and more slowly for the latter.12 The external validation, whose population features significantly differed from the derivation set, demonstrated that the refined Lab-score was robust enough to be applied to a different population.

Medical decision making is essentially a binary process (treat or not, pursue diagnostic investigations or not). This aspect requires critical thinking when deriving any model because most scores, probabilities, and biomarkers reflect continuous data. The choice of threshold that transforms a continuous variable into a binary decision is crucially important. All consecutive discriminative capacities will depend on the chosen cutoff. Sensitivity and specificity vary in opposite directions—as one rises, the other falls, so that the cutoff choice is a trade-off between the 2. In febrile children without source, the objective was not to replace the gold standard test for SBI diagnosis, but to offer an intermediate strategy based on the weight of missing a patient (false negative) versus performing the standard test on nondiseased patients (false positive). The tradeoff between sensitivity and specificity was chosen as to favor high sensitivity at the cost of specificity, because clinicians would not want to misdiagnose any children with SBI. The high negative predictive value also reflects the same trade, and the ability of the refined Lab-score to rule out SBI, and discharge safely children without SBI. However, the refined Lab-score sensitivity, although very high, was not 100%, so close follow-up should be ensured to identify the very small proportion of children with SBI not initially detected by the model.

Specificity reflected the number of useless hospitalizations, antibiotic treatments, or complementary examinations avoided. Therefore, a goal of 100% specificity is not necessary because the aim of an intermediate strategy is not to be as perfect as a gold standard, but to limit over-hospitalizations, and over-prescriptions of investigations and treatments. The utility of the sensitivity/specificity modeling can be raised since an experienced pediatrician will base his decision-making on his own clinical experience, and probably does not require a decisional aide. However, although triage in the emergency department may be performed rapidly, in a high-pressure environment, and by clinicians with limited pediatric experience, such algorithms are useful in that they can help organize some of the implicit criteria used by trained pediatricians, and thus help junior physicians make adequate decisions.

The refined Lab-score also offered a more interesting predictive ability than any of the clinical signs, or all biomarkers considered individually. Indeed, none of the clinical variables remained independently associated with SBI, and the area under the ROC curve of the refined Lab-score was higher than those of any other biomarker. These results agree with other recently published studies.25–27 First, the predictive value of clinical features has been extensively studied, mainly by a systematic review and meta-analysis performed by Van Den Bruel et al25 and in a large Australian multicenter cohort study.26 Both identified a few clinical signs, termed “red flags” (cyanosis, rapid breathing, poor peripheral perfusion, and petechial rash) that could be of value; however, the latter signs were still no guarantee that serious illness would not be overlooked. The authors concluded that no single clinical feature had rule-out value but some combinations could be used to exclude the possibility of serious infection.25,26 Second, a recent systematic review and meta-analysis evaluated the diagnostic value of all possible blood tests for ruling in and ruling out SBI in children in ambulatory settings and their added value after clinical signs and symptoms.27 They showed that measuring inflammatory markers in an emergency department setting can be diagnostically useful, but adequate cutoff values had to be better defined, depending on whether clinicians were trying to rule in or rule out serious infection.

There were several limitations to this study that must be addressed. First, SBI prevalence was higher than in primary setting (1%), both in the derivation (24%) and external validation (16%) sets, because of recruitment in the emergency department of tertiary care hospitals. This bias may modify predictive values, thus making applicability of the refined Lab-score less appropriate for primary care settings that usually have a lower SBI prevalence. However, sensitivities and specificities would not be impacted by such prevalence changes. Second, the complexity of the mathematical formula may reduce the user-friendliness of the score. However, we succeeded in simplifying application rules. If clinicians would like to change the threshold used, computer interfaces or smartphone applications can make it easy and friendly to use even with very complex algorithms (as it has been proposed in epilepsy for example28); this would avoid sacrificing model reliability for use considerations. Third, derivation and validation populations significantly differed on their age, fever duration, maximum temperature, and SBI rate. However, it did not impact the refined Lab-score construction as the only variable included I the age, and it was kept as a continuous variable. Moreover, it was of interest to evaluate the robustness of the refined Lab-score across various types of population.

To conclude, the refined Lab-score demonstrated higher predictive ability for SBI than the original Lab-score, with promising wider applicability across settings. These results require validation in additional populations (children less than 3 months of age) and settings.

REFERENCES

1. World Health Organization. Global Health Observatory (GHO) data. Child mortality and causes of death. Available at: http://www.who.int/gho/child_health/en/. Accessed March 29, 2018.
2. Fleming DM, Smith GE, Charlton JR, et al. Impact of infections on primary care—greater than expected. Commun Dis Public Health. 2002;5:712.
3. Watt K, Waddle E, Jhaveri R. Changing epidemiology of serious bacterial infections in febrile infants without localizing signs. PLoS One. 2010;5:e12448.
4. Van den Bruel A, Aertgeerts B, Bruyninckx R, et al. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract. 2007;57:538546.
5. Kai J. What worries parents when their preschool children are acutely ill, and why: a qualitative study. BMJ. 1996;313:983986.
6. Riordan FA, Thomson AP, Sills JA, et al. Who spots the spots? Diagnosis and treatment of early meningococcal disease in children. BMJ. 1996;313:12551256.
7. Thompson MJ, Ninis N, Perera R, et al. Clinical recognition of meningococcal disease in children and adolescents. Lancet. 2006;367:397403.
8. Baraff LJ, Bass JW, Fleisher GR, et al. Practice guideline for the management of infants and children 0 to 36 months of age with fever without source. Agency for Health Care Policy and Research. Ann Emerg Med. 1993;22:11981210.
9. Baraff LJ. Management of fever without source in infants and children. Ann Emerg Med. 2000;36:602614.
10. American College of Emergency Physicians Clinical Policies Committee; American College of Emergency Physicians Clinical Policies Subcommittee on Pediatric Fever. Clinical policy for children younger than three years presenting to the emergency department with fever. Ann Emerg Med. 2003;42:530545.
11. Kuppermann N. Occult bacteremia in young febrile children. Pediatr Clin North Am. 1999;46:10731109.
12. van Rossum AM, Wulkan RW, Oudesluys-Murphy AM. Procalcitonin as an early marker of infection in neonates and children. Lancet Infect Dis. 2004;4:620630.
13. Hsiao AL, Baker MD. Fever in the new millennium: a review of recent studies of markers of serious bacterial infection in febrile children. Curr Opin Pediatr. 2005;17:5661.
14. Fernández Lopez A, Luaces Cubells C, García García JJ, et al.; Spanish Society of Pediatric Emergencies. Procalcitonin in pediatric emergency departments for the early diagnosis of invasive bacterial infections in febrile infants: results of a multicenter study and utility of a rapid qualitative test for this marker. Pediatr Infect Dis J. 2003;22:895903.
15. Olaciregui I, Hernández U, Muñoz JA, et al. Markers that predict serious bacterial infection in infants under 3 months of age presenting with fever of unknown origin. Arch Dis Child. 2009;94:501505.
16. de Kruif MD, Limper M, Gerritsen H, et al. Additional value of procalcitonin for diagnosis of infection in patients with fever at the emergency department. Crit Care Med. 2010;38:457463.
17. Lacour AG, Zamora SA, Gervaix A. A score identifying serious bacterial infections in children with fever without source. Pediatr Infect Dis J. 2008;27:654656.
18. Galetto-Lacour A, Zamora SA, Andreola B, et al. Validation of a laboratory risk index score for the identification of severe bacterial infection in children with fever without source. Arch Dis Child. 2010;95:968973.
19. Bressan S, Gomez B, Mintegi S, et al. Diagnostic performance of the lab-score in predicting severe and invasive bacterial infections in well-appearing young febrile infants. Pediatr Infect Dis J. 2012;31:12391244.
20. Lacroix L, Manzano S, Vandertuin L, et al. Impact of the lab-score on antibiotic prescription rate in children with fever without source: a randomized controlled trial. PLoS One. 2014;9:e115061.
21. Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006;25:127141.
22. Manzano S, Bailey B, Gervaix A, et al. Markers for bacterial infection in children with fever without source. Arch Dis Child. 2011;96:440446.
23. Levy MM, Fink MP, Marshall JC, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003:31:12501256.
24. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26:565574.
25. Van den Bruel A, Haj-Hassan T, Thompson M, et al.; European Research Network on Recognising Serious Infection investigators. Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet. 2010;375:834845.
26. Craig JC, Williams GJ, Jones M, et al. The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ. 2010;340:c1594.
27. Van den Bruel A, Thompson MJ, Haj-Hassan T, et al. Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review. BMJ. 2011;342:d3082.
28. Patterson V, Singh M, Rajbhandari H, et al. Validation of a phone app for epilepsy diagnosis in India and Nepal. Seizure. 2015;30:4649.
Keywords:

fever without source; children; prediction; serious bacterial infection; algorithm

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.