A Prognostic Model for Critically Ill Children in Locations With Emerging Critical Care Capacity*

OBJECTIVES: To develop a clinical prediction model to risk stratify children admitted to PICUs in locations with limited resources, and compare performance of the model to nine existing pediatric severity scores. DESIGN: Retrospective, single-center, cohort study. SETTING: PICU of a pediatric hospital in Siem Reap, northern Cambodia. PATIENTS: Children between 28 days and 16 years old admitted nonelectively to the PICU. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Clinical and laboratory data recorded at the time of PICU admission were collected. The primary outcome was death during PICU admission. One thousand five hundred fifty consecutive nonelective PICU admissions were included, of which 97 died (6.3%). Most existing severity scores achieved comparable discrimination (area under the receiver operating characteristic curves [AUCs], 0.71–0.76) but only three scores demonstrated moderate diagnostic utility for triaging admissions into high- and low-risk groups (positive likelihood ratios [PLRs], 2.65–2.97 and negative likelihood ratios [NLRs], 0.40–0.46). The newly derived model outperformed all existing severity scores (AUC, 0.84; 95% CI, 0.80–0.88; p < 0.001). Using one particular threshold, the model classified 13.0% of admissions as high risk, among which probability of mortality was almost ten-fold greater than admissions triaged as low-risk (PLR, 5.75; 95% CI, 4.57–7.23 and NLR, 0.47; 95% CI, 0.37–0.59). Decision curve analyses indicated that the model would be superior to all existing severity scores and could provide utility across the range of clinically plausible decision thresholds. CONCLUSIONS: Existing pediatric severity scores have limited potential as risk stratification tools in resource-constrained PICUs. If validated, our prediction model would be a readily implementable mechanism to support triage of critically ill children at admission to PICU and could provide value across a variety of contexts where resource prioritization is important.


H
istorically, pediatric critical care has been perceived as too complex, expensive, or unethical to provide in settings where resources are scarce (1).These presumptions are countered by the fact that simple, low-cost interventions result in substantial improvements in health outcomes (2,3).Consequently, capacities for pediatric critical care are growing in many resource-limited settings (2).However, need for critical care often outstrips supply (4).Resource stewardship is essential to promote sustainability of critical care services, especially in rural regions of many low-and middle-income countries (LMICs) (1,2,5).
Risk stratification tools can help target scarce resources optimally.However, tools developed in PICUs in high-income settings are time-consuming to compute and require diagnostic tests not routinely available in resource-constrained regions of LMICs (6).Furthermore, prognosis is influenced by the level of care available and underlying host susceptibility states, and hence adapted tools are required to support context-specific clinical decision-making (5,7,8).There have been calls to validate existing severity scores and develop new risk stratification tools for resourceconstrained PICUs (5,9).Unfortunately, most studies from LMIC PICUs are limited to urban centers, hampered by small sample sizes, and use methods incompatible with development of robust clinical severity scores or prediction models (10)(11)(12)(13)(14).
In this study, first, we externally validated nine existing severity scores using data from a PICU in northern Cambodia.Second, we developed a prognostic model, specifically to support risk stratification in resource-constrained PICU contexts.

MATERIALS AND METHODS
This retrospective cohort study used consecutive admissions to the PICU at Angkor Hospital for Children (AHC), Siem Reap, Cambodia, between January 1, 2018, and January 1, 2020.Nonelective admissions of children older than 28 days old and younger than 16 years old were included.The study was approved by the AHC Research Committee (AHC 0656/20; October 2020), Cambodian National Ethics Committee for Health Research (NECHR 257; October 2020), and the Oxford Tropical Research and Ethics Committee (OxTREC 565-20; November 2020).
Procedures followed were in accordance with the ethical standards of the responsible committees on human experimentation and with the Helsinki Declaration of 1975.The study is reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines (Appendix 1, http://links.lww.com/PCC/C445)(14).

Study Setting
AHC is a pediatric healthcare organization with a nationwide catchment area.The hospital has 89 beds situated on two medical wards, a surgical ward, a special care baby unit, and neonatal and PICUs.The 14-bedded PICU has approximately 1000 annual admissions and is staffed by 30 nurses, four senior doctors, and five doctors in training.The level II unit provides mechanical and noninvasive ventilation (oxygen cylinders are delivered fortnightly), inotropic therapy, peritoneal dialysis, and specialist nursing (minimum 1:3 nursepatient ratio) for critically ill children (15).A backup generator ensures continuity of electrical supply during infrequent power outages.

Data Collection
PICU admissions were identified from the Hospital Information System and cross-checked against the unit's admission logbook.Extracted data were recorded on a structured case report form (CRF).It was not possible to blind data collectors to patient outcome.The hospital admission and PICU vital sign proforma (Appendix 2, http://links.lww.com/PCC/C445)helped standardize data extraction.All variables were prospectively defined in a data dictionary to ensure consistency of interpretation across the research team.Each CRF was reviewed by a study physician (A.C. or S.K.) in consultation with the clinical records.Data were entered into an electronic database and 10% of CRFs were reviewed by a Data Manager (P.V.) to ensure a data entry error rate less than 0.5%.

Severity Scores and Candidate Predictors
The results of two recent systematic reviews were supplemented by searching PubMed using synonyms of "pediatric" AND "severity score OR prediction model" (16,17).Forty-nine scores or models were longlisted

RESEARCH IN CONTEXT
• Capacity and demand are growing in settings with scarce resources for critical care and contextualized approaches are required to ensure sustainability.
• Generalizability of existing risk stratification tools to resource-constrained PICU contexts is uncertain.
• This study validated the performance of nine existing pediatric severity scores and developed a bespoke prognostic model for critically ill children admitted to a PICU in northern Cambodia.
(Appendix  (24).Neither the setting, population, nor outcome used for derivation were prerequisites for selection for external validation.Baseline variables at PICU admission were extracted from the clinical records.Laboratory parameters measured within 24 hours of PICU admission were considered available at admission.A sensitivity analysis restricting this period to between 2 hours prior and up to 4 hours after admission was performed (Appendix 5, http://links.lww.com/PCC/C445)(25).For derivation of the new model, candidate predictors were selected a priori based on existing literature, expert knowledge, feasibility for implementation, and availability of data in the clinical records.Variables were selected across five domains (i.e., background, illness journey, cardiovascular, respiratory, and neurologic) to ensure inclusion of important contextual determinants of outcome often neglected by clinical risk scores developed in high-income settings.The 11 selected predictors were age, comorbidity status, weight-for-age z score, estimated travel time to hospital, route of admission to PICU, heart rate, capillary refill time, respiratory rate, peripheral oxygen saturation (Spo 2 ), receipt of supplemental oxygen, and mental state.

Outcomes
The primary outcome was death during PICU admission.Patients discharged from PICU to die at home were classified as meeting the primary outcome.A sensitivity analysis excluded these patients and those whose death was judged by either study physician (A.C. or S.K.) to have been related to a separate illness acquired during the PICU stay (Appendix 5, http:// links.lww.com/PCC/C445).
The secondary outcome was death in the 12 months following PICU discharge.Caregivers of patients for whom postdischarge outcomes could not be determined from the clinical records were telephoned to ascertain vital status.

Statistical Analyses
Routinely collected data indicated that 100 deaths were expected over 2 years, which would ensure sufficient outcome events for external validation of the existing severity scores (26).At this prevalence, and assuming a conservative Nagelkerke R 2 of 0.15, up to 10 candidate predictors (events per parameter = 9.7) could be used to build the prediction model (R package: pmsampsize) (27,28).In order to include interaction terms between age and each of heart rate and respiratory rate, penalization was used to shrink regression coefficients and permit inclusion of up to 13 parameters while minimizing the risk of overfitting.
Missing data were summarized (Appendix 6, http:// links.lww.com/PCC/C445;R package: naniar) (29).Median imputation conditional on outcome status was proposed to address missingness.Sensitivity analyses comparing this to a full-case approach, as well as bestand worst-case imputation, produced similar results, confirming that median imputation was appropriate for the primary analysis.
Discrimination and calibration of each existing score was assessed by quantifying the area under the receiver operating characteristic curve (AUC; R package: pROC) (30) and plotting the proportion of admissions that met the primary outcome at each level of a score.Positive and negative likelihood ratios (PLRs and NLRs) were reported at each score's cutoff to quantify the change in pre-test probability that a PICU admission would result in death.As a rule-ofthumb, a PLR greater than 10 or NLR less than 0.1 is deemed conclusive, a PLR between 5 and 10 or NLR between 0.1 and 0.2 is considered substantial, a PLR between 2 and 5 or NLR between 0.2 and 0.5 is regarded as small but important, and a PLR between 1 and 2 or NLR between 0.5 and 1 is likely clinically insignificant (31).
The relationship between continuous predictors and PICU survival status was examined using locally weighted scatterplot smoothing.Age-specific relationships for heart rate and respiratory rate were explored to account for known changes associated with physiologic maturation.Stratum-specific odds ratios and likelihood ratio tests (LRTs) were used to identify important interactions between age and each of heart rate and respiratory rate, as well as between Spo 2 and receipt of supplemental oxygen.Penalized (ridge) logistic regression (penalty parameter = 0.026) was used to derive the model and shrink model coefficients to adjust for optimism (R package: ridge) (32).All predictors were prespecified and no predictor selection was performed during model development.
Discrimination (AUC), calibration (calibration intercept, slope, and plots), and classification indices (R package: reportROC) (33) were reported to summarize model performance.Recognizing that the relative value of a true positive (TP; PICU admission correctly identified as at high-risk of death) and false positive (FP; PICU admission incorrectly identified as at high-risk of death) will be context-dependent (e.g., depending on human and material capacities of a high-acuity area that at-risk PICU admissions might be triaged to), the clinical utility of the model was compared with the best-performing existing scores using decision curves to visualize their netbenefits over a range of clinically plausible decision thresholds (R package: dcurves) (34,35).
All analyses were done in R, Version 4.2.2 (36).

RESULTS
Between January 1, 2018, and January

Derivation of a New Prediction Model for Resource-Constrained PICU Contexts
Assessment of the relationship between continuous predictors and the primary outcome did not identify serious violations of linearity (Appendix 17, http:// links.lww.com/PCC/C445).Age-dependent relationships between the primary outcome and heart rate and respiratory rate were evident (LRT; p < 0.001).There was no evidence of interaction between Spo 2 and use of supplemental oxygen (LRT; p = 0.92).The full model, including the formulae to calculate the probability that a PICU admission will result in death, is shown in Table 1.
Discrimination of the new model (Fig. 2A; AUC, 0.84; 95% CI, 0.80-0.88)was significantly better than all existing scores (DeLong test; p < 0.001).This is in part expected for a comparison between a newly derived model and external validation of existing scores.Calibration appeared best at lower predicted probabilities (Fig. 2B), with the model underestimating risk for admissions with probabilities of death greater than 25%.Precision-recall curves for the new model and existing scores are shown in Appendix 18 (http:// links.lww.com/PCC/C445).
The ability of the model to triage PICU admissions into high-and low-acuity groups at cutoffs of 2.5%, 5%, 7.5%, 10%, and 15% is shown in Table 2; and Appendix 19 (http://links.lww.com/PCC/C445).A cutoff of 10% reflects a triage strategy whereby all PICU admissions with a predicted probability of death greater than or equal to 10% are directed to a high-acuity area and all other PICU admissions are managed on the main unit.At this cutoff, PICU admissions triaged to the highacuity area would have a probability of death almost five times that of the general PICU population (PLR, 5.75; 95% CI, 4.57-7.23),whereas the probability among those triaged to the low-acuity area would be less than half that of the general PICU population (NLR, 0.47; 95% CI, 0.37-0.59),and almost a tenth of The model estimates the log odds of death during a PICU admission, using the sum of the intercept and the predictors multiplied by their coefficients, according to the following equation: To support clinical decision-making, the output of the model (log odds) is converted into the probability that a PICU admission will result in death using the following transformation: those triaged to the high-acuity area.At the 10% cutoff, approximately 13.0% of all PICU admissions would be triaged to the high-acuity area, resulting in a ratio of 3:1 incorrect to correct (FP:TP) high-acuity triages.

Generalizability and Applicability
There is great heterogeneity in critical care provision across different resource-constrained contexts, with  Performance of the model at five cutoffs (decision thresholds or threshold probabilities).A cutoff of 10% reflects a triage strategy whereby all admissions with a predicted probability of death ≥ 10% are directed to a high-acuity area and all other admissions managed on the main unit.A decrease in threshold probability (cutoff) is associated with an increase in the sensitivity of the triage strategy for identifying high-risk admissions, at the cost of a greater proportion of admissions being directed to the high-acuity area.Additional classification indices are provided in Appendix 18 (http://links.lww.com/PCC/C445).
the relative value of a TP and FP depending on available resources.Decision curve analyses accounting for differing contexts indicate that using the model to support triage decisions could provide utility at cutoffs greater than or equal to 7.5% (Fig. 3; and Appendix 20, http://links.lww.com/PCC/C445), or simply put, in contexts where it might be desirable and feasible to manage up to a quarter (22.7%) of critical care admissions in a high-acuity area and tolerate up to 5:1 incorrect to correct (FP:TP) highacuity triages.

DISCUSSION
In this study, we report the ability of nine pediatric severity scores to risk stratify children at admission to a PICU in Cambodia and compared their performance to that of a novel prognostic model derived specifically for locations where critical care resources are scarce.Three scores (i.e., qPELOD-2, qSOFA, PAWS) had moderate diagnostic utility, but the new model proved superior and, if validated, could be used to support risk stratification of critically ill children in a variety of resource-constrained contexts.
This study provides one of few descriptions of pediatric critical care delivery in regions of LMICs with emerging critical care capacity.Children often had protracted illness journeys, consisting of multiple care encounters.The short average length of stay is striking and likely reflects the fact that in many resource-limited settings effective intensive care consists of providing simple, life-saving interventions for critically ill children with readily reversible conditions (1,3).The ~4% post-PICU discharge mortality rate is likely an underestimate due to considerable losses to follow-up (37).
Except SIRS, which is known to perform poorly for risk assessment of acutely unwell children (38), the other eight scores had comparable discrimination.However, discrimination is a poor indicator of clinical utility (39).Only three scores were associated with meaningful changes in pre-test probability that a PICU admission might result in death, such that a single cutoff could be used to triage children into high-or low-risk groups.While separate cutoffs could identify high-and low-risk PICU admissions, in settings where resources are scarce it is unclear how a middle or "indeterminate" group might be managed, and dividing PICU admissions into multiple risk categories may not be practical.
Discrimination of FEAST-PET, qSOFA, and qPELOD-2 were comparable to their original development studies (18,23,38), which may reflect similarities in the population (i.e., critically ill children), mortality outcome, and for the FEAST-PET study, contextual factors (i.e., access to care, etc.).Performance of LqSOFA was inferior to the original development study (19), which is not unexpected since LqSOFA was devised for screening outside the PICU (19,40).We note that discrimination and classification of PAWS, PEWS, and PEWS-RL were considerably worse in our study (19)(20)(21)(22).These are "diagnostic" scores developed to predict events within 24 hours of calculation, and it is therefore unsurprising that prognostication of outcomes days later is suboptimal.
The model developed in our study considers the child's background, illness journey, and vital organ function to contextualize assessment of critical illness and estimate probability of death before PICU discharge, given the resources available in a level II PICU outside a major urban center in Southeast Asia.Discrimination of our model was better than all nine severity scores.It provided good diagnostic value and was well calibrated over the threshold probabilities (cutoffs) of interest.In contexts where it might be feasible to resource a particular clinical area to manage up to a quarter of the highest risk PICU admissions, the model, if validated, could identify children who might benefit most from being cared for in such an area.Importantly, as the output of the model is continuous (as opposed to discrete as is the case for point-based scores), the cutoff for triage to the high-acuity area could be tailored to account for staffing availability, seasonal bed-pressures, hospital policy, and other dynamic contextual factors.Of note, no risk prediction model should replace clinical assessment; rather it could help healthcare professionals efficiently organize care for critically ill children.
In our study, best-practice methods were followed in model development, with particular care taken to prespecify and limit the number of candidate predictors and use penalized regression to avoid overfitting (14).Our analytical approach acknowledged that the relative importance of a TP and FP is context-dependent.Important contextual determinants of outcome were included in the model and likely contributed to its promising performance.

WHAT THIS STUDY MEANS
• Existing pediatric severity scores show limited clinical utility in resource-constrained PICU settings.
• Including contextual determinants of clinical outcome can support holistic assessment of critical illness and improve risk stratification.
• If validated, the prognostic model developed in this study would be a readily implementable mechanism to support resource prioritization for critically ill children cared for in locations with emerging critical care capacities.
The main limitation of the model is the lack of external validation, which would be needed before it can be recommended for clinical use.A prospective validation study is underway in which the out-of-sample performance of the model will be compared with the best-performing of the existing severity scores.Although the model appears to underestimate the risk of death at predicted probabilities greater than or equal to 25%, miscalibration in this range is likely to be inconsequential (i.e., unlikely to affect a child's triage category), as it is implausible that decision thresholds would be set so high, even in the most resource-constrained context.The study was conducted in a single center and hence findings may not be applicable to all resource-limited PICUs, particularly those in which causes of critical illness differ.Due to the retrospective nature of this study, travel time was estimated based on a child's location of residence.This may not reflect actual travel time.It was not possible to evaluate the performance of all longlisted scores.In particular, only two of the nine included scores were developed in LMICs and it is disappointing that seven LMIC-derived scores had to be excluded (18,22).The use of routine records means that measurement of clinical parameters was not standardized.However, use of clinical records did ensure the new model contains predictors feasible for collection and will hopefully increase the likelihood of successful out-of-sample validation.

CONCLUSIONS
This study presents a new prognostic model for estimating the probability that a child admitted to a PICU in a resource-constrained context will not survive to discharge.The model contains predictors from multiple domains to ensure holistic assessment of critical illness.It outperformed nine existing severity scores and, if validated, offers a readily implementable and flexible mechanism to support risk stratification of critically ill children in resource-constrained contexts.For information regarding this article, E-mail: arjun.chandna@oriel.ox.ac.ukDe-identified, individual participant data from this study will be available to researchers whose proposed purpose of use is approved by the data access committees at the Angkor Hospital for Children and the Mahidol Oxford Tropical Medicine Research Unit.Inquiries or requests for the data may be sent to datasharing@tropmedres.ac or arjun@tropmedres.ac.

Figure 1 .
Figure 1.Summary performance measures of the quick Pediatric Logistic Organ Dysfunction-2 score.Top left: Discrimination (area under the receiver operating characteristic curve, 0.75; 95% CI, 0.70-0.80).Top right: Calibration.Proportion of admissions at each level of the score that died during their PICU stay; error bars indicate Wilson 95% CIs.Bottom left: Negative (black line) and positive (gray line) likelihood ratios at different cutoffs, illustrated on a log 10 scale.Bottom right: Sensitivity (black line) and specificity (gray line) at different cutoffs; gray shaded ribbons indicate 95% CIs.NLR = negative likelihood ratio, PLR = positive likelihood ratio.

a
Assessed by a Principal Investigator (C.T.) blinded to outcome status using the following working definition: any previous health condition known to be present at PICU admission severe enough to require specialty pediatric care and probably a period of hospitalization over 12 mo (51).b Calculated (R package: zscorer) (52) using World Health Organization (children < 10 yr) (53, 54) and U.S. Centers for Disease Control and Prevention (children ≥ 10 yr) (55) reference ranges.If weight was not recorded at PICU admission the closest value during the same hospital stay was used.c Travel by road estimated using GoogleMaps.d Admission from acute medical or surgical ward.e Capillary refill time > 2 s. f Glasgow Coma Scale < 15 and/or Alert/Voice/Pain/Unresponsiveness scale < A. g Estimated using penalized (ridge) logistic regression; penalty parameter = 0.026 (R package: ridge) (32).Predictors spanning the five clinical domains are presented along with their regression coefficients.

Figure 2 .
Figure 2. Discrimination and calibration of the new model.A, Discrimination of the new model.Perfect discrimination is indicated by an area under the receiver operating characteristic curve (AUC) of 1.0.B, Calibration of the new model.Dashed line indicates perfect calibration.Solid line indicates calibration of the model, with 95% CI (gray ribbon).Rug plots indicate distribution of predicted risks for participants who did (top) and did not (bottom) meet the primary outcome.

Figure 3 .
Figure 3. Clinical utility of the new model across a range of plausible decision thresholds.A cutoff (decision threshold or threshold probability) of 10% reflects a triage strategy whereby all admissions with a predicted probability of death greater than or equal to 10% are directed to a high-acuity area and all other admissions managed on the main unit.The net benefit of the new model (dotted line) is compared with "Treat All" (solid line; all PICU admissions are triaged to the high-acuity area) and "Treat None" (shortdash; no PICU admissions are triaged to the high-acuity area) strategies, as well as the three existing scores that demonstrated potential for stratifying admissions into low-and high-risk groups from the external validation (Pediatric Advanced Warning Score [PAWS] = dot-shortdash; quick Sequential Organ Failure Assessment [qSOFA] = longdash; quick Pediatric Logistic Organ Dysfunction-2 [qPELOD-2] = dot-longdash).Above a cutoff of 7.5% using the new model to triage admissions appears to be the optimal strategy.A color version of the figure is provided (Appendix 20, http://links.lww.com/PCC/C445) for additional clarity.

TABLE 1 .
Prognostic Model to Estimate the Probability That a PICU Admission Will End in Death

TABLE 2 .
Ability of the Model to Triage PICU Admissions

6
Angkor Hospital for Children, Siem Reap, Cambodia.Supplemental digital content is available for this article.Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website (http://journals.lww.com/pccmjournal).Drs.Chandna, Keang, Patel, Riedel, Turner, Chanpheaktra, and Turner conceptualized the study.Dr. Chandna shortlisted the existing severity scores.Dr. Chandna, Dr. Keang, Dr. Sambou, Ms. Chhingsrean, and Ms. Sina collected the data.Drs.Chandna and Keang reviewed the case report forms.Dr. Chandna, Dr. Sambou, Ms. Chhingsrean, Ms. Sina, and Mr. Vichet entered the data.Mr. Vichet conducted data entry checks.Dr. Riedel calculated the travel times.Dr. Chandna curated and cleaned the data.Dr. Chandna, Dr. Keang, VM, Dr. Chanpheaktra, and Dr. Turner selected the variables for the clinical prediction model.Dr. Chandna conducted the analysis under the supervision of Drs.Mwandigha, Koshiaris, and Perera-Salazar.Dr. Chandna wrote the original draft of the article.All authors reviewed, edited, and approved the article.Dr. Chandna verified the underlying data.This research was funded by the U.K. Wellcome Trust (219644/Z/19/Z).Dr. Chandna, Ms. Sambou, Ms. Chhingsrean, Ms. Sina, Mr. Vichet, Dr. Patel, Dr. Habsreng, Dr. Perera-Salazar, and Dr. Turner received support for article research from Wellcome Trust/COAF (220211).Dr. Mwandigha received funding from Oxford University through the NHIR Community HealthTech and Imperial College London through the Bill and Melinda Gates Foundation.Dr. Koshiaris is supported by a Wellcome Trust/Royal Society Sir Henry Dale Fellowship (211182/Z/18/Z).Dr. Perera-Salazar acknowledges part support from the NIHR Applied Research Collaboration Oxford & Thames Valley, the NIHR Programme Grants for Applied Research, the NIHR Oxford Medtech and In Vitro Diagnostics Cooperative, and the Oxford Martin School.Drs.Perera-Salazar's and Turner's institutions received funding from Wellcome Trust.The remaining authors have disclosed that they do not have any potential conflicts of interest.