Development of a Pediatric Risk Assessment Score to Predict Perioperative Mortality in Children Undergoing Noncardiac Surgery

Abstract

BACKGROUND: 

Although there have been numerous attempts to predict perioperative mortality in adults, an objective model to predict mortality in children has not been developed. In this study, we aimed to develop a Pediatric Risk Assessment (PRAm) score to predict perioperative mortality in children undergoing noncardiac surgery.

METHODS: 

We included all children recorded in the 2012 and 2013 American College of Surgeons National Surgical Quality Improvement Program Pediatric databases in a derivation cohort and those from the 2014 database in a validation cohort. The primary outcome was the incidence of in-hospital mortality. A total of 115,229 (63%) were included in the derivation cohort and 68,194 (37%) in the validation cohort. We used multivariable logistic regression to determine the predictors for mortality and designed the PRAm score.

RESULTS: 

On the basis of the multivariable regression model, we created a simplified risk assessment tool (PRAm score) ranging from 0 to ≥9, including the presence of any comorbidities, factors of critical illness, age <12 months, the requirement for an urgent procedure, and the diagnosis of a neoplasm. The PRAm score showed an excellent discriminative ability with an apparent “optimistic” area under the receiver operating characteristic curve (AUC) of 0.950 (95% confidence interval [CI], 0.942–0.957) in the derivation cohort. In the validation cohort, we observed similar performances with an area under the “naive” receiver operating characteristic curve of 0.950 (95% CI, 0.938–0.961). The AUC was also calculated from a bootstrap procedure and then applied to the original derivation sample to estimate “optimism” for each bootstrap sample with an AUC of 0.943 (95% CI, 0.929–0.9956). The optimism in apparent performance was 0.007, corresponding to an optimism-corrected area of 0.943. Calibration was assessed graphically by plotting the observed outcome against the predicted mortality (Pearson correlation coefficient = 0.995, calibration in the large = 0.001 [P = .974], calibration slope = 0.927).

CONCLUSIONS: 

In this study, we developed a simplified PRAm tool (PRAm score) as a predictor of perioperative mortality in children undergoing noncardiac surgery. The PRAm score has excellent accuracy. In patients assigned American Society of Anesthesiologists physical status classification ≥4, there is wide variability in objectively obtained PRAm scores.

In Brief

Published ahead of print August 29, 2016.

The American Society of Anesthesiologists physical status (ASA PS) classification is used worldwide by anesthesiologists and other health care providers to characterize the preoperative PS of patients.¹ Despite the fact that it was never designed as a tool to predict perioperative risk of mortality and adverse events, the ASA PS classification is routinely used in clinical practice and research for this purpose.² ASA PS classification is a subjective score that does not include standardized criteria for defining comorbidities and does not take into account the complexity of the surgical procedure performed. In addition, important interrater variability in assignment of status has been reported in adult patients.^3,4 Interrater variability has also been reported in children, and it has been suggested that the reliability of the ASA PS classification in children is poorer than in adults.^5–7

Although there have been numerous attempts to develop models that quantify perioperative risk in adults,^8,9 no one has endeavored to design an objective model for the quantification of perioperative risk in children. Utilizing a large cohort of children undergoing noncardiac surgery, we aimed to develop a simplified Pediatric Risk Assessment (PRAm) score to predict perioperative mortality in children undergoing noncardiac surgery.

METHODS

Data Source

This study was performed using data from the 2012, 2013, and 2014 pediatric databases of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP). The ACS NSQIP Pediatric collects deidentified data on children younger than 18 years undergoing noncardiac surgery and includes 129 variables, including preoperative risk factors, intraoperative characteristics, 30-day postoperative outcomes, and mortality in both the inpatient and outpatient settings.¹⁰ A systematic sampling strategy is used to avoid bias in case selection and to ensure a diverse surgical case mix. A site’s trained and certified Surgical Clinical Reviewer captures these data using a variety of methods including medical chart review. Adverse events and comorbidities reported in the database are determined by strict inclusion criteria. To ensure the quality of the data collected, the ACS NSQIP Pediatric conducts interrater reliability audits of selected participating sites.¹¹ The results of the audits completed to date reveal an overall disagreement rate of approximately 2% for all assessed program variables. For the databases, exclusion criteria included patients aged 18 years or older, trauma cases, solid organ transplantation, and cardiac surgery. In addition, cases coming from hospitals with an interrater reliability audit disagreement rate >5% or a 30-day follow-up rate <80% were excluded.

Study Population

We included all children recorded in the 2012 and 2013 ACS NSQIP Pediatric databases in a derivation cohort and those recorded in the 2014 database in a validation cohort. The primary outcome variable for our analysis was the incidence of in-hospital mortality.

Variables

The following characteristics were considered in development of the PRAm score: age, body weight, height, gender, ASA PS classification, prematurity (<24, 24–36, and >36 weeks of gestation), type of procedure (elective versus urgent surgery), preoperative respiratory disease (eg, asthma, chronic lung or airway diseases, cystic fibrosis), preoperative oxygen supplementation, tracheostomy, liver and pancreatic diseases, diabetes, congenital heart disease (CHD), acute or chronic kidney disease, neurologic disease (eg, mental retardation, cerebral palsy, central nervous system disease, intracerebral hemorrhage, seizure), immune disease, preoperative use of steroids, neoplasm, chemotherapy, preoperative inotropic support, preoperative mechanical ventilation, preoperative cardiopulmonary resuscitation, and preoperative transfusion (defined as transfusion of whole blood or red blood cells during the 48 hours before surgery). To standardize the univariable analysis (eg, variables are either categorical or ordinal), children were stratified into 5 clinically relevant age groups: <6 months, ≥6 to 12 months, ≥1 to 6 years, ≥6 to 12 years, and ≥12 years. Surgical type was categorized based on Current Procedural Terminology (CPT) codes and divided into 9 groups of anatomically related categories: musculoskeletal procedures (CPT range, 10000–29999), respiratory/thoracic/lymphatic procedures (CPT range, 30000–32999 and 38000–39999), cardiovascular procedures (CPT range, 33001–34900), vascular procedures (CPT range, 35001–37799), upper digestive tract procedures (CPT range, 40000–43499), other digestive tract/abdominal procedures (CPT range, 43500–49429 and 49650–49999), hernia repair procedures (CPT range, 49491–49611), endocrine procedures (CPT range, 60000–60999), and other including urinary and nervous system procedures (CPT range, 50000–59999 and 61000–99999).¹²

Statistical Analysis

Categorical variables are expressed as number (percentage). Preoperative variables were compared between survivors and nonsurvivors using the χ² test. To control for possible confounding among variables, we used univariable and multivariable logistic regression using backward selection to determine the independent predictors for mortality using a univariable cutoff of P < .10 for inclusion and P > .05 for removal. The results are expressed as regression coefficient (B), and its standard error, the odds ratio as a measure of risk, the 95% confidence interval (CI), and P values were obtained from the Wald test.¹³ The overall area under the receiver operating characteristic curve (AUC) was used as a measure of discrimination between those with and without mortality, and an area ≥0.900 (corresponding to 80% of the way between random [AUC = 0.500] and perfect discrimination [AUC = 1.000]) were considered excellent discrimination.¹⁴ Additional performance measures include Brier score (squared difference between patient outcome and predicted risk).

Because our objective was to design a simple risk assessment tool that could be used in clinical practice to predict mortality in children undergoing noncardiac surgery, we designed a simplified risk score based on the predictors obtained from multivariable logistic regression and their clinical relevance using 5 factors with each factor assigned an integer value from 1 to 4 with a maximum possible score of 13. The score was assigned for every variable by multiplying the regression coefficient by 2 and rounding to the nearest integer. We also considered the clinical relevance of each parameter such that in the case of comorbidities, a single score was assigned if 1 or more of the following were present: respiratory disease, CHD, preoperative acute or chronic kidney disease, neurologic disease, and hematologic disease. Based on the number of patients allocated to each score, the score was further simplified by combining scores of 9 to 13 into a single value of ≥9.

We first assessed the discriminative ability of the risk score in the study cohort (2012–2013) based on the AUC and then validated the risk score in our validation cohort (2014). We used a bootstrap procedure that validates the model for the population (ACS NSQIP 2012–2013–2014) with a sample size of 61,077 (corresponding to one-third of the overall study cohort) with the aim to correct for optimism. Bootstrapping allows the internal validation of a predictive model by repeatedly (repetition = 500) taking a large number of samples with replacement from the original data set. Calibration was assessed graphically by plotting the observed outcome against the predicted mortality. A smooth nonparametric calibration line was created with the locally weighted scatterplot-smoothing algorithm to estimate the observed probabilities in relation to the predicted probabilities.^8,15 Pearson correlation coefficient and calibration slope were then calculated to assess their correlation across the spectrum of predicted risk.¹⁶ Calibration in the large (comparison between mean predicted versus mean observed probability) and calibration slope (regression slope of linear predictor) were calculated. The magnitude of miscalibration was calculated as the calibration slope; the closer the slope coefficient is to 1, the better the calibration.¹⁷ The AUC calculated from the derivation model was considered the “optimistic” AUC. We also calculated the AUC from a derivation cohort (ACS NSQIP 2014) defined as the “naive” AUC. The AUC from each bootstrap sample was calculated and then applied to the original derivation sample to estimate “optimism” for each bootstrap sample and subtracted from the naive AUC to get optimism-corrected AUC.^18,19

Statistical analysis was performed using STATA (version 14.1 for Mac OS; Stata Corp, College Station, TX) with 2-tailed P < .05 considered as statistically significant.

RESULTS

Among the 183,423 children included in the 2012, 2013, and 2014 ACS NSQIP database, 115,229 (63%) were included in the derivation cohort and 68,194 (37%) in the validation cohort. The incidence of mortality was 0.5% (563/115,229) in the derivation cohort and 0.4% (290/67,904) in the validation cohort.

In the derivation cohort (Table 1), all potential predictors were evaluated for inclusion in the predictive model. Of these, 13 were retained in the final multivariable logistic regression model (Table 2): hematologic disorder, preoperative transfusion, CHD, neurologic disease, urgent procedure, respiratory disease, preoperative cardiopulmonary resuscitation, acute kidney injury, chemotherapy, preoperative inotropic support, age <12 months, preoperative mechanical ventilation, and neoplasm. The multivariable logistic regression model showed an excellent discriminative ability to predict in-hospital mortality with an AUC of 0.958 (95% CI, 0.953–0.965). The Brier score in the derivation cohort was 0.0041.

Table 1.: Demographic Characteristics and Comorbidities in the Derivation Cohort

Table 2.: Variables Obtained From Multivariable Logistic Regression Analysis to Predict Postoperative Mortality

The weighted variables necessary to determine the PRAm score are summarized in Table 3. The PRAm score showed an excellent discriminative ability with an apparent “optimistic” AUC of 0.950 (95% CI, 0.942–0.957) in the derivation cohort. In the validation cohort, we observed similar performances with an area under the “naive” receiver operating characteristic curve of 0.950 (95% CI, 0.938–0.961). We also used a bootstrap procedure with an AUC of 0.943 (95% CI, 0.929–0.9956). The optimism in apparent performance was 0.007, corresponding to an optimism-corrected area of 0.943. Assessment of calibration showed good calibration (Figure 1) with a high concordance between the predicted probabilities obtained from the logistic regression and the observed frequencies using the PRAm score (Pearson correlation coefficient = 0.995, calibration in the large = 0.001 [P = .974], calibration slope = 0.927). The Brier score in the validation cohort was 0.0037. The distribution of the PRAm score and the associated observed mortality in the validation cohort are reported in Figure 2.

Table 3.: PRAm Score to Predict Postoperative Mortality

Figure 1.: Calibration plot for predicted probability versus observed frequency of in-hospital mortality for the Pediatric Risk Assessment (PRAm) score in the validation cohort. Observed in-hospital mortality (blue points) with 95% confidence interval (CI; gray). Nonparametric calibration line (red) was created using the locally weighted scatterplot-smoothing algorithm. Pearson correlation coefficient = 0.995, calibration in the large = 0.001 (P = .974), calibration slope = 0.927, and Brier score = 0.0037.

Figure 2.: Distribution of the Pediatric Risk Assessment (PRAm) score values in the derivation cohort (red bars) in relation to the observed in-hospital mortality rate (smoothed line) for each score.

Figure 3.: Distribution of the Pediatric Risk Assessment (PRAm) scores after stratification for the American Society of Anesthesiologist (ASA) physical status classification.

We assessed the variability PRAm score in the validation cohort after stratification for ASA PS classification (Figure 3). The figure demonstrates the wide variability of PRAm scores in children assigned ASA PS classification ≥4 with PRAm scores ranging from 0 to ≥9.

DISCUSSION

In this study, we developed a simplified PRAm score to predict perioperative mortality in children undergoing noncardiac surgery. We rigorously validated the PRAm score in a large cohort and demonstrated it to have excellent accuracy in predicting perioperative mortality in children undergoing noncardiac surgery. In a secondary analysis, we demonstrated that for patients assigned an ASA PS classification ≥4, there is wide variability in the objectively obtained PRAm score. Because our PRAm score is based on objective preoperative characteristics, it likely has better reproducibility and decreased interrater variability than ASA PS classification.

Developed in 1941, the ASA PS classification was created to establish a scoring system for the evaluation of a patient’s general health and comorbidities immediately before an operative procedure.^1,20 This score was designed to identify surgical patients at risk for developing postoperative complications, taking into account the patient’s PS but neglecting the impact of surgery (type, complexity, and urgency). Nonetheless, it has been shown to have a positive correlation with length of stay and can potentially be used to determine postoperative costs.²¹ It has been established as a significant predictive factor for perioperative risk assessment, perioperative mortality, complication rates, and postoperative outcomes in multiple surgical specialties.^21–24 Although ASA PS classification was not initially intended to be used in children, it has been widely adopted and has been used to predict perioperative outcomes in children.^7,25

The simple classification system and ease of communication of an ASA PS classification make the score practical to use. However, the ASA PS classification of individual adult and pediatric patients has wide variability leading to inconsistencies in classification between raters.³ Levels of agreement between anesthesiologists assigning ASA PS classification classes range from 40% to 60%.²⁶ In other words, anesthesiologists are equally likely to disagree or agree on a particular ASA PS classification for a patient. Therefore, although the ASA PS classification can be used to predict mortality and outcomes, its variability calls for the development and validation of a more precise risk assessment tool.²⁷

Several risk assessment scores have been created and validated in adults.^28–30 In fact, a recent systematic review evaluated 34 risk stratification tools designed to predict morbidity and mortality in adult surgical patients.⁹ Most recently, Le Manach et al developed and validated a preoperative score to predict postoperative mortality. Derived from a very large data set collected over 1 year in France, it identified 17 predictors of mortality including age, medical comorbidities, and the type of surgical procedure.⁸ Using the ACS NSQIP data including preoperative risk factors and 30-day postoperative mortality for pediatric patients, we used a similar objective method for perioperative PRAm. The PRAm score is a comprehensive score derived by taking into consideration patient characteristics, comorbidities, and surgical type. It scores 5 variables derived from the 13 variables in the final multivariable logistic regression model (Table 2). Of particular interest is the fact that the need for an urgent surgical intervention proved to be a more robust predictor of risk than the actual surgical procedure in our model.

The limitations of this study include the use of a large multiinstitutional database that likely has missing data, miscoded diagnoses, or procedures. However, the ACS NSQIP is well designed and undergoes a thorough audit that makes it more accurate and informative than other administrative databases. One limitation to the use of the PRAm score compared with ASA PS classification is that 5 variables need to be taken into consideration. In addition, the anesthesiologist’s and surgeon’s type of training and years of practice may play a role in determining patients’ outcomes and are not included in the database.³¹ Intraoperative and postoperative adverse events that influence postoperative outcomes and mortality are not included in the PRAm score. The PRAm score was not designed to predict mortality in children undergoing cardiac surgery, traumatized children, or children undergoing solid organ transplant. Those are specific populations that need a specific risk assessment tool. The primary endpoint was in-hospital mortality, and we cannot guarantee that mortality was directly related to the surgical procedure and anesthesia, but could be explained by the underlying disease (eg, neoplasm), the treatment (eg, chemotherapy), or the PS at the time of the surgery. However, the objective of the study was to develop a PRAm score to predict perioperative mortality not limited to the intraoperative or immediate postoperative period.

In conclusion, we developed the PRAm score and validated that it has excellent accuracy to predict perioperative mortality in children undergoing noncardiac surgery. The wide variability in PRAm scores seen in patients assigned ASA PS classification ≥4 supports the conclusion that ASA PS classification (as originally designed) is not suitable as a predictive score for perioperative mortality in individual patients. The PRAm score could be used to enhance outcome-based quality and cost analysis by accurately assessing baseline risk factor differences between patients undergoing comparable surgical procedures. Furthermore, it is conceivable that more accurate estimation of preoperative risk will lead to improved patient outcomes because of recognition of the need for and use of specialized expertise. However, further prospective studies are now required to externally validate the PRAm score in a large cohort of children.

DISCLOSURES

Name: Viviane G. Nasr, MD.

Contribution: This author helped design the study and write the manuscript.

Name: James A. DiNardo, MD, FAAP.

Contribution: This author helped design the study and write the manuscript.

Name: David Faraoni, MD, PhD, FCCP.

Contribution: This author helped design the study, perform statistical analysis, and write the manuscript.

This manuscript was handled by: Richard C. Prielipp, MD.