Model predictive discrimination was assessed on the validation dataset (n=1,462). For the pre-ECMO model, C statistic was 0.65 (95% confidence interval [CI], 0.62–0.68). Applying the development data set used in this study, the Neo-RESCUERS equation discrimination was lower (C = 0.59, 95% CI, 0.56–0.62). Note that, the 95% CIs only overlap at the end point of 0.62 (upper limit for Neo-RESCUERS and lower limit for our pre-ECMO score). Thus, there is substantial improvement relative to the Neo-RESCUERS score. The results suggest that the pre-ECMO score of our study discriminates better as it specifically focuses on the CDH population. Revalidation of the PIPER equation in our CDH-specific training data set resulted in C statistic (C = 0.60; 95% CI, 0.57–0.63). Similarly, compared to the PIPER score there is little overlap in the CIs (upper limit of 0.63 for PIPER and lower limit of 0.62 our pre-ECMO score).
For the on-ECMO model, improved performance to discriminate mortality was observed, given a higher C statistic of 0.73 (95% CI, 0.71–0.76). Based on the final variables selected by the model, complications during the ECMO procedure as well as some ECMO-related variables played a significant role in predicting mortality, resulting in a higher C-statistic score compared with pre-ECMO model, as expected. When revalidated using the same development data set for this study, the C statistic for the Haricharan model was 0.67 (95% CI, 0.68–0.71). Again, note that CI overlap is only at the end point (0.71 is upper limit for Haricharan, and it is the lower upper limit for our on-ECMO model), thus demonstrating better discrimination with our model. Similarly, when PIPER+ was revalidated in our development data set and had decreased discrimination accuracy (C = 0.70, 95% CI, 0.67–0.73). There was only a slight overlap in the CIs (upper limit of 0.73 for PIPER+ and lower limit of 0.71 for our on-ECMO score).
A Hosmer–Lemeshow test was used to test the calibration: the χ2 goodness-of-fit statistic was 5.85 (p = 0.67) for the pre-ECMO model and 6.26 (p = 0.62) for the on-ECMO model, indicating that both prediction models fit (p < 0.05). The shrinkage factor γ based on 2000 bootstraps is 0.89 (95% CI, 0.79–1.00) in the pre-ECMO and 0.90 (95% CI, 0.83–0.99) in the on-ECMO model, which was used to adjust the final prediction models. Figure 1 shows the predicted mortality as a function of (A) pre-ECMO and (B) on-ECMO RSs (smooth curve) along with the actual observed mortality rate by decile of the RS in the development and validation data sets. The close agreement between observed and predicted mortality in Figure 1 provide additional validation of the goodness-of-fit of the prediction models.
To assess the robustness of these models to missing data, we refitted the models using only complete data, as well as multiple imputation using 10 imputed data sets. The estimates of coefficients were quite similar for the models in both sensitivity analyses (results not shown). For the pre-ECMO model, the C statistic was 0.65 (95% CI, 0.62–0.68) on complete data analysis and 0.64 (95% CI, 0.61–0.68) on multiple imputation analysis. For the on-ECMO model, C statistics were both 0.73 (95% CI, 0.70–0.76), which matched the main results presented above based on mean imputation.
Exploration of Clinical RGs and Patient Features Within RGs
We examined predicted mortality in five clinical RGs, defined a priori based on percentiles of the RS, as (1) lowest 5%, (2) 5%–25%, (3) 25%–75%, (4) 75%–95%, and (5) highest 5% of the RS for both pre- and on-ECMO models. In pre- and on-ECMO data sets, RSs detected 2–4 fold differences in mortality. For the pre-ECMO model, groups 1–5 corresponded to RS ≤ −0.9, (0.9, −0.3), (−0.3, 0.5), (0.5, 1.2) and RS > 1.2, respectively (Figure 2A). The observed mortality rates in validation data set for groups 1–5 were 38%, 35%, 51%, 66%, and 75%, respectively (Figure 2A); thus, mortality for neonates with RS in the 5th to 25th percentile appeared to be the same as those in the lowest 5% of the RS, while mortality increased for those with RS greater than the 25th percentile. This suggested combining groups 1 and 2 into a single lower RG. Similarly, we defined the RGs for on-ECMO model based on the same percentile groups as the pre-ECMO model above; here the five groups corresponded to on-ECMO RS ≤ −1.4, (1.4, −0.6), (-−.6, 0.8), (0.8, 2.0), and > 2.0 (Figure 2B). The observed mortality rates in the validation set corresponding to the five RGs were 26%, 24%, 53%, 74%, and 86%, respectively (Figure 2B).
Finally, we illustrate how the models predict pre-ECMO and on-ECMO mortality for several “new” (potential) neonates. Table 4 shows the predicted probability of death for 3 distinct neonates (patients 1A–1C) pre-ECMO and on-ECMO (patients 2A–2C) with the RGs depicted. Overall, these demonstrate how the models estimate mortality based on each patient characteristics within the ELSO Registry data elements.
The primary objective of our study was to develop and validate mortality risk prediction models specifically for the CDH-ECMO population. We have noted that prior1,9 pre-ECMO risk models can overestimate mortality if the presence or absence of on ECMO complications are not considered. And, we wanted to be able to compare initial mortality risk to risk during ECMO to allow for assessment of quality of ECMO care provided. This was the reason for choosing to develop two independent models to estimate mortality risk for the CDH-ECMO population. Our models were divided into distinct clinical time points where this information could be most useful: pre- and on-ECMO. We believe that the risk models presented in our study use clinically relevant predictor variables and enable clinicians to ask questions such as: “What is the mortality risk of a low BW infant with a right-sided diaphragmatic defect if were to be treated with ECMO?” and “How does the mortality risk change after 2 weeks of ECMO with severe intraventricular hemorrhage and/or other complications?”. The most suitable application of these models is to properly risk-stratify infants, retrospectively, accounting for all available clinical data for research and quality improvement.
Parallels exist between the pre-ECMO model developed in this study and previous risk models developed for the general CDH population, which combined ECMO and non-ECMO data. The CDH Study Group (CDHSG) score was based on 5 min Apgar and BW3. The Wilford Hall/Santa Rosa prediction equation (WHSR = highest PaO2 − highest PCO2) was developed next.4 Hoffman et al.5 later showed that neither of these scores were adequately discriminatory when specifically revalidated within the ECMO population. More recently, Brindle et al.6 developed a simple CDH scoring equation based on low BW (<1.5 kg), Apgar scores, severe pulmonary hypertension, critical congenital heart disease, and chromosomal anomalies. Unfortunately, the Brindle score is not applicable to the ECMO population as BW < 1.5 kg is not feasible for ECMO. Kays et al.7 also reported a CDH mortality prediction model, derived from a single institution experience (n = 172), based on CDHSG score, 1 min Apgar, and first pH. Revalidation of the Kays equation with our data set is not possible as first pH is not coded as a variable within ELSO registry data. We revalidated and compared the Neo-RESCUERs and PIPER equations in our data set; based on C statistic, our pre-ECMO risk model provided improved prediction.
We next compared the on-ECMO model to previously developed mortality risk models. The first study for comparison is by Seetharamaiah et al.,14 who determined from CDHSG data (1995–2005) predictors associated with survival in the CDH-ECMO population that underwent CDH repair. Seetharamaiah et al.14 identified GA, BW, prenatal diagnosis, length of ECMO, and patch repair as survival indicators. We cannot comparatively revalidate the Seetharamaiah predictors with ELSO data, as the ELSO Registry does not record whether repair with patch was used. Our on-ECMO score can be directly compared with the Haricharan’s equation. When revalidated using the same development data set for this study, the C statistic for the Haricharan model and PIPER+ had lower discrimination accuracy, thus, demonstrating better discrimination with our model. This improved discrimination can be attributed to expanded data points and model selection methods used in this study.20
We made several observations after examination of the RGs for the pre- and on-ECMO models. For both models, analysis of RG distributions in the two lowest RGs (1 and 2) does not differ significantly with similar neonatal characteristics. Also, the pattern of increasing mortality as a function of increasing RGs is similar for both models. Several subtle differences exist between the two models in the distribution of RGs. First, for the pre-ECMO model, mortality estimate is greater by about 10% for groups 1 and 2 (low risk) compared with the same RGs of the on-ECMO model. Second, the two highest RGs of the on-ECMO model have observed mortality about 10% higher than the corresponding RGs for the pre-ECMO model. This improved discrimination of mortality between lower and higher RGs is attributed to additional information (predictor variables) for the on-ECMO model. It is also critical to point out that the pre-ECMO model demonstrated here and by previous studies can overestimate risk in absence of length of ECMO and on-ECMO complications. This point becomes important as CDH patients represent the largest group of neonatal respiratory failure patients experiencing prolonged ECMO courses.23 Therefore, the pre-ECMO model provides an average risk of mortality assuming some patients will develop certain complications and have prolonged ECMO runs. This can be helpful as the interplay between the RSs provide a means to address, pinpoint, and improve ECMO care. The on-ECMO model, therefore, is a better prediction tool to estimate mortality risk, assuming those clinical parameters are known.
Clinicians should be very cautious in the application of this or other RSs at the bedside. We specifically discourage clinicians from withholding ECMO for neonates based on high RSs, as survival in the highest RG is 35% and the RS should never come before clinical acumen. Including on-ECMO data may help teams and families understand why support is continuing or occasionally with explaining why discontinuation of support is being considered. Although ideally clinical risk indexes can be used at the bedside, the RSs developed in this study, as well as all other ECMO mortality RSs mentioned above, are best suited for analyzing groups of patients as opposed to the individual neonate. The ECMO risk equations can be used similar to the pediatric American College of Surgeons National Surgical Quality Improvement Program risk equation to provide risk-stratified outcome information to institutions on a periodic basis on CDH infants requiring ECMO.24 Furthermore, the scores can be used to analyze patients for quality improvement purposes within the same organization. Future iterations of the risk equation may include local institutional adjustments, as predicted outcomes may be different, for instance, at ECMO centers of excellence or high volume centers, which can only be identified with proper risk adjustment methods, and we believe that the mortality risk equations developed in this study provide the statistically most accurate means to provide such information for the CDH population. Finally, the risk equations can be used for multiple research questions and comparative analyses.
Although our findings add to existing data on CDH-ECMO risk prediction, limitations exist. Similar to most retrospective studies, our study may include potential coding errors and/or missing data. Precise indications for employing ECMO are not standardized across institutions, neither are ECMO care protocols. There are variations in treatment of CDH before ECMO and during ECMO and timing of diaphragm repair across institutions. The clinical variability introduces unmeasurable heterogeneity and randomness, which may affect outcomes. Another limitation was the inability to know the contribution of ECMO to mortality, as the ELSO Registry only includes data for ECMO patients. Therefore, the pre-ECMO risk model should only be calculated in infants who will be treated with ECMO or where ECMO is strongly considered. As is inherent in many databases, the general issue of selection bias is a major limitation, and for the ELSO Registry, there is a selection bias in that it contains patients for whom ECMO has been selected as therapy. Thus, ELSO data reflect the outcomes of patients with CDH for whom ECMO was chosen. Therefore, our prediction model is not a general prediction model of outcome for all CDH patients to be used to decide whether to select ECMO as a therapy or not. Finally, we note that potential candidate predictor variables are limited by what is available in ELSO.
The models developed in this study account for whether or not prenatal diagnosis was established. Important potential information on prenatal prognosticators including lung–head ratio, MRI lung volumes and liver up or down were not available as data elements in the ELSO Registry. Had they been available, these could have potentially improved prediction performance, only in those patients who are prenatally diagnosed. Given, however, prenatal measurements such as lung–head ratio or MRI lung volumes are highly variable on GA, as well as center, and standardization is lacking such that these could be reported to a central registry with accuracy, that is, different centers measure slightly different versions of these anatomic indexes at different gestation ages and not all reports were observed to expected values.25 Furthermore, there are more centers who provide ECMO than centers who have established fetal centers. Future studies could be aimed at standardizing fetal prognostication and comparatively validating prenatal risk assessment to postnatal risk assessment methods.
In conclusion, we have developed risk models for CDH that allow mortality risk prediction just before and during ECMO using data reported to the ELSO Registry. The equations developed in this study improve upon previous efforts to define risk in the CDH-ECMO population with increased statistical accuracy. At present, our scores can serve as excellent research tools and for benchmarking outcomes amongst different centers. The ability to assess outcome risk systematically and objectively may allow for a greater patient-centered decision making process and improve the care of these high RGs of neonates. Online calculators for both pre- and on-ECMO models are freely accessible at www.choc.org/ecmocalc, where the predicted mortality, confidence interval, and RG can be calculated rapidly and efficiently.
1. Barbaro RP, Bartlett RH, Chapman RL, et al. Development and validation of the neonatal risk estimate score for children using extracorporeal respiratory support. J Pediatr 2016.173: 56–61.e3.
2. Guner YS, Khemani RG, Qureshi FG, et al. Outcome analysis of neonates with congenital diaphragmatic hernia treated with venovenous vs venoarterial extracorporeal membrane oxygenation. J Pediatr Surg 2009.44: 1691–1701.
3. Congenital Diaphragmatic Hernia Study Group: Estimating disease severity of congenital diaphragmatic hernia in the first 5 minutes of life. J Pediatr Surg 2001.36 (1): 141–5.
4. Schultz CM, DiGeronimo RJ, Yoder BA; Congenital Diaphragmatic Hernia Study Group: Congenital diaphragmatic hernia: a simplified postnatal predictor of outcome. J Pediatr Surg 2007.42: 510–516.
5. Hoffman SB, Massaro AN, Gingalewski C, Short BL. Survival in congenital diaphragmatic hernia: use of predictive equations in the ECMO
population. Neonatology 2011.99: 258–265.
6. Brindle ME, Cook EF, Tibboel D, Lally PA, Lally KP; Congenital Diaphragmatic Hernia Study Group: A clinical prediction rule for the severity of congenital diaphragmatic hernias in newborns. Pediatrics 2014.134: e413–e419.
7. Kays DW, Islam S, Perkins JM, Larson SD, Taylor JA, Talbert JL. Outcomes in the physiologically most severe congenital diaphragmatic hernia (CDH
) patients: Whom should we treat? J Pediatr Surg 2015.50: 893–897.
8. Haricharan RN, Barnhart DC, Cheng H, Delzell E. Identifying neonates at a very high risk for mortality among children with congenital diaphragmatic hernia managed with extracorporeal membrane oxygenation. J Pediatr Surg 2009.44 (1): 87–93.
9. Maul TM, Kuch BA, Wearden PD. Development of risk indices for neonatal respiratory extracorporeal membrane oxygenation. ASAIO J 2016.62: 584–590.
10. Bartlett RH, Gazzaniga AB, Toomasian J, et al. Extracorporeal membrane oxygenation (ECMO
) in neonatal respiratory failure. 100 cases. Ann Surg 1986.204: 236–245.
11. Bohn DJ, James I, Filler RM, et al. The relationship between PaCO2 and ventilation parameters in predicting survival in congenital diaphragmatic hernia. J Pediatr Surg 1984.19: 666–671.
12. Krummel TM, Greenfield LJ, Kirkpatrick BV, et al. Alveolar-arterial oxygen gradients versus the neonatal pulmonary insufficiency index for prediction of mortality in ECMO
candidates. J Pediatr Surg 1984.19 (4): 380–384.
13. Tiruvoipati R, Vinogradova Y, Faulkner G, Sosnowski AW, Firmin RK, Peek GJ. Predictors of outcome in patients with congenital diaphragmatic hernia requiring extracorporeal membrane oxygenation. J Pediatr Surg 2007.42: 1345–1350.
14. Seetharamaiah R, Younger JG, Bartlett RH, Hirschl RB; Congenital Diaphragmatic Hernia Study G: Factors associated with survival in infants with congenital diaphragmatic hernia requiring extracorporeal membrane oxygenation: a report from the Congenital Diaphragmatic Hernia Study Group. J Pediatr Surg2009.44 (7): 1315–21.
15. Murthy K, Pallotto EK, Gien J, et al. Predicting death or extended length of stay in infants with congenital diaphragmatic hernia. J Perinatol 2016.36: 654–659.
16. Grizelj R, Bojanić K, Pritišanac E, et al. Survival prediction of high-risk outborn neonates with congenital diaphragmatic hernia from capillary blood gases. BMC Pediatr 2016.16: 114.
17. Murphy D, Pak Y, Cleary JP. Pulse oximetry overestimates oxyhemoglobin in neonates with critical congenital heart disease. Neonatology 2016.109: 213–218.
19. Harrell F. Regression Modeling Strategies with Applications to Linear Models, Logistic Regression, and Survival Analysis. 2001.New York: Springer-Verlag, New York, Inc.
20. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996.15: 361–387.
21. Steyerberg EW, Eijkemans MJ, Harrell FE Jr, Habbema JD. Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 2000.19: 1059–1079.
22. Steyerberg EW, Eijkemans MJC, Habbema JDF. Application of shrinkage techniques in logistic regression analysis: a case study. Statistica Neerlandica 2001.55(1): 76–88.
23. Prodhan P, Stroud M, El-Hassan N, et al. Prolonged extracorporeal membrane oxygenator support among neonates with acute respiratory failure: a review of the Extracorporeal Life Support Organization registry. ASAIO J 2014.60: 63–69.
24. Kraemer K, Cohen ME, Liu Y, et al. Development and evaluation of the American College of Surgeons NSQIP Pediatric Surgical Risk Calculator. J Am Coll Surg 2016.223: 685–693.
25. Oluyomi-Obi T, Kuret V, Puligandla P, et al. Antenatal predictors of outcome in prenatally diagnosed congenital diaphragmatic hernia (CDH
). J Pediatr Surg 2017.52: 881–888.
Keywords:Copyright © 2018 by the American Society for Artificial Internal Organs
ECMO; CDH; mortality risk; risk score