Hospital outcomes can only be evaluated or benchmarked after adjusting for the patient case mix or severity of illness.1–4 Risk adjustment models tend to work best when they are applied to the same population on which they are built.
Pediatric risk of mortality (PRISM) III, pediatric index of mortality (PIM) 2, and pediatric logistic organ dysfunction (PELOD)5–7 are three models developed for use in the pediatric critical care population. Numerous studies have proven these scores to be valuable risk adjustment tools, and they are therefore applied to benchmark the performance of more than 100 pediatric intensive care units (PICU) participating in the Virtual PICU System.8
Because of their successes, admission PRISM-III, PIM-2, and PELOD are now collected by the Extracorporeal Life Support Organization (ELSO) emphasizing their scope and increasing their ease of application within this subgroup of PICU patients.9,10 Exceptional models often are applied to subpopulations with mixed results.11–16 We hypothesize that these admission scores will not provide good discrimination of mortality for the narrow subpopulation of PICU patients requiring extracorporeal membrane oxygenation (ECMO) support for respiratory failure.
Separately, researchers have developed ECMO-specific severity of illness scores using the ELSO Registry.17–22 However, existing ELSO-derived scores perform modestly with area under the receiver operating curves (AUCs) ranging from 0.63 to 0.77. These ELSO-derived scores lack laboratory and clinical measures of renal, neurologic, hepatic, and hematologic dysfunction that were not collected in the ELSO Registry but may improve discrimination.5–7,18,20,21 We hypothesize that the previously developed and published ELSO-derived score, Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support (Ped-RESCUERS), could be improved by additional pre-ECMO clinical measures not currently collected by ELSO.
This study is a retrospective cohort study performed at three international ECMO referral PICUs: University of Michigan, Children’s Healthcare of Atlanta at Egleston, and Toronto Sick Kids. Data were collected on children aged 29 days to 17 years who received ECMO support for respiratory failure during the years 2007–2016. Children were excluded if they were cannulated before arrival at a study center because this prevented pre-ECMO data collection. As in Ped-RESCUERS, children were also excluded if they had a prior ECMO run, if they went on ECMO for cardiac support, or if they went on ECMO to support cardiopulmonary resuscitation (ECPR). The respective institutional review board or research ethics board approved this study before data collection.
To ensure consistency of the data collection process, the first 10 cases collected at each institution were gathered under supervision of the principal investigator (R.P.B.). The data definitions manual is included in the Supplementary Material (see Supplementary material, Supplemental Digital Content, http://links.lww.com/ASAIO/A271). Variables collected included two types of variables: standard ELSO collected variables and new variables (see Table 1, Supplemental Digital Content, http://links.lww.com/ASAIO/A272). Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support was developed using only ELSO collected variables in the Registry. Unique variables collected for this study will be referred to as new variables. They include the admission PRISM-III, PIM-II, PELOD, the vasoactive infusion score (VIS),23 and the highest lactate within 6 hours before ECMO. Other new variables include the most abnormal creatinine, bilirubin, alanine aminotransferase (ALT), lactate and international normalized ratio (INR), platelet count, and white blood cell (WBC) count within 24 hours before ECMO cannulation.
Of note, ELSO data collection instructions do not specify the need to collect any variables at the same time, but to calculate an oxygenation index, the arterial partial pressure of oxygen (PaO2) with the fraction of inspired oxygen (FiO2), and corresponding mean airway pressure (MAP) must be measured at the same time point (time aligned). For this study, time-aligned variables include the PaO2, FiO2, and MAP. Additionally, the lowest systolic blood pressure, mean arterial pressure, and VIS are time-aligned variables.
Baseline creatinine measurements are not always available. After excluding children with preexisting renal injury, we assigned all patients a normal baseline creatinine as described by Zappitelli et al.24 We defined baseline creatinine as the median creatinine for the patient’s age provided the patient did not have known preexisting renal injury or failure.25 We then calculated patients’ change in creatinine from presumed baseline using the Kidney Disease Improving Global Outcomes consensus definition of acute kidney injury (AKI).26,27
We have previously described Ped-RESCUERS, a risk adjustment model for children requiring ECMO for respiratory failure that uses risk factors contained in the ELSO registry.18 However, the above-described new variables (see Table 1, Supplemental Digital Content, http://links.lww.com/ASAIO/A272) collected for this study were either not available in the ELSO registry or were heterogeneously reported and thus not statistically validated to be included in Ped-RESCUERS.
Both to account for the knowledge learned in Ped-RESCUERS and to consider new information, we fit a new Bayesian logistic regression model (model 2) for mortality, calculating new regression coefficients. This model considers the Ped-RESCUERS variables and the new variables differently through differential shrinkage. Shrinkage is a way of reducing or discounting a variable’s ability to the affect the model. The estimated associations with mortality for the Ped-RESCUERS variables were minimally shrunk, reflecting the established evidence that the Ped-RESCUERS variables have some association with mortality. We implemented minimal shrinkage via diffuse Cauchy priors.28 Conversely, we placed stronger, hierarchical shrinkage priors on the hypothesized set of risk factors unique to this study (11 new predictors listed in Table 1), which automatically tune the degree of shrinkage for each risk factor.29
Typical measures of statistical significance, e.g., p values, cannot be properly interpreted in our context, a consequence of the Bayesian shrinkage in these regressions. The adaptive nature of the shrinkage priors is designed to force small, or “nonsignificant,” associations close to zero. Instead, for each risk factor, we report the probability that each association falls entirely to the left or right of zero or, equivalently, that each odds ratio falls entirely to the left or right of one. A priori, these probabilities are 50% in either direction. In the presence of shrinkage, this probability will substantially increase in one direction (and decrease in the other) only for truly large associations.
In addition to differences in numbers of risk factors, the study cohort used to build model 2 also differed in terms of patients included from the cohort upon which Ped-RESCUERS was developed. Thus, we fit a third, intermediate model (model 1.1), which was limited to estimating associations in those risk factors from Ped-RESCUERS but reestimated on the current study cohort. Thus, model 1.1 is intended to quantify differences in estimation because of differences between cohorts.
Model calibration was assessed by calculating each model’s Hosmer–Lemeshow goodness of fit test and Brier score, defined as the average squared difference between each model-estimated risk of mortality (any number between 0 and 1) and the true outcome (either 0 [alive] or 1 [dead]). Thus, a perfect model’s Brier score is 0, and a useless model will have a Brier score equal to 0.25.30 Because of the proprietary nature of the PRISM-III predictions for risk, we were unable to assess its Brier score. We measured models’ mortality discrimination with the AUC. We used Bayesian cross-validation techniques to adjust for optimistic bias because of quantifying model performance on the same cohort upon which it was trained.31
Missing variables are reported in Supplemental Table 2 (see Table 2 (Supplemental Digital Content, http://links.lww.com/ASAIO/A273). When PaO2 was missing, and SpO2 was less than 97%, we estimated the PaO2 using the hemoglobin oxygen dissociation curve (15 cases).32 When a measured MAP was missing, we calculated the MAP using respiratory rate, inspiratory time, peak inspiratory pressure, and positive end-expiratory pressure (4 cases).33 In all other instances, missing variables were imputed as previously described.18
Two hundred thirty-six patients received ECMO to support respiratory. Fifty-eight children met exclusion criteria (Figure 1). The mortality rate of excluded patients was 50% (29/58). Among the remaining, 178 eligible patients were placed on ECMO for respiratory support with a median pre-ECMO oxygenation index of 46 and a mortality rate of 26% (47/178).
Overall, 69% of the eligible patients were transferred before receiving ECMO and 17% of those were transferred multiple times before arriving at an ECMO center (Table 2). Among transferred patients, the median time from presentation to arrival at an ECMO center was 1.8 days [interquartile range, 0.3–6.4 days]. After admission, 102/178 (57%) went on ECMO in less than 24 hours and 85/178 (48%) went on ECMO support in less than 12 hours.
The centers differed in their median annual ECMO PICU volume (before exclusions). The admission PRISM-III, PIM-2, and PELOD scores measured at admission differed between PICUs. Alternatively, Ped-RESCUERS, which is derived from measures collected 6 hours before ECMO cannulation, was not statistically different between PICUs. Centers differed in their pre-ECMO ventilation strategy and in whether they offered ECMO as a bridge to lung transplant. In terms of outcomes, centers differed in hospital length of stay, but not in their mortality rates (Table 2).
The AUCs of the admission PRISM-III, PIM-2, and PELOD to predict mortality fell between 0.53 and 0.57 (Table 3). The predicted average mortality using admission PIM-2 and admission PELOD was 19% and 42%, respectively, and the corresponding Brier scores were 0.24 and 0.33 (Table 4). Among patients who went on ECMO less than 24 hours after admission, the scores’ performance improved, with AUCs between 0.55 and 0.69 (Table 3). Among the three admission scores, PELOD had the best discrimination. The AUC for Ped-RESCUERS was 0.68 [0.59–0.77], higher than that of any of the other standard PICU severity scores. The average predicted mortality was 37%, and the corresponding Brier score was 0.19.
We then compared the distribution of new variables among survivors and nonsurvivors (Table 1). Simple, unadjusted comparisons showed children who died had a higher lactate, ALT, and bilirubin.
Figure 2 presents the adjusted odds ratios for Ped-RESCUERS and model 1.1. The estimated odds ratios in Ped-RESCUERS and model 1.1 were generally similar, as expected. One exception to this is diagnosis of asthma, as none of the 10 patients with asthma in this smaller cohort of 178 patients died, a statistical phenomenon known as “quasi-complete separation.”34
Model 2, which considers the inclusion of additional clinical and laboratory measures, yielded an AUC of 0.75 [0.66–0.82] (Table 3). Model 2 was significantly better than PRISM-III, PIM 2, and PELOD, but the AUC of model 2 was not significantly different than Ped-RESCUERS (Table 3). The average predicted mortality rate was 25%, with a Brier score of 0.17 (Table 4). Among the clinical measures considered in model 2, the largest odds ratios were ALT (6.25), lactate (2.86), and ratio of the arterial partial pressure of oxygen the fraction of inspired oxygen (PF; 0.58) (Figure 2). The probability that a higher ALT and lactate as well as a lower PF ratio are independently associated with mortality is 99.8%, 93.9%, and 86.4%, respectively. Four patients had liver failure and two of them died. When we restricted our analysis to the 174 patients without liver failure, we found similar results as reported in Table 1, and the univariate associations between ALT and mortality as well as bilirubin and mortality were preserved.
Admission PRISM-III, PIM-2, and PELOD were developed in complete PICU cohorts and have excellent discrimination in the general PICU population with AUCs of 0.94, 0.90, and 0.91, respectively. Pediatric risk of mortality III, PIM-2, and PELOD are collected in ELSO for risk adjustment, but they were not developed for the narrow subgroup of patients requiring ECMO for respiratory failure.9,10 Therefore, it is not surprising that we found they do not discriminate well between children who will live or die after ECMO respiratory support (AUC, 0.53–0.57).
Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support discriminated ECMO mortality in this sample of children requiring ECMO for respiratory failure with a modest AUC of 0.68. We found that ALT, lactate, and PF ratio were independently associated with mortality after accounting for the variables in Ped-RESCUERS. The AUC of this model is 0.77. We did not find a statistically significant difference between the AUC for Ped-RESCUERS and model 2.
The data collected for this research were collected by trained data abstractors, with the aid of a database definitions manual and for the intended purpose of evaluating the efficacy of risk adjustment tools for children receiving ECMO. Although our sample is modest in size with 178 pediatric acute respiratory failure patients, it is also informed by data from the thousands of patients in the ELSO Registry. We have incorporated the existing knowledge about the variables in Ped-RESCUERS by only minimally shrinking these respective associations and allowing for greater shrinkage of the associations for new variables. In ECMO and pediatrics, it will be common for large multisite registries, such as the ELSO Registry, to have fixed data inputs and for more detailed datasets from smaller groups of centers to have relatively small sample sizes. Our approach offers one method of leveraging the combined strength of large sample sizes in multi-institutional data and the more detailed data from a smaller subsample of patients. Consequently, we believe our approach will have the opportunity for future application as well.
We have not created a new clinically deployable pediatric risk adjustment score for children with respiratory failure requiring ECMO. Rather, we have identified new variables that are independently associated with mortality in children with respiratory failure requiring ECMO, after adjusting for the variables in Ped-RESCUERS. Nonetheless, there is compelling evidence that the clinical variables reported herein do explain previously uncaptured variation in mortality.
These data were collected retrospectively, and data elements were occasionally missing. Among Ped-RESCUERS variables, each variable was missing for <1% of cases except for MAP, which was missing in 7% of cases. Newly considered variables were missing in 1–22% of cases (see Table 2, Supplemental Digital Content, http://links.lww.com/ASAIO/A273). Unfortunately, two variables that we recommend for inclusion in the ELSO registry were missing frequently: lactate (22%) and ALT (13%). We recommend these variables because they are clinically meaningful measures of oxygen debt and liver injury, respectively. Furthermore, our imputation strategy appropriately adjusts for the uncertainty introduced by missingness. In other words, each reported odds ratio and corresponding interval incorporates the added variability because of missingness in that predictor.
Pediatric Risk Estimate Score for Children Using Extracorporeal Respiratory Support was developed in an ECMO population to facilitate benchmarking outcomes and research among the population of children receiving ECMO to support respiratory failure.18 The admission PRISM-III and PIM-2 were rigorously developed among PICU patient cohorts to benchmark PICU outcomes.5–7 Risk adjustment tools operate best when they are applied to a population that is similar to the population they were designed to risk adjust.11 This is because the key driver of these models’ discrimination of mortality is the composite risk factors and their associated odds ratios. A model’s odds ratios for given variables are derived from the population on which the model was built. The populations on which PRISM-III, PIM-2, and PELOD were developed are different than the ECMO population on which Ped-RESCUERS was developed and different than the sample in this research.5–7,18 For example, in this study, all children had severe respiratory failure when compared with the PELOD sample where 809/1806 (44%) had no respiratory failure.
Separately, PRISM-III and PIM-2 are intended to measure illness at hospital admission, but 43% of this study’s sample went on ECMO more than 24 hours after admission. In those instances, Ped-RESCUERS had the benefit of using more recent clinical data.35 ELSO is building off Ped-RESCUERS and all published ELSO-derived severity of illness tools to develop real-time risk-adjusted outcome reporting, through a partnership with a healthcare analytics company, ArborMetrix.10,17–22
The AUC of Ped-RESCUERS in this population was similar to the development dataset with an AUC of 0.68. This externally collected sample validates and replicates the previously published Ped-RESCUERS findings. Alanine aminotransferase, lactate, and PF ratio are associated with ECMO mortality after accounting for Ped-RESCUERS. Liver injury has been associated with ECMO mortality in previous studies.20,36 In this study, elevations in ALT may represent injury from hypoxic hepatitis. Similarly, lactate also represents accumulated local or global tissue hypoxemia37 and has been associated with ECMO mortality.38
In this study, pre-ECMO AKI was not independently associated with mortality. Previous studies found AKI was associated with ECMO mortality,20,27 and its absence from our study may indicate a limitation in power. Abnormal pupillary response,5,7 VIS,23 platelet count, INR, and WBC5,7 also did not appear to be independently associated with mortality. The absence of an association between pupillary light reflex response and mortality is surprising. It could reflect study power or suboptimal inter-rater reliability in pupillary light reflex assessment.39 Thrombocytopenia and prolonged INR were not associated with mortality. It is possible that these risks are mitigated by standing orders for blood product transfusions to meet goal criteria once the patient has been placed on ECMO.40
In Ped-RESCUERS, a higher pre-ECMO PCO2 was associated with a lower risk of mortality.18 In this study, after accounting for the association between a higher pre-ECMO lactate and mortality, the association between pre-ECMO CO2 and mortality was no longer significant (Figure 2). This suggests that the Ped-RESCUERS model was actually identifying patients who had an acidosis without an elevated CO2 as having a worse outcome. Once lactate was introduced into the model, the model identified that lactate was associated with higher mortality and a higher CO2 was no longer associated with lower mortality. Future models should reevaluate the role of CO2 after measuring lactate. Of note, HCO3 was not associated with mortality in Ped-RESCUERS, likely because it is also administered as a drug and, therefore, the measure of HCO3 is not associated with disease severity.
Pediatric risk of mortality III, PIM-2, and PELOD are not designed to discriminate ECMO mortality risk, and they should not be used to risk adjust ECMO outcomes for children with respiratory failure requiring ECMO. Risk adjustment scores, such as Ped-RESCUERS, which are developed in ECMO populations, perform better in discriminating mortality risk for children with respiratory failure that require ECMO, and we believe such scores should be used preferentially to risk adjust this ECMO population. Lactate, ALT, and PF ratio appear to be important variables for risk-adjusting ECMO outcomes. We recommend purposeful collection of these variables to create a larger dataset to be used in the further development of the most apt tools to risk adjust for this vulnerable population, thus improving evaluation of hospital outcomes, and ultimately discovering areas in which we can improve our care.
1. Patrick SW, Schumacher RE, Davis MM. Methods of mortality risk adjustment
in the NICU: A 20-year review. Pediatrics 2013.131(suppl 1): S68–S74.
2. Pollack MM, Ruttimann UE, Getson PR. Accurate prediction of the outcome of pediatric
intensive care. A new quantitative method. N Engl J Med 1987.316: 134–139.
3. Pollack MM, Holubkov R, Funai T, et al. The Pediatric
Risk of Mortality
score: Update 2015. Pediatr Crit Care Med 2016.17: 2–9.
4. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality
assessment for today’s critically ill patients. Crit Care Med 2006.34: 1297–310.
5. Leteurtre S, Martinot A, Duhamel A, et al. Validation of the paediatric logistic organ dysfunction (PELOD) score: Prospective, observational, multicentre study. Lancet 2003.362: 192–197.
6. Slater A, Shann F, Pearson G; Paediatric Index of Mortality
(PIM) Study Group: PIM2: A revised version of the Paediatric Index of Mortality
. Intensive Care Med 2003.29: 278–285.
7. Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric
Risk of Mortality
score. Crit Care Med 1996.24: 743–752.
8. Bennett TD, Spaeder MC, Matos RI, et al.; Pediatric
Acute Lung Injury and Sepsis Investigators (PALISI): Existing data analysis in pediatric
critical care research. Front Pediatr 2014.2: 79.
9. Thiagarajan RR, Barbaro RP, Rycus PT, et al. Extracorporeal Life Support Organization Registry International Report 2016. ASAIO J 2017.63: 60–67.
10. Barbaro RP, Paden ML, Guner YS, et al. Pediatric
Extracorporeal Life Support Organization Registry International Report 2016. ASAIO J 2017.63: 456–463.
11. D’Agostino RB Sr., Grundy S, Sullivan LM, Wilson P. Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. JAMA 2001.286: 180–187.
12. Berndtson AE, Sen S, Greenhalgh DG, Palmieri TL. Estimating severity of burn in children: Pediatric
Risk of Mortality
(PRISM) score versus Abbreviated Burn Severity Index (ABSI). Burns 2013.39: 1048–1053.
13. Carroll CL, Goodman DM, Superina RA, Whitington PF, Alonso EM. Timed Pediatric
Risk of Mortality
Scores predict outcomes in pediatric
liver transplant recipients. Pediatr Transplant 2003.7: 289–295.
14. Gérardin P, Rogier C, Leteurtre S, Jouvencel P, Ka AS, Imbert P. Evaluation of Pediatric
Risk of Mortality
(PRISM) scoring in African children with falciparum malaria. Pediatr Crit Care Med 2006.7: 45–47.
15. Schneider DT, Lemburg P, Sprock I, Heying R, Göbel U, Nürnberger W. Introduction of the oncological pediatric
risk of mortality
score (O-PRISM) for ICU support following stem cell transplantation in children. Bone Marrow Transplant 2000.25: 1079–1086.
16. Russell RA, Rettiganti M, Brundage N, Jeffries HE, Gupta P. Performance of Pediatric
Risk of Mortality
score among critically ill children with heart disease. World J Pediatr Congenit Heart Surg 2017.8: 427–434.
17. Barbaro RP, Bartlett RH, Chapman RL, et al. Development and validation of the neonatal risk estimate score for children using extracorporeal respiratory support. J Pediatr 2016.173: 56–61.e3.
18. Barbaro RP, Boonstra PS, Paden ML, et al. Development and validation of the pediatric
risk estimate score for children using extracorporeal respiratory support (Ped-RESCUERS). Intensive Care Med 2016.42: 879–888.
19. Maul TM, Kuch BA, Wearden PD. Development of risk indices for neonatal respiratory extracorporeal membrane oxygenation
. ASAIO J 2016.62: 584–590.
20. Bailly DK, Reeder RW, Zabrocki LA, et al. Development and validation of a score to predict mortality
in children undergoing extracorporeal membrane oxygenation
for respiratory failure: Pediatric
pulmonary rescue with extracorporeal membrane oxygenation
prediction score. Crit Care Med 2016.45: e58–e66.
21. Schmidt M, Bailey M, Sheldrake J, et al. Predicting survival after extracorporeal membrane oxygenation
for severe acute respiratory failure. The Respiratory Extracorporeal Membrane Oxygenation
Survival Prediction (RESP) score. Am J Respir Crit Care Med 2014.189: 1374–1382.
22. Schmidt M, Burrell A, Roberts L, et al. Predicting survival after ECMO for refractory cardiogenic shock: the survival after veno-arterial-ECMO (SAVE)-score. Eur Heart J 2015.36: 2246–2256.
23. Gaies MG, Gurney JG, Yen AH, et al. Vasoactive-inotropic score as a predictor of morbidity and mortality
in infants after cardiopulmonary bypass. Pediatr Crit Care Med 2010.11: 234–238.
24. Zappitelli M, Parikh CR, Akcan-Arikan A, Washburn KK, Moffett BS, Goldstein SL. Ascertainment and epidemiology of acute kidney injury varies with definition interpretation. Clin J Am Soc Nephrol 2008.3: 948–954.
25. Finney H, Newman DJ, Thakkar H, Fell JM, Price CP. Reference ranges for plasma cystatin C and creatinine measurements in premature infants, neonates, and older children. Arch Dis Child 2000.82: 71–75.
26. Kellum JA, Lameire N, Aspelin P, et al.; Kidney disease: Improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl 2012.2(1): 1–138.
27. Fleming GM, Sahay R, Zappitelli M, et al. The incidence of acute kidney injury and its effect on neonatal and pediatric extracorporeal membrane oxygenation
outcomes: A multicenter report from the kidney intervention during extracorporeal membrane oxygenation
study group. Pediatr Crit Care Med 2016.17: 1157–1169.
28. Gelman A, Jakulin A, Pittau MG, Su Y-S. A weakly informative default prior distribution. Ann of Appl Stat 2008.2: 1360–1383.
29. Carvalho CM, Polson NG, Scott JG. The horseshoe estimator for sparse signals. Biometrika 2010.97: 465–480.
30. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: A framework for traditional and novel measures. Epidemiology 2010.21: 128–138.
31. Vehtari A, Gelman A, Gabry J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput 2017.27: 1413–1432.
32. Severinghaus JW. Simple, accurate equations for human blood O2 dissociation computations. J Appl Physiol Respir Environ Exerc Physiol 1979.46: 599–602.
33. Marinelli PV. Mean airway pressure calculations. J Pediatr 1981.99: 168–169.
34. Albert A, Anderson JA. On the existence of maximum likelihood estimates in logistic regression models. Biometrika 1984.71: 1–10.
35. Iezzoni LI, Ash AS, Shwartz M, Landon BE, Mackiernan YD. Predicting in-hospital deaths from coronary artery bypass graft surgery. Do different severity measures give different predictions? Med Care 1998.36: 28–39.
36. Zabrocki LA, Brogan TV, Statler KD, Poss WB, Rollins MD, Bratton SL. Extracorporeal membrane oxygenation
respiratory failure: Survival and predictors of mortality
. Crit Care Med 2011.39: 364–370.
37. Kraut JA, Madias NE. Lactic acidosis. N Engl J Med 2014.371: 2309–2319.
38. Buijs EA, Houmes RJ, Rizopoulos D, et al. Arterial lactate for predicting mortality
in children requiring extracorporeal membrane oxygenation
. Minerva Anestesiol 2014.80: 1282–1293.
39. Olson DM, Stutzman S, Saju C, Wilson M, Zhao W, Aiyagari V. Interrater reliability of pupillary assessments. Neurocrit Care 2016.24: 251–257.
40. Dalton HJ, Reeder R, Garcia-Filion P, et al. Factors associated with bleeding and thrombosis in children receiving extracorporeal membrane oxygenation
(ECMO). Am J Respir Crit Care Med 2017.196: 762–771.