Observed mortality in the sample was 8.04% (531/6,602), whereas mortality predicted by PIM3 was 6.17% (407 deaths). SMR was 1.3 (95% CI, 1.2–1.42). The difference between the number of deaths observed and PIM3 predicted deaths was statistically significant (p < 0.001).
The AUC-ROC for the entire cohort was 0.83 (95% CI, 0.82–0.85), showing an adequate performance of the score for discriminating between nonsurvivors and survivors.
Table 3 shows the observed mortality and expected mortality in the different risk deciles, according to the goodness-of-fit test (Hosmer-Lemeshow). In all cases, the observed mortality was higher than PIM3 predicted mortality. The difference was statistically significant in the general population and for most deciles of mortality risk (χ2, 135.63; 8 degrees of freedom; p < 0.001). However, the difference between observed and expected mortality in the highest predicted mortality deciles (> 6.48%) was not statistically significant.
An analysis of discrimination and calibration of the model according to the volume of PICU number of admissions showed that for units with less than or equal to 200 admissions, the AUC-ROC was 0.84 (95% CI, 0.82–0.87) versus 0.82 (95% CI, 0.80–0.85) in units with more than 200 admissions. The SMR was 1.37 (95% CI, 1.21–1.55) and 1.25 (95% CI, 1.1–1.4), respectively.
Analysis by Age Group
The discrimination ability of PIM3 was adequate in all age groups. The observed mortality was higher than the mortality predicted by the score in all groups, especially in patients more than 120 months old. The difference between observed mortality and PIM3 predicted mortality was statistically significant (Table 4), except in children less than 1 year old.
Analysis by Diagnostic Group
PIM3 showed adequate discrimination ability in all diagnostic groups. The lowest discrimination ability was observed in patients admitted for respiratory disease, showing an AUC-ROC of 0.70, with a 95% CI lower limit of 0.66.
Regarding the score calibration, the observed mortality was higher than the predicted mortality in all diagnostic groups, except in patients admitted for injuries. The difference between the observed mortality and PIM3 predicted mortality was statistically significant in all diagnostic groups, except in the neurologic category, injury category, and, although borderline, in postoperative admissions (Table 5).
Analysis According to the Presence of CCC
PIM3 showed adequate discrimination ability in patients with some CCC when admitted to the PICU and in previously healthy patients. In this group, the observed mortality was higher than the PIM3 predicted mortality; the difference was statistically significant (276 observed deaths vs 179.3 predicted deaths). Although the observed mortality in previously healthy patients was higher than the expected mortality, this difference was not statistically significant (Table 6).
Mortality risk prediction models in PICUs are usually built in developed countries, based on a population with particular characteristics according to its case mix, available resources, and health system organization. Before being implemented by a country as tools for measuring intensive care quality and individual performance of PICUs, they must be validated in a locally representative sample of patients.
This study was carried out to assess the performance of the PIM3 score in a population of patients admitted to PICUs in Argentina, a medium-to-high income country (according to the World Bank classification), and with health system characteristics different from the countries where the score was developed (23). The obtained results indicate that the score has an adequate capacity for discriminating between survivors and nonsurvivors, in the general population and all the different age and diagnostic subgroups. However, the observed mortality exceeds the mortality predicted by the score.
In general terms, its performance is comparable to PIM2 according to the validation study carried out in Argentina in 2009 and Latin America in 2013, both in terms of discrimination and calibration capacity (10, 11). This study yielded a PIM3 AUC-ROC of 0.83 in the general population: 83% of nonsurvivors showed a higher PIM3 predicted death probability than survivors, compared with 88% of patients in the population in which the score was developed (14). Similarly, previous PIM2 validation studies carried out in Argentina and Latin America showed an adequate discrimination ability, with AUC-ROCs of 0.84 and 0.82, respectively (10, 11).
Instead of 407 predicted deaths, 531 deaths were observed in the sample. The observed mortality was higher than the expected mortality in all intervals of risk probability and in the majority of the analyzed subgroups (age, diagnostic, presence of CCC). Similarly, during the PIM2 validation study carried out in Argentina in 2009, 297 deaths were observed versus 246 predicted deaths. This tends to happen when mortality risk prediction scores are used in populations other than those they were developed in, especially when said populations have different characteristics, in terms of admission pathologies in PICUs, comorbidities, and health system (fragmentation, accessibility, available resources, and social-sanitary conditions).
Regarding the characteristics of the admitted population, we can mention that PIM3 predicted mortality in our population was 6.17%, higher than the mortality predicted in the regions in which the model was developed (4.1% in the United Kingdom/Ireland and 2.8% in Australasia). Only 18.8% of patients in our sample had an elective admission versus 41% in the original population. Likewise, 16% of patients in our population were admitted for postsurgery recovery versus 39.7% of patients in the original sample.
The profile of children admitted in PICU in our population, more severely ill and non elective admissions, may reflect differences in admission criteria in argentine PICUs, or less accessibility to these units. These differences might not be adequately expressed by the model, affecting its performance when used in our area. But, at the same time, the excess of deaths observed might be interpreted as differences in the quality of care provided in our PICUs compared with the units in which PIM3 was developed.
So far, few studies assessing the performance of the model in external populations have been published. In a retrospective study, Wofler et al (15) reported an adequate performance of the score in a sample of Italian PICUs. In this population, AUC-ROC was 0.88 and SMR was 0.98. There were no statistically significant differences between predicted and observed mortality. On the contrary, Lee et al (16) reported an AUC-ROC of 0.77 and SMR of 1.29 in a sample of 1,656 patients from one Korean PICU. The PIM3 score showed higher performance in Italy, possibly as it has a population and health system of characteristics similar to the United Kingdom and Australasia.
The analysis by age group indicated that teenagers showed the greatest difference between observed mortality and model-predicted mortality, reporting an SMR of 1.89. Similarly, Wofler et al (15) reported an SMR of 1.4 for this population in Italian PICUs (15). These groups will likely show characteristics affecting the score performance, such as different CCCs that were not considered as an adjustment variable by PIM3. This challenge in mortality risk prediction for teenage patients is also evidenced by other scores built with different statistical techniques such as the one introduced by Arzeno et al (2), who suggest the need to develop specific scores for this population.
In the analysis performed according to the diagnosis at admission, observed mortality was higher than mortality predicted by PIM3 in all groups, except in patients admitted for injury. This is similar to the results observed in the PIM2 validation carried out in Latin America, in which 22 Argentinian PICUs participated (11). This finding is possibly related to a higher score performance in patients without previous comorbidity because only 25 patients (4%) admitted for injuries had some CCC in our sample. Calibration capacity was inadequate, showing statistically significant differences between expected and observed mortality in all groups, except in patients admitted for neurologic problems and injuries. Paradoxically, calibration in the original sample failed in patients admitted for neurologic problems.
The analysis of the population according to the presence of CCC showed that PIM3 performance in patients with previous comorbidities was inadequate in terms of calibration. Although mortality in patients without CCC was higher than expected (6.99% vs 6.25%), the difference was not statistically significant and SMR was 1.12. In contrast, the SMR for patients with CCC was 1.54. Currently, no studies on the assessment of score performance in this particular group have been published. The PIM3 model considers leukemia, postinduction lymphoma, neurodegenerative diseases, or bone marrow transplant, as adjustment variables of high and very high risk of death, but excludes as risk factors conditions such as HIV or post-liver transplant admissions, which are still associated with higher mortality rates in our country. According to reports from the World Health Organization, the HIV-AIDS mortality rate reached 0.9 per 100,000 inhabitants in Australia versus 8.9 per 100,000 in Argentina (24) by 2012. Similarly, other oncologic or immune-hematologic pathologies, which are not considered in the score as risk factors, may be associated with a higher mortality rate in PICUs in our region. Decreased availability of palliative care and anticipated decisions regarding end-of-life care may influence access to ICUs for patients with low recovery capacity in Argentina. The above mentioned factors might partially explain a better performance of the model in children admitted without comorbidities.
Other conditions that might explain the excess of deaths observed in our population are the characteristics of the Argentinean health system, highly fragmented and not centralized, existing high number of PICUs that treat a small volume of patients (25). In our study, 77% of the units that participated had a low volume of admissions (200 or less). In these units, the SMR was higher than in the units with larger volume of admissions. It is possible that reductions in mortality could be achieved if critical patients were admitted to large PICUs, as proposed by other authors (26).
As a limitation of the study, we can mention that neonatal patients were not included even though the original PIM3 model has included this in their assessments. This is because neonatal (<28 d old) admissions are managed in neonatal ICUs in our country. Another limitation is that not all PICUs in the country were included, as no entity groups them in a mandatory manner. However, units in public and private hospitals, and in general and pediatric hospitals, are represented in the sample. Furthermore, PICUs from all five regions of the country considered by the Ministry of Health took part in the study although the central region provinces showed a clear predominance, which reflects the concentration of the Argentine population in that region (27).
In contrast, the study results could be generalized for PICUs that are members of the SATI-Q pediatric program, as 30 of the 33 units participating in the registry in 2015 were also included in this research (28). This program, sponsored by the SATI since 2005, has the voluntary participation of units located in different provinces of the country. It represents a source of free and publicly available data, which provides information on quality indicators in Argentinean PICUs for benchmarking purposes. A general report on predefined quality indicators, like the SMR resulting from the analysis of the total number of admissions in the participating units, is prepared and published annually.
Understanding the performance of PIM3 in a local representative sample allows us to use the score as a mortality prediction tool for the construction of SMR in each individual PICU and at a national level. The results of this study show that using PIM3 to predict mortality, the actual SMR for PICUs participating in the SATI-Q program is 1.3, instead of the values observed in recent years using PIM2 (21). The performance of each participating PICU can be assessed by comparing their obtained SMRs with this value. This analysis conducted on an annual basis, can detect changes in population characteristics and the score performance, and will allow the comparison of each PICU against a local SMR and the comparison of SMR over time, as it has been performed with the use of PIM2 to date.
We suggest that it would be optimal to switch from PIM2 to PIM3 as the score to predict mortality in Argentine PICUs given that using a nonupdated model, like PIM2, might result in the misconception that care in our units is better than it actually is. On the other hand, using an up-to-date tool to compare care provided in local PICUs with international care for benchmarking purposes is necessary to highlight characteristics in the local care model that could have an impact on our patients’ outcomes.
An objective measurement of the results is necessary to evaluate the impact of the measures aimed at improving the care of the critical pediatric patient, either by improving the detection, the quality of the initial care, the accessibility to the PICUs, or the human and technological resources available in them.
This study assessed the performance of the PIM3 score in a large sample of patients admitted to PICUs in Argentina. The score showed an adequate ability to discriminate between the population of patients who survive and those who die. Instead, observed mortality was higher than predicted mortality in the general population and the population stratified by age, diagnosis or presence of CCC. The use of an updated instrument such as PIM3 will allow an actual comparison between pediatric intensive care provided in the country and care provided internationally. This might also allow future planning of pediatric intensive care services in Argentina.
We express our most sincere gratitude to all PICU members who participated in the study and for their commitment to the project. We also thank the Argentine Society of Intensive Care who provided the SATI-Q software as a data collection tool for this study and has continuously supported research projects proposed by its members.
1. Breslow MJ, Badawi O. Severity scoring in the critically ill: Part 1–interpretation and accuracy of outcome prediction scoring systems. Chest 2012; 141:245252
2. Arzeno N, Lawson K, Duzinski S, et al. Designing optimal mortality
risk prediction scores that preserve clinical knowledge. J Biomed Inform 2015; 56:145156
3. Aczon M, Ledbetter D, Ho L, et al. Dynamic Mortality
Risk Predictions in Pediatric Critical Care Using Recurrent Neural Networks. Available at: https://arxiv.org/pdf/1701.06675v1.pdf
. Accessed October 20, 2017
4. Power GS, Harrison DA. Why try to predict ICU outcomes? Curr Opin Crit Care 2014; 20:544549
5. Timmers TK, Verhofstad MH, Moons KG, et al. Intensive care performance: How should we monitor performance in the future? World J Crit Care Med 2014; 3:7479
6. Duke GJ, Pilcher DV, Shann F, et al. ANZROD, COPE 4 and PIM 3: Caveat emptor. Crit Care Resusc 2014; 16:155157
7. Capitulo de Terapia Intensiva Pediátrica, Comité Nacional de Emergencias y Cuidados Críticos: Normas de Categorización, Organización y Funcionamiento de las Unidades de Cuidados Intensivos e Intermedios Pediátricos en los Establecimientos Asistenciales. Medicina Intensiva 2014; 31:219
9. Comité de Gestión, Calidad y Escores: Sociedad Argentina de Terapia Intensiva. Programa SATI-Q: Una experiencia local en Quality Benchmarking
. Revista Argentina de Terapia Intensiva. 2016. Available at: http://revista.sati.org.ar/index.php/MI/article/view/485
. Accessed October 20, 2017
10. Fernandez A, Arias López M, Ratto M, et al. Validación del índice pediátrico de mortalidad 2 (PIM2) en Argentina: Un estudio prospectivo, multicéntrico, observacional. Arch Arg Pediatr 2015; 113:221228
11. Arias Lopez MP, Fernández AL, Ratto ME, et al.; ValidarPIM2 Latin American Group: Pediatric Index of Mortality
2 as a predictor of death risk in children admitted to pediatric intensive care units
in Latin America: A prospective, multicenter study. J Crit Care 2015; 30:13241330
12. Alexander J, Tregea S, Slater A. Report of the Australian and New Zealand Paediatric Intensive Care Registry. 2010. 2012Melbourne, Australia, Australian and New Zealand Intensive Care Society.
13. Universities of Leeds and Leicester: Paediatric Intensive Care Audit Network National Report 2009–2011. 2012Leeds, Leicester, Universities of Leeds and Leicester.
14. Straney L, Clements A, Parslow RC, et al.; ANZICS Paediatric Study Group and the Paediatric Intensive Care Audit Network: Paediatric Index of Mortality
3: An updated model for predicting mortality
in pediatric intensive care. Pediatr Crit Care Med 2013; 14:673681
15. Wolfler A, Osello R, Gualino J, et al.; Pediatric Intensive Therapy Network (TIPNet) Study Group: The importance of mortality
risk assessment: Validation of the Pediatric Index of Mortality
3 score. Pediatr Crit Care Med 2016; 17:251256
16. Lee OJ, Jung M, Kim M, et al. Validation of the Pediatric Index of Mortality
3 in a single pediatric intensive care unit in Korea. J Korean Med Sci 2017; 32:365370
17. Sari D, Saputra I, Triratna S, et al. The Pediatric Index of Mortality
3 score to predict mortality
in a pediatric intensive care unit in Palembang, South Sumatera, Indonesia. Paediatr Indones 2017; 57:164170
18. Feudtner C, Christakis DA, Connell FA. Pediatric deaths attributable to complex chronic conditions: A population-based study of Washington State, 1980-1997. Pediatrics 2000; 106:205209
19. Slatter A, Shann F, McEniery J, et al. The ANZPIC Registry diagnostic codes: A system for coding reasons for admitting children to intensive care. Intensive Care Med 2003; 29:271277
20. Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49:13731379
26. Pearson G, Shann F, Barry P, et al. Should paediatric intensive care be centralised? Trent versus Victoria. Lancet 1997; 349:12131217
APPENDIX 1. MEMBERS OF VALIDARPIM3 ARGENTINE GROUP
Luis Aramayo (Hospital Zonal Ramón Carrillo, Rio Negro); Pedro Portero (Hospital Interzonal General de Agudos “Dr. Abraham Piñeyro,” Buenos Aires); Priscilla Botta (Hospital Del Niño Jesús, Tucumán); Marta Mosciaro (Hospital Dr. Humberto Notti, Mendoza); Segundo Español (Hospital pediátrico Juan Pablo II, Corrientes); Walter Lorenz (Hospital Zonal General de Agudos Dr. Lucio Melendez, Buenos Aires); Alberto Hernández (Hospital de Pediatría” Prof. Dr. Juan P. Garrahan” Unidad 72 Ciudad Autónoma de Buenos Aires); Rosana Poterala (Sanatorio Anchorena, Ciudad Autónoma de Buenos Aires); Gustavo Gonzalez (Complejo Medico Policial “Churruca-Visca,” Ciudad Autónoma de Buenos Aires); Ramon Pogonza (Hospital De Niños De La Santísima Trinidad, Córdoba); Facundo Jorro (Sanatorio De La Trinidad Mitre, Ciudad Autónoma de Buenos Aires); Carolina Sabatini (Hospital General de Niños Pedro de Elizalde, Ciudad Autónoma de Buenos Aires); Marta De Barelli (Hospital Provincial, Rosario, Hospital Español, Rosario); Karina Cinquegranni (Hospital El Cruce Dr. Néstor Carlos Kirchner, Alta Complejidad en Red, Buenos Aires); Sergio Suarez (Clinica del Niño, Quilmes, Buenos Aires); Javier Ponce (Hospital Guillermo Rawson, San Juan); Sandra Chuchuy (Hospital Publico Materno Infantil de Salta, Salta); Gustavo Sciolla (Hospital de Niños Zona Norte, Santa Fe); Maria Eugenia Passini (Hospital de San Luis, San Luis); Rose Marie Deheza (Clínica Modelo, Morón, Buenos Aires); Maria Mackern (Hospital Dr. H Notti Cardiovascular, Mendoza); Juan Fabris (Hospital Penna, Bahia Blanca, Buenos Aires); Ana Rodriguez Calvo (Hospital Isola Puerto Madryn, Chubut); Claudia Benaroya (Hospital Regional de Rio Gallegos, Santa Cruz); Maria A. Boretto (Sanatorio de Niños, Rosario, Santa Fe); German Kaltenbach (Hospital Regional Castro Rendón, Neuquén); Carlos Rodriguez (Hospital Zonal de Caleta Olivia, Santa Cruz); Marisol Ramos (Hospital Avelino Castelán, Chaco); Silvia Lanatti (Hospital de Niños VJ Vilela, Santa Fe); Paula Medici (Hospital Interzonal Especializado Materno Infantil de Mar del Plata, Buenos Aires); Claudia Pedraza (Hospital de Niños Sor María Ludovica, Unidad Cardiovascular, La Plata, Buenos Aires); Juan Varón Redondo (Hospital de Clínicas José de San Martin, Ciudad Autónoma de Buenos Aires); Marcelo Itharte (Hospital Materno Infantil San Roque, Entre Ríos); Gabriel Boggio (Clinica Velez Sarfield, Cordoba); Sebastián De Giuseppe (UCIP Sagrado Corazón, Ciudad Autónoma de Buenos Aires); Marlene Velazquez (Hospital Pediátrico del Niño Jesús, Córdoba); Yanina Fortini (Hospital Municipal de Trauma y Emergencias Dr. Federico Abete, Buenos Aires); Alejandra Ribonetto (Hospital de Niños Dr. Héctor Quintana, Jujuy); Gaston Morales (Corporación Medica de General San Martin, Buenos Aires); Jorge Cavagna (Hospital de Quemados, Ciudad Autónoma de Buenos Aires); Matias Penazzi (Hospital de Niños de San Justo, Buenos Aires); Daniel Capra (Sanatorio Trinidad Ramos Mejía, Buenos Aires); and Ariel Albano (Hospital de Niños Dr O Allassia, Santa Fe).
benchmarking; healthcare; mortality; pediatric intensive care units; quality indicators; risk adjustment
Supplemental Digital Content
Copyright © 2018 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies.