Journal Logo

Multi-institutional Collaborative and QI Network Research

Set of Quality Indicators of Pediatric Intensive Care in Spain: Delphi Method Selection

Hernández-Borges, Angel A. MD, PhD*; Pérez-Estévez, Elena MD; Jiménez-Sosa, Alejandro PhD*; Concha-Torre, Andrés MD; Ordóñez-Sáez, Olga MD§; Sánchez-Galindo, Amelia C.; Murga-Herrera, Vega MD; Balaguer-Gargallo, Mónica MD**; Nieto-Moro, Montserrat MD††; Pujol-Jover, Montserrat MD‡‡; Aleo-Luján, Esther MD§§; On behalf of the Working Group on Safety of the Spanish Society of Pediatric Intensive Care

Author Information
doi: 10.1097/pq9.0000000000000009
  • Open


The quality of healthcare management requires monitoring of (1) key results of adequate assistance (clinical, professional, financial, and user satisfaction) and (2) adverse events arising from patient care. Monitoring usually involves the use of so-called indicators of quality.1

By means of measurement, publication, and recognition, such metrics can help reduce the variability of healthcare quality.2 Professionals and managers of pediatric critical care units can use quality indicators and their standard values as a guide to set their units’ objectives and design their improvement projects. The so-called composite indicators (combining several individual indexes) summarize in a single measurement several critical aspects of the process and outcome of care, considered benchmarks of the performance, and quality of healthcare units and institutions. Health organizations also use composite indicators as a tool for external benchmarking and internal tracking of improvements.3

Several medical societies have established sets of indicators and standards of quality in their field.4 Some intensive care medicine organizations have performed exhaustive reviews of possible indicators, leaving the final decision about their use as tools for improvement to the respective health unit.5 However, the most recent initiatives focus on a few indicators with a high degree of expert agreement.1,6

In this work, we show the results of the project “QuaCIP” (Quality, Cuidados Intensivos Pediátricos) of the Working Group on Safety of the Spanish Society of Pediatric Intensive Care (SECIP). Our goal was to identify, select, and define a set of quality and safety indicators for use in pediatric intensive care units (PICUs) with the greatest possible consensus of experts from our scientific society.


The present study was performed in stages as follows:

  • Identification of potential quality indicators (from October to December 2014). An initial set of indicators was drawn up based on (1) existing metrics of the Spanish Society of Intensive Care Medicine of adults,5 (2) a literature search in Medline (keywords: “ pediatric critical care,” “quality markers,” and “quality indicators”), and (3) contributions of SECIP working groups through their coordinators. We also requested the collaboration of members who explicitly expressed interest in participating in the subsequent stages of the study.
  • Indicator assessment (from January to May 2015). This stage was carried out after basic Delphi methodology in 2 rounds.
  • The first round involved 31 evaluators randomly selected from members of SECIP working groups, all physicians with current clinical activity, who agreed to collaborate. Evaluators used Excel (Microsoft Office, Microsoft Corporation, Redmond, WA) spreadsheets to record their evaluation of the relevance and feasibility of each indicator, defined as follows: (1) relevance: the indicators regarded as most relevant were those that reflect either very prevalent aspects in PICUs, those associated with high morbidity and mortality, or those closely related to clinically relevant outcomes (ie, with impact on patient health). These indicators are important for a variety of stakeholders (patients and/or relatives, PICU staff, and hospital managers).2 Feasibility: the easier the measurement according to resource availability, the more useful the indicator. Evaluators used Likert ordinal scales (from 1 to 9) to evaluate relevance and feasibility, and they were encouraged to add comments and observations about each indicator.

We expressed the results of evaluations as average values and ranges. Only those indicators whose average relevance and feasibility values fell within the highest tertile in this first round were carried forward to the second round.

  • In the second round, 24 of the 31 initial evaluators with an exclusive dedication to a PICU repeated their relevance and feasibility scoring of the remaining indicators after the first round. The anonymized comments made by the evaluators in the first round were distributed to the entire group of experts to facilitate consensus. The evaluators were also asked to indicate their degree of agreement with the final selection of each potential indicator (using a 5-point ordinal scale, from “absolute disagreement” to “absolute agreement”). Only those indicators with an average value higher than 7 in relevance and feasibility, favorable vote from 70% of evaluators and no more than 2 unfavorable opinions, were positively selected from this round.
  • Final definition (from May to July 2015). In this stage, we defined the components of the final 20 indicators. We emphasized the importance of establishing mean values and standard ranges for each indicator, the potential barriers to measurement, and published evidence to support the selection of the indicator. When such evidence was scarce or not available, the indicator-defining element was decided by consensus.


The initial list of indicators comprised the contributions of 23 members of SECIP. Of the 300 members of SECIP, 60 (20%) explicitly agreed to collaborate in the subsequent stages of the project. All 31 selected evaluators were attending physicians working in PICUs located in 11 different regions of Spain. All evaluators involved in the project completed their evaluations in a timely manner.

The initial list included 136 potential indicators. This list was reduced to 43 in the first round and finally to 20 indicators after the second round (Table 1). The top 20 indicators showed a mean relevance value of 7.8 (range, 7.3–8.5) and a mean feasibility value of 7.7 (range, 6.6–8.3), of a maximum score of 9. There was unanimous consensus on only 2 indicators (rate of central venous catheter–associated bacteremia and rate of mechanical ventilation–associated respiratory infection).

Table 1.
Table 1.:
Indicators Selected after the Second Round, Classified According to the Proportion of Experts in Favor of Their Inclusion in the Final Set, Based on Average Relevance and Feasibility (Ordinal Scale from 1 to 9)

The top 20 set comprised 9 process indicators, 9 outcome metrics (especially treatment-related adverse effects), and 2 structural indicators (eg, availability of a system for notification of adverse events and nursing protocols for drug administration). Seven indicators required group consensus to propose a standard, especially some process indicators. Specific evidence-based support proved difficult to obtain for only 1 result indicator (rate of pressure ulcers [PUs] associated with noninvasive ventilation). Table 2 summarizes the main characteristics of the final top 20 indicators and bibliographic references.

Table 2.
Table 2.:
Quality Indicators Included in the Final Set (Top 20 Indicators); Main Characteristics and Published Evidence


In the present work, we identified, defined, and selected a set of readily measurable quality indicators for pediatric intensive care in Spain. This set was based on the consensus of experts belonging to working groups in our medical society.

Several groups have previously attempted to define the characteristics of ideal quality indicators in intensive care medicine, such as sufficient variability to distinguish between good and poor performers, relative insensitivity to severity adjustment, and the ability to capture what providers do rather than patients’ characteristics.26 The main conclusion is that even traditional indicators such as standardized mortality rates or mean hospital length of stay are not ideal indicators. In short, there is no gold standard in this field. In this context, the opinion of attending physicians themselves becomes more relevant. If attending physicians regard a given indicator either as unimportant or as nonmeasurable, then it is empirically useless. For this reason, we selected our indicators by expert consensus using the Delphi methodology, as described by others.1,4,6

We did not select the evaluators according to their expertise in quality management or their professional prestige in our society, as in other studies. We gave priority to the fact that all of them were interested in participating in the study, were active members of working groups of our society, and were physicians with current clinical activity. In summary, they were the same type of professionals who eventually will use the tool we are proposing.

Most of our top 20 indicators already seem as quality indicators in intensive care medicine (eg, rates of bacteremia, infection associated with mechanical ventilation, PUs, etc.).1,6 However, indicators of clinical efficiency, such as length of stay, were not prioritized in the final selection. This exclusion may reflect the different level of bed demand and case mix among the Spanish PICUs. Finally, a subset within our top 20 group represents a healthcare priority focused on children and their families, such as the parental training in cases of children with significant risk because of technological dependence or the unlimited parent visitation with their child during hospitalization.27

Many quality indicators reflect healthcare complications, and in general, their current standards are amply supported by published reports. However, we found it difficult to propose a standard for some of them: that is, the rate of PUs associated with noninvasive mechanical ventilation , which is not frequently referred to in published studies. Probably, its incidence is buried within the overall incidence of PU, with around 30% of PU in pediatrics.28 Others, such as the rate of infection associated with mechanical ventilation, are currently decreasing because of improvement projects (“pneumonia zero”), so that any standard we propose now will become obsolete in the short term.

The present work is the first consensus-based study of quality indicators in pediatric intensive care in our country, but it has limitations. First, it did not explicitly consider the criteria of other stakeholders, such as nursing staff or parents, as some previous studies have done.29,30 Second, none of the final set of indicators relates to certain complex healthcare processes (eg, cardiac surgery and organ transplantation) and techniques (eg, extracorporeal membrane oxygenation). Instead, we wished to identify general quality indicators for use in any PICU.

We believe that our results can serve as the cornerstone of a nationwide network of collaborating PICUs to monitor results, identify benchmarks, and initiate a national system of continuous improvement. There is currently no nationwide infrastructure to systematically improve pediatric intensive care in Spain. A pediatric critical care registry would be an excellent tool to drive improvement in our field, as it has been in other fields.31 In any case, our scientific society seems to be the ideal framework to collaborate with government agencies, scientific societies, or private institutions in this mission. The first step is probably to test the utility of our top 20 indicators and resolve practical problems of implementation in operational PICUs, which may even lead to further simplification of the list of quality indicators.


With broad participation of experts from our scientific society and using a collaborative methodology, we reached consensus on a set of 20 essential quality indicators for pediatric intensive care in Spain. Most are traditional indicators of intensive care medicine, but others reflect the peculiarities of pediatric intensive care. This work is a starting point for future projects of network collaboration between PICUs in Spain.


We thank the evaluators who participated in this study: Amalia Martínez, Fernando Gómez, Mónica Balaguer, Ángela Ferrer, Juan Luis Pérez, César Pérez-Caballero, Marta Brezmes, Antonio Morales, Corsino Rey, Juan I. Sánchez, Silvia Belda, Arturo Hernández, Andrés Concha, Fco. José Cambra, Antonio Rodríguez, Ramón Hernández, Julio López, Francisco Fernández, Montserrat Pujol, Pedro Gómez, Mª José Salmerón, José D. López, Elvira Morteruel, Ricardo López, David Ruano, Alberto García, Iria González, Irene García, César Villa, Victoria García, and Juan C. de Carlos. We also thank Mr. Michael McClean for his help in the translation of this article.


The authors have no financial interest to declare in relation to the content of this article.


1. Scanlon MC, Kshitij PM, Jeffries HE. Determining pediatric intensive care unit quality indicators for measuring pediatric intensive care unit safety.. Pediatr Crit Care Med. 2007;8(2 Suppl):S3S10.
2. Profit J, Gould JB, Zupancic JA, et al. Formal selection of measures for a composite index of NICU quality of care: Baby-MONITOR.. J Perinatol. 2011;31702710.
3. Kowalkowski M, Gould JB, Bose C, et al. Do practicing clinicians agree with expert ratings of neonatal intensive care unit quality measures?. J Perinatol. 2012;32247252.
4. Van den Bosch C, Hulscher M, Natsch S, et al. Development of quality indicators for antimicrobial treatment in adults with sepsis.. BMC Infect Dis. 2014;14345.
5. Sociedad Española de Medicina Intensiva Crítica y Unidades CoronariasIndicadores de calidad en el enfermo crítico. Available at: Accessed March 12, 2016.
6. Rhodes A, Moreno RP, Azoulay E, et al. Prospectively defined indicators to improve the safety and quality of care for critically ill patients: a report from the Task Force on Safety and Quality of the European Society of Intensive Care Medicine (ESICM).. Intensive Care Med. 2012;38598605.
7. Wheeler DS, Giaccone MJ, Hutchinson N, et al. A hospital-wide quality-improvement collaborative to reduce catheter-associated bloodstream infections.. Pediatrics. 2011;128e995.
    8. Lucas da Silva PS, de Carvalho WB. Unplanned extubation in pediatric critically ill patients: a systematic review and best practice recommendations.. Pediatr Crit Care Med. 2010;11287294.
      9. McLane KM, Bookout K, McCord S, et al. The 2003 national pediatric pressure ulcer and skin breakdown prevalence survey: a multisite study.. J Wound Ostomy Continence Nurs. 2004;31168178.
        10. Curley MA, Quigley SM, Lin M. Pressure ulcers in pediatric intensive care: incidence and associated factors.. Pediatr Crit Care Med. 2003;4284290.
          11. Best KM, Boullata JI, Curley MA. Risk factors associated with iatrogenic opioid and benzodiazepine withdrawal in critically ill pediatric patients: a systematic review and conceptual model.. Pediatric Crit Care Med. 2015;16175183.
            12. Aneja R, Heard AM, Fletcher JE, et al. Sedation monitoring of children by the Bispectral Index in the pediatric intensive care unit.. Pediatr Crit Care Med. 2003;46064.
              13. Simpson CD, Hawes J, James AG, et al. Use of bundled interventions, including a checklist to promote compliance with aseptic technique, to reduce catheter-related bloodstream infections in the intensive care unit.. Paediatr Child Health. 2014;19e20e23.
                14. Frey B, Argent A. Safe paediatric intensive care. Part 2: workplace organisation, critical incident monitoring and guidelines.. Intensive Care Med. 2004;3012921297.
                  15. Cifra CL, Bembea MM, Fackler JC, et al. The morbidity and mortality conference in PICUs in the United States: a national survey.. Crit Care Med. 2014;4222522257.
                    16. Jordan García I, Bustinza Arriourtúa A, Concha Torre JA, et al. Estudio multicéntrico nacional sobre la infección nosocomial en la UCIP.. An Pediatr (Barc). 2014;802833.
                      17. Kochanek PM, Carney N, Adelson PD, et al. Guidelines for the acute medical management of severe traumatic brain injury in infants, children, and adolescents – second edition.. Pediatr Crit Care Med. 2012;13(Suppl 1):S1S82.
                        18. Matzke GR, Aronoff GR, Atkinson AJ Jr, et al. Drug dosing consideration in patients with acute and chronic kidney disease-a clinical update from Kidney Disease: Improving Global Outcomes (KDIGO).. Kidney Int. 2011;8011221137.
                          19. Rinke ML, Bundy DG, Velasquez CA, et al. Interventions to reduce pediatric medication errors: a systematic review.. Pediatrics. 2014;134338360.
                            20. Slater A, Shann F; ANZICS Paediatric Study Group. The suitability of the Pediatric Index of Mortality (PIM), PIM2, the Pediatric Risk of Mortality (PRISM), and PRISM III for monitoring the quality of pediatric intensive care in Australia and New Zealand.. Pediatr Crit Care Med. 2004;5447454.
                              21. Lee GY, Yamada J, Kyololo O, et al. Pediatric clinical practice guidelines for acute procedural pain: a systematic review.. Pediatrics. 2014;133500515.
                                22. Knight LJ1, Wintch S, Nichols A, et al. Saving a life after discharge: CPR training for parents of high-risk children.. J Healthc Qual. 2013;35916.
                                  23. Calvo Macías C, López-Herce Cidb J, Carrillo Álvarezb A, et al. Material del carro de Reanimación de reanimación cardiopulmonar pediátrica.. An Pediatr (Barc). 2007;665154.
                                    24. Davidson JE, Powers K, Hedayat KM, et al. Clinical practice guidelines for support of the family in the patient-centered intensive care unit: American College of Critical Care Medicine Task Force 2004–2005.. Crit Care Med. 2007;35605622.
                                      25. Cravero JP, Blike GT, Beach M, et al. Incidence and nature of adverse events during pediatric sedation/anesthesia for procedures outside the operating room: report from the Pediatric Sedation Research Consortium.. Pediatrics. 2006;11810871096.
                                        26. Brown SE, Ratcliffe SJ, Halpern SD. An empirical comparison of key statistical attributes among potential ICU quality indicators.. Crit Care Med. 2014;4218211831.
                                        27. Meert KL, Clark J, Eggly S. Family-centered care in the pediatric intensive care unit.. Pediatr Clin North Am. 2013;60761772.
                                        28. Visscher M, King A, Nie AM, et al. A quality-improvement collaborative project to reduce pressure ulcers in PICUs.. Pediatrics. 2013;131e1950e1960.
                                        29. Kea B, Sun BCA. Consensus development for healthcare professionals.. Intern Emerg Med. 2015;10373383.
                                        30. Boulkedid R, Abdoul H, Loustau M, et al. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review.. PLoS One. 2011;6e20476.
                                        31. Davids MR, Eastwood JB, Selwood NH, et al. A renal registry for Africa: first steps.. Clin Kidney J. 2016;9162167.
                                        Copyright © 2017 the Author(s). Published by Wolters Kluwer Health, Inc. All rights reserved.