Clinical trials provide evidence for best clinical practice, but comparisons of findings across trials are difficult because of differences in the selection and reporting of trial outcomes. The resulting inability to compare findings not only diminishes our ability to synthesize evidence and address knowledge gaps but wastes research resources (1). Several Cochrane Reviews of trials evaluating ventilator weaning protocols spanning 2 decades have highlighted substantial variation in outcome selection, definition, measurement, and reporting (2–4). More recently, we (B.B., M.C., J.C.M., L.R., D.F.M.) have shown in a review of 66 trials that inconsistency in ventilation outcome reporting in critical care trials remains problematic: among the 66 trials, only 12 (25%) provided a definition and 16 different definitions of duration of mechanical ventilation were used (5). Ideally, trials focused on the same health condition and intervention should assess the same clinically meaningful outcomes and measure them in a similar fashion to facilitate more appropriate comparisons.
Standardization in selection and measurement of a core outcome set (COS) has been proposed as a method of addressing problems of inconsistency. A COS is defined as the minimum set of outcomes that should be measured and reported in all clinical trials of a specific condition (6). Using a COS ensures that trials collect the same outcomes in standard ways, which increases availability of the most important and relevant information for meta-analyses. The Core Outcome Measures in Effectiveness Trials (COMET) initiative, a resource repository for COS researchers and users, summarizes the current methodological approach for COS development (7). The recommended multistage process advocates obtaining consensus on “what” outcomes should be in the COS, before addressing “how” and “when” these outcomes should be measured (7). Within critical care, the effort to drive COS development forward is championed by the International Forum for Acute Care Trialists (InFACT) and is receiving widespread attention by the critical care research community (8). COS for cardiac arrest (9) and long-term outcomes in acute respiratory failure are already completed (10), with others relating to physical rehabilitation (11), delirium (12), and subarachnoid hemorrhage (13) are at different stages of development.
We sought to develop a COS for trials testing any intervention intended to modify mechanical ventilation duration in critical care. In recognizing that trial outcomes should include health results most relevant to patients, our specific objectives were to engage relevant participants in identifying important core outcomes for these trials and to obtain agreement on how they should be defined, measured, and reported. This COS is intended for use in randomized trials of interventions for invasively mechanically ventilated adults in critical care, but would also be suitable for nonrandomized studies.
MATERIALS AND METHODS
This Core Outcomes in Ventilation Trials (COVenT) study used a two-stage, mixed methods, consensus approach (Fig. 1). COVenT Stage I determined “what” outcomes to include. COVenT Stage II determined “how” the agreed outcomes should be measured. Methods included systematic reviews of published (5), unpublished (J Friedrich and J Marshall, InFACT February 29, 2016, personal communication); Ringrow et al (14) and online information sources (Improve Long-Term Outcome COS www.improvelto.com); a three-round, online Delphi study supported by a bespoke e-management system (DelphiManager, Version 1.0 2014, University of Liverpool, United Kingdom); webinar and face-to-face consensus meetings. For more details on methods, see eAppendix 1 and eAppendix 2 for the Delphi questionnaire (Supplemental Digital Content 1, http://links.lww.com/CCM/E744).
To recruit Delphi participants, we contacted a broad range of international organizations representing: ICU survivor and carer support groups; nursing, allied health professional (AHP), and medical critical care societies; critical care clinical trials groups and trial investigators; and industries involved with ventilation equipment. Organization leads either notified members about the study for self-selection or identified member representatives. Consensus meeting participants were recruited from Delphi respondents completing all three rounds. Additionally, we purposively invited a cohort of non-Delphi participants, including a statistician, health economist, two critical care journal, and social media editors and others to provide specific methodological expertise. Names and affiliations of COVenT participants are listed in eAppendix 3 (Supplemental Digital Content 1, http://links.lww.com/CCM/E744).
COVenT was registered on the COMET database, and the study protocol was published (1516). The COVenT study was ethically approved by Queen’s University Belfast, School of Medicine Ethics Committee (reference: 14.34v2; October 3, 2014). Findings are reported according to accepted standards for reporting COS development (17).
In total, 222 participants were recruited and 183 contributed to the final consensus process. The final 183 included 161 participants that completed all three rounds of the Delphi study and an additional 22 non-Delphi participants. Table 1 shows the stakeholder composition and geographical spread of the 183 participants.
COVenT Stage I Results
The Delphi study was conducted from December 2015 to March 2016 and 200 participants participated in round one. Subsequently, 178 of 200 (89%) completed round two and 161 of 178 (90%) completed round three (Fig. 1). The proportions of panel participants were 9% ICU survivors/carers; 50% nurses, AHP, and physicians; 35% trials group members and investigators; and 6% industry. Participant stakeholder groups and geographical locations are shown in eTable 1 (Supplemental Digital Content 1, http://links.lww.com/CCM/E744).
Round one contained 24 outcomes and Delphi participants recommended 23 additional outcomes that were added to round two (47 outcomes in total) (see eAppendix 2 for the questionnaire, Supplemental Digital Content 1, http://links.lww.com/CCM/E744). After round three, 19 outcomes met consensus criteria (> 70% participants scoring the outcome as critical and not more than 15% rating the outcome not critical). We classified outcomes using the COMET taxonomy structure into categories of death; physiologic/clinical; life impact; resource use; and adverse events (18) (eTable 2, Supplemental Digital Content 1, http://links.lww.com/CCM/E744). The research team (authors) considered the taxonomy and noted overlapping similarities in outcomes within categories (e.g., mortality and survival). Furthermore, 19 outcomes were considered too large for a COS; therefore, these were taken forward for further discussion and voting “in,” “out,” or “unsure” at three consensus meetings: two webinar meetings (October 11, 2016, and October 17, 2016; n = 33 participants); and a follow-up teleconference with ICU survivors and family carers (October 25, 2016; n = 4). Webinar voting details are presented in eTables 3 and 4 (Supplemental Digital Content 1, http://links.lww.com/CCM/E744).
From the consensus meetings, nine outcomes were voted in as “core” (Table 2), but only six were universally agreed by all. These six form the COS are as follows: extubation, reintubation, duration of mechanical ventilation, length of stay, health-related quality of life (HRQOL), and mortality. The three outcomes not universally agreed were survival, pulmonary complications, and delirium.
COVenT Stage II Results
We convened two webinar meetings (November 13, 2017, and November 17, 2017) and a face-to-face meeting (February 6, 2018) including 37 participants to determine definitions and measures for the six core outcomes. At the meetings we presented a set of preliminary definitions and measures derived from the aforementioned systematic reviews and based on four recommended components for defining an outcome (specific variable, analysis metric, aggregation method, and time point) (19) (preliminary list detailed in eTable 5, Supplemental Digital Content 1, http://links.lww.com/CCM/E744). Consensus was gained through discussion and voting. Webinar participants reached agreement on definitions and measures for four outcomes (duration of mechanical ventilation, length of stay, HRQOL, and mortality), but did not reach consensus on two (extubation and reintubation). The remaining two outcomes were discussed and agreed in the final face-to-face meeting (eTable 5, Supplemental Digital Content 1, http://links.lww.com/CCM/E744).
Table 3 presents the final core outcome measurement set detailing how the outcomes should be specifically defined and measured and, where necessary, provides additional clarification and recommendations. Table 4 details the study variable fields that investigators can insert into an electronic case report form for a clinical trial.
This large, international consensus study, including participants from 38 organizations in 27 countries has established a COS for critical care trials of interventions intended to modify mechanical ventilation duration. The six core outcomes are extubation, reintubation, duration of mechanical ventilation, length of stay, HRQOL, and mortality. Additionally, this study provides the specific measurement variable; analysis metric; aggregation method; and measurement time-point for each outcome. All outcome data should be reported for survivors and, where appropriate, nonsurvivors. This COS represents the minimum number of outcomes. Additional outcomes and measurement time points can be added at the investigators’ discretion. To our knowledge, this study is the first to develop a COS for ventilation trials, as such, this represents important new insight for critical care investigators.
The validity of this COS is strengthened by following a robust development process, including a Delphi study and consensus meetings that enabled wide representation and engagement. The sample size for consensus processes is not determined by achieving statistical power, rather the main consideration is maximizing representation from key stakeholder groups (7). We achieved this through broad geographical and stakeholder composition. The importance of a wide stakeholder group in COS development is not to be underestimated in achieving agreement on outcomes that are relevant and important to patients, clinicians, and researchers.
Duration of mechanical ventilation and length of stay outcomes relate to successful medical management and prompt liberation from mechanical ventilation. Although they are commonly used in ventilation trials, they are inconsistently defined and measured (5); therefore, their appearance in the COS was not unexpected. Likewise, extubation and reintubation, while less frequently measured in trials (20), influence ventilation duration and length of stay and thus require clinically meaningful definitions. Conversely, outcomes important to patients, such as HRQOL are not generally profiled in trials. Indeed, a recent systematic review of critical care trials found only 22% (160/713) reported patient-important outcomes with HRQOL accounting for only 3% (n = 22 trials) (20). The inclusion of less commonly reported, but nevertheless, clearly important outcomes in the COS serves to show the value of engaging patients in COS development.
This COS defines randomization as the measurement start point for the time-related outcomes, duration of mechanical ventilation, and length of stay. Participants considered randomization a more reliable start point as it marks commencing delivery of the experimental or control intervention to modify an outcome. However, for research designs where randomization does not signal commencement of intervention delivery, the COS recommends that alternative start point should be clearly justified and defined.
The recommended endpoint for extubation and duration of mechanical ventilation is defined as “successful extubation” and “unassisted breathing,” respectively, at 48 hours. A previous round table conference on weaning defined 48 hours as the time-point measure of success, but this is now more than 10 years old (21). In the light of newer practices incorporating noninvasive ventilation as a weaning strategy or to treat post extubation respiratory distress, arguably identification of success of extubation should be at a more distal time-point (22). However, although there is no evidence for adopting the 48-hour period to define success, in the current study, participant expert opinion upheld 48 hours because any subsequent need for intubation and ventilation would likely be related to a new clinical event. Additionally, the endpoint for duration of mechanical ventilation, “unassisted breathing,” is defined as a patient being free from invasive ventilation, including extracorporeal lung support and noninvasive ventilation delivering volume or pressure support. Although we recognize that continuous positive airway pressure and high-flow oxygen therapy may not be truly “unassisted,” these modes of ventilation were excluded from the definition of assisted breathing. Further, high-flow oxygen therapy devices technically only modify percentage of oxygen delivery and rate of gas flow rather than assisted support as provided by the other forms of mechanical ventilation (23).
Participants recognized that achieving standardized measures of length of stay was challenging due to international differences in healthcare provision. Length of stay is generally used as a benchmark to assess healthcare systems, with shorter stays typically associated with system efficiency (24) and for this reason, it was deemed important for the COS. Although length of stay may reflect important progress in a patient’s trajectory of recovery, structural and process factors may impact its standardization. The average length of stay, in both ICU and hospital, varies considerably between countries, with nonclinical factors, such as professional or cultural settings, differing public and private healthcare reimbursement schemes, and access to long-term care and ventilator facilities contributing to international variability (2526). Given the influence of these factors on duration of stay and challenges in modifying them, the COS recommends that investigators clearly define the setting (hospital and/or critical care facility) and any censoring points in their study protocol to facilitate understanding of comparisons and generalizability.
The reported mortality endpoint in critical care trials has varied from 28 to 90 days and, in some trials, up to 1 year (5). In selecting a pragmatic mortality time-point for the COS, participants considered a number of important reasons for agreeing on 60 days. First, measuring ICU and hospital mortality was considered limiting due to potential variability in duration of stay as a result of factors unrelated to the patient’s condition. Second, the time-point should be stable and realistically capture the full effects of the intervention. Although a short time-point (e.g., 28 d) might capture immediate survival impact, the mortality rate could be manipulated by delaying clinical decisions (e.g., withdrawal of treatment); therefore, participants considered a more distal time-point preferable. The 60-day choice was informed by a systematic review of trials reporting mortality at three or more time-points that showed significant incremental risk difference between treatment arms to at least 60 days after randomization (J. Friedrich and J. Marshall, personal communication, 2016). A third reason for choosing 60 days was that the time-point should not substantially increase the burden of follow-up.
Defining the HRQOL outcome measure was greatly assisted because the Improve Long-Term Outcome group had already completed a COS that included HRQOL measures. The group recommended the EuroQol 5 dimensions 5 levels (EQ-5D-5L) measurement tool at a 6-month time point (https://www.improvelto.com/) (10). A number of COVenT participants had also participated in the Improve Long-Term Outcome project, and so agreement was readily reached that it was appropriate for the COVenT COS.
At the end of stage I, three outcomes (survival, pulmonary complications, and delirium) reached inclusion criteria for the COS by one or more consensus groups, but as they were not universally agreed by all groups, they were excluded from the COS. Nevertheless, these outcomes may be captured in other ways. For the mortality outcome measure, the COVenT COS recommends documenting date and time of randomization and death which would enable a survival analysis to be undertaken. Pulmonary complications can be documented and reported alongside other trial adverse events. Delirium is being addressed in a separate COS development study (12); therefore, ventilation studies that also want to incorporate standardized delirium outcomes could add these, when they become available, to their trial measures.
A limitation of our study was the low number of patient participants, a challenge that has been experienced in other COS development studies (27). We planned to recruit from ICU support groups, but accessing these proved difficult as they are not well established internationally. Given the importance of patient involvement in research (28), we asked clinical trials groups and the European Federation of Critical Care Nurses associations for contact details of support groups in their countries, but received no suggestions highlighting a need for more work to encourage patient and carer engagement in research. ICU survivors and family carers in our study were recruited from Canada and a U.K. charity (https://icusteps.org/).
We also experienced challenges in engaging participants representing research funding organizations. U.K. funders were concerned about the potential for exponential requests to participate in Delphi studies due to the growing interest in this field of research. However, the role of funding organizations in the overall COS development process may be more valuable for subsequent promotion and dissemination through mandating their use in funding applications. This is the case in the United Kingdom, where the National Institute for Health Research, Health Technology Assessment guidance documents encourage applicants to use established COS among the list of outcomes where these are available (29). As a result, it is likely that we will see a rising tide of COS development and use in future trials.
Moving forward, we plan to disseminate the COS through the COVenT participants and their organizations, conference presentations, social media, research funders, and relevant Cochrane Review Groups. We will monitor the uptake of the COVenT COS with a planned cohort study that aims to identify if participation in the COVenT COS development effects uptake by individual researchers over the next decade (16).
Using rigorous and well-established methods, we reached international consensus on the COVenT COS that should be reported in all future trials of interventions designed to modify the duration of invasive mechanical ventilation in critical care. The set comprises standardized definitions and measures for six outcome measures (extubation, reintubation, duration of mechanical ventilation, length of stay, HRQOL, and mortality). We are confident that by standardizing these outcomes, this COS will benefit future research in this field. We recommend that trialists and systematic reviewers use this COS to enable cross-study comparisons, minimize outcome reporting bias, and ensure that their research findings provide information deemed important by patients, clinicians, and policy makers.
We thank all those who took part in the Core Outcomes in Ventilation Trials (COVenT) study; without their support, time, and expertise this study could not have been conducted. We acknowledge the support of the following 38 organizations and societies in facilitating participant recruitment: Acute Respiratory Distress Syndrome Clinical Network (ARDSNet); American Thoracic Society (ATS); Asian Critical Care Trial Group; Australia and New Zealand Intensive Care Research Centre at Monash University; Australia and New Zealand Intensive Care Society Clinical Trials Group (ANZICS CTG); Canadian Critical Care Trials Group (CCCTG); Canadian Society of Respiratory Therapists; Chinese Society of Critical Care Medicine; Clinical Research, Investigation and Systems Modeling of Acute Illness (CRISMA); Cochrane: Anaesthesia, Critical Care and Emergency Group; Core Outcome Measures in Effectiveness Trials Initiative (COMET Initiative); Dräger; European Federation of Critical Care Nurses Associations; European Respiratory Society; European Society of Paediatric Neonatal Intensive Care (ESPNIC); European Society of Intensive Care Medicine (ESICM); Faculty of Intensive Care Medicine; Hamilton Medical; Health Experiences Research Group (HERG); Hellenic Sepsis Network; ICUSteps; IMT Medical; Intensive Care National Audit and Research Centre (ICNARC); International Forum for Acute Care Trialists (InFACT); International Sepsis Forum (ISF); Latin American Critical Care Trial Investigators Network (LACCTIN); Paediatric Intensive Care Society; Pediatric Acute Lung Injury and Sepsis Investigators (PALISI); Queens University Belfast (QUB); Réseau européen de Recherche en ventilation Artificielle (REVA); Scandinavian Critical Care Trials Group; Scandinavian Society of Anaesthesiology and Intensive Care; Scottish Intensive Care Society of Critical Care Medicine; The Clinical Trials Network for the Prevention and Early Treatment of Acute Lung Injury (PETAL Network); The United States Critical Illness and Injury Trials Group (USCIITG); United Kingdom Critical Care research Forum (UKCCRF); United Kingdom Intensive Care Society (ICS); United Kingdom Intensive Care Society Clinical Trials Group (UK ICS CTG); and World Federation of Physical Therapy. We thank Melanie Harper-Jones, Duncan Appelbe, and Paula Williamson from the University of Liverpool for online Delphi development and management; and members of Centre Cochrane Français (Amelie Yavchitz and Philippe Ravaud) for conducting the PubMed search to identify clinical trial investigators. We also thank Professor David Harrison (Head Statistician, Intensive Care National Audit & Research Centre, London, United Kingdom) for assistance with defining outcome analysis metrics and aggregation methods.
1. Chan AW, Song F, Vickers A, et al. Increasing value and reducing waste: Addressing inaccessible research. Lancet 2014; 383:257–266
2. Blackwood B, Murray M, Chisakuta A, et al. Protocolized versus non-protocolized weaning for reducing the duration of invasive mechanical ventilation
in critically ill paediatric patients. Cochrane Database Syst Rev 2013; (7):CD009082
3. Blackwood B, Burns KEA, Cardwell CR, et al. Protocolized versus non-protocolized weaning for reducing the duration of mechanical ventilation
in critically ill adult patients. Cochrane Database Syst Rev 2010; (5):CD006904
4. Rose L, Schultz MJ, Cardwell CR, et al. Automated versus non-automated weaning for reducing the duration of mechanical ventilation
for critically ill adults and children. Cochrane Database Syst Rev 2014; (6):CD009235
5. Blackwood B, Clarke M, McAuley DF, et al. How outcomes are defined in clinical trials
of mechanically ventilated adults and children. Am J Respir Crit Care Med 2014; 189:886–893
6. Williamson PR, Altman DG, Blazeby JM, et al. Developing core outcome sets for clinical trials
: Issues to consider. Trials 2012; 13:132
7. Williamson PR, Altman DG, Bagley H, et al. The COMET handbook: Version 1.0. Trials2017; 18(Suppl 3):280
8. Blackwood B, Marshall J, Rose L. Progress on core outcome sets for critical care research. Curr Opin Crit Care 2015; 21:439–444
9. Haywood K, Whitehead L, Nadkarni VM, et al.; COSCA Collaborators: COSCA (Core Outcome Set for Cardiac Arrest) in adults: An advisory statement from the International Liaison Committee on Resuscitation. Circulation 2018; 137:e783–e801
10. Needham DM, Sepulveda KA, Dinglas VD, et al. Core outcome measures for clinical research in acute respiratory failure survivors. An international modified Delphi consensus study. Am J Respir Crit Care Med 2017; 196:1122–1130
11. Connolly B, Denehy L, Hart N, et al. Physical rehabilitation core outcomes in critical illness (PRACTICE): Protocol for development of a core outcome set. Trials 2018; 19:294
12. Rose L, Agar M, Burry LD, et al.; Del-COrS group: Development of core outcome sets for effectiveness trials of interventions to prevent and/or treat delirium (Del-COrS): Study protocol. BMJ Open 2017; 7:e016371
13. Finfar S, Delaney A, Andersen C. A Core Outcome Set in Subarachnoid Haemorrhage Research. 2016. Available at: http://www.comet-initiative.org/studies/details/1198
. Accessed October 7, 2018
14. Ringrow S, Liu D, Reade M, et al. Defining and Measuring Successful Extubation and Reintubation As Part of the COVenT Core Outcome Set; A Protocol for a Systematic Review. 2017. Available at: http://www.comet-initiative.org/studies/details/994?result=true
. Accessed October 7, 2018
15. Blackwood B, Ringrow S, McAuley DF, et al. Standardizing Reporting of Core Outcome Measures in Ventilation Studies. 2012. Available at: http://www.comet-initiative.org/studies/details/292
. Accessed October 7, 2018
16. Blackwood B, Ringrow S, Clarke M, et al. Core outcomes in ventilation trials (COVenT): Protocol for a core outcome set using a Delphi survey with a nested randomised trial and observational cohort study. Trials 2015; 16:368
17. Kirkham JJ, Gorst S, Altman DG, et al. Core outcome set-standards for reporting: The COS-STAR statement. PLoS Med 2016; 13:e1002148
18. Dodd S, Clarke M, Becker L, et al. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. J Clin Epidemiol 2018; 96:84–92
19. Chan AW, Tetzlaff JM, Gøtzsche PC, et al. SPIRIT 2013 explanation and elaboration: Guidance for protocols of clinical trials
. BMJ 2013; 346:e7586
20. Gaudry S, Messika J, Ricard JD, et al. Patient-important outcomes in randomized controlled trials in critically ill patients: A systematic review. Ann Intensive Care
21. Boles JM, Bion J, Connors A, et al. Weaning from mechanical ventilation
. Eur Respir J 2007; 29:1033–1056
22. Thille AW, Richard JC, Brochard L. The decision to extubate in the intensive care
unit. Am J Respir Crit Care Med 2013; 187:1294–1302
23. Helviz Y, Einav S. A systematic review of the high-flow nasal cannula for adult patients. Crit Care 2018; 22:71
24. Needham DM, Anderson G, Pink GH, et al. A province-wide study of the association between hospital resource allocation and length of stay. Health Serv Manage Res 2003; 16:155–166
25. Tiessen J, Kambara H, Sakai T, et al. What causes international variations in length of stay: A comparative analysis for two inpatient conditions in Japanese and Canadian hospitals. Health Serv Manage Res 2013; 26:86–94
26. Kahn JM, Werner RM, Carson SS, et al. Variation in long-term acute care hospital use after intensive care
. Med Care Res Rev 2012; 69:339–350
27. Gargon E. Choosing important health outcomes for comparative effectiveness research: A systematic review. PLoS One 2014; 9:e99111
28. Price A, Albarqouni L, Kirkpatrick J, et al. Patient and public involvement in the design of clinical trials
: An overview of systematic reviews. J Eval Clin Pract 2018; 24:240–253
29. NIHR: Guidance Notes for Completing HTA Expressions of Interest. 2017. Available at: https://www.nihr.ac.uk/funding-and-support/documents/current-funding-opportunities/hta/HTA-EoI-Guidance-Notes.pdf
. Accessed October 7, 2018