Secondary Logo

Journal Logo

Institutional members access full text with Ovid®

Maintaining Interrater Agreement of Core Assessment Instruments in a Multisite Randomized Controlled Clinical Trial

The Randomized Evaluation of Sedation Titration for Respiratory Failure (RESTORE) Trial

Lebet, Ruth; Hayakawa, Jennifer; Chamblee, Tracy B.; Tala, Joana A.; Singh, Nakul; Wypij, David; Curley, Martha A. Q.

doi: 10.1097/NNR.0000000000000224

Background RESTORE (Randomized Evaluation of Sedation Titration for Respiratory Failure) was a cluster randomized clinical trial evaluating a sedation strategy in children 2 weeks to <18 years of age with acute respiratory failure supported on mechanical ventilation. A total of 31 U.S. pediatric intensive care units (PICUs) participated in the trial. Staff nurse rater agreement on measures used to assess a critical component of treatment fidelity was essential throughout the 4-year data collection period.

Objective The purpose of the study is to describe the method of establishing and maintaining interrater agreement (IRA) of two core clinical assessment instruments over the course of the clinical trial.

Methods IRA cycles were carried out at all control and intervention sites and included a minimum of five measurements of the State Behavioral Scale (SBS) and Withdrawal Assessment Tool-Version 1 (WAT-1). Glasgow Coma Scale scores were also obtained. PICUs demonstrating <80% agreement repeated their IRA cycle. Fleiss’s kappa coefficient was used to assess IRA.

Results Repeated IRA cycles were required for 8% of 226 SBS cycles and 2% of 222 WAT-1 cycles. Fleiss’s kappa coefficients from more than 1,350 paired assessments were .86 for SBS and .92 for WAT-1, demonstrating strong agreement and similar to .91 for the Glasgow Coma Scale. There was no difference in Fleiss’s kappa for any of the instruments based on unit size or timing of assessment (earlier or later in the study). For SBS scores, Fleiss’s kappa was significantly different in larger and smaller PICUs (.82 vs. .92, p = .003); however, Fleiss’s kappa for both groups indicated excellent agreement.

Conclusion Monitoring measurement reliability is an essential step in ensuring treatment fidelity and, thus, the validity of study results. Standardization on the use of these core assessment instruments among participating sites was achieved and maintained throughout the trial.

Ruth Lebet, MSN, CCNS-P, is Lecturer, School of Nursing, University of Pennsylvania, Philadelphia.

Jennifer Hayakawa, DNP, PCNS-BC, CNRN, CCRN, is Clinical Nurse Specialist, Pediatric Intensive Care Unit, CHOC Children’s Hospital, Orange, California, and Clinical Faculty, Western University of Health Sciences, Pomona, California.

Tracy B. Chamblee, PhD, APRN, PCNS-BC, is Clinical Nurse Specialist, Pediatric Intensive Care Unit, Children’s Medical Center Dallas, Texas.

Joana A. Tala, MD, is Research Coordinator, Pediatric Intensive Care Unit, Yale New Haven Hospital/Yale University, Connecticut.

Nakul Singh, ScM, is Biostatistician, Department of Cardiology, Boston Children’s Hospital, Massachusetts.

David Wypij, PhD, is Senior Biostatistician, Department of Cardiology, Boston Children’s Hospital; Associate Professor, Department of Pediatrics, Harvard Medical School; and Senior Lecturer, Department of Biostatistics, Harvard T. H. Chan School of Public Health, Boston, Massachusetts.

Martha A. Q. Curley, RN, PhD, FAAN, is Ellen and Robert Kapito Professor in Nursing Science, School of Nursing, University of Pennsylvania, Philadelphia, and Nurse Scientist, Boston Children’s Hospital, Massachusetts.

Accepted for publication December 19, 2016.

The authors gratefully acknowledge the statistical support of Lisa A. Asaro, MS, Biostatistician II, and IRA cycle coordination support of Donna M. Duva, Project Manager, Data Coordinating Center, Department of Cardiology, Children’s Hospital, Boston. In addition, they acknowledge the support of the RESTORE Investigators and site teams.

This article is a supplementary analysis of data collected during the Randomized Evaluation of Sedation Titration for Respiratory Failure (RESTORE) clinical trial. Trial Registration:, Identifier: NCT00814099.

This article was supported, in part, by the National Institutes of Health; the National Heart, Lung, and Blood Institute; and the National Institute of Nursing Research (U01 HL086622 and U01 HL086649; PI: Curley and Wypij).

The authors have no conflicts of interest to report.

Corresponding author: Ruth Lebet, MSN, CCNS-P, University of Pennsylvania School of Nursing, Claire M. Fagin Hall, 418 Curie Boulevard, Philadelphia, PA 19104-4217 (e-mail:

Copyright © 2017 Wolters Kluwer Health, Inc. All rights reserved