Secondary Logo

Share this article on:

Multi-modal Educational Curriculum to Improve Richmond Agitation-sedation Scale Inter-rater Reliability in Pediatric Patients

Kihlstrom, Margaret J., MD*; Edge, Ashley P., MHA, CSSBB; Cherry, Kelly M., RN*; Zarick, Paul J., BSN, RN, CCRN*; Beck, Shawna D., PharmD; Boyd, Jenny M., MD*

doi: 10.1097/pq9.0000000000000096
Individual QI projects from single institutions

Introduction: The Richmond Agitation-sedation Scale (RASS) is a reliable and valid scale for assessing sedation in critically ill pediatric patients. This investigation evaluates the inter-rater reliability of the RASS in mechanically ventilated pediatric patients before and after an educational intervention.

Methods: This prospective, interventional quality improvement study was completed in a 20-bed pediatric intensive care unit from July 2013 to July 2014. Children 0–18 years of age requiring mechanical ventilation and receiving sedative or analgesic medications were eligible. Staff completed simultaneous paired RASS assessments in 3 phases: baseline, after educational intervention, and maintenance.

Results: Staff completed 347 paired assessments on 45 pediatric intensive care unit patients: 49 in the baseline phase, 228 in the postintervention phase, and 70 in the maintenance phase. There was a significant increase in the weighted κ after the intervention, from 0.56 (95% CI, 0.39–0.72) to 0.86 (95% CI, 0.77–0.95; P < 0.001). The improvement was maintained months later with weighted κ 0.78 (95% CI, 0.61–0.94). In subgroup analysis, there was an increase in weighted κ in patients less than 1 year of age (0.41–0.87) and those with developmental delay (0.49–0.84).

Conclusions: The RASS is a reliable tool for sedation assessment in mechanically ventilated, sedated pediatric patients after implementation of an educational intervention. It is also reliable in patients less than 12 months of age and patients with developmental delay. The ability to easily educate providers to utilize a valid, reliable sedation tool is an important step toward using it to provide consistent care to optimize sedation.

From the *Division of Pediatric Critical Care Medicine, Department of Pediatrics, University of North Carolina at Chapel Hill, Chapel Hill, N.C.

Practice Quality and Innovation, University of North Carolina Health Care, Chapel Hill, N.C.

Department of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, N.C.

Received for publication February 18, 2018; Accepted July 10, 2018.

Published online August 7, 2018

Presented at the Society of Critical Care Medicine Annual Critical Care Congress, January 2015, Phoenix, Ariz (Kihlstrom MJ, Boyd, JM. Richmond Agitation Sedation Scale Inter-Rater Reliability in Critically Ill Pediatric Patients).

To cite: Kihlstrom MJ, Edge AP, Cherry KM, Zarick PJ, Beck SD, Boyd JM. Multi-modal Educational Curriculum to Improve Richmond Agitation-sedation Scale Inter-rater Reliability in Critically Ill Pediatric Patients. Pediatr Qual Saf 2018;3:096.

*Corresponding author. Address: Margaret J. Kihlstrom, MD, 417 MacNider Hall, CB#7221, Chapel Hill, NC 27599-7221, PH: 919-966-7495; fax: 919-843-7036, Email: margaret_kihlstrom@med.unc.edu

This is an open access article distributed under the Creative Commons Attribution License 4.0 (CCBY), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Back to Top | Article Outline

INTRODUCTION

Providing appropriate sedation and analgesia for mechanically ventilated critically ill pediatric patients is a vital aspect of their care. When delivered optimally, sedation and analgesia medications can alleviate pain and anxiety, enhance patient safety, decrease hospital length of stay, and facilitate performance of invasive procedures.1–7 However, no standard practice exists to assess or achieve the desired sedation level, leading to wide variation.8–10 Similarly, no consensus exists regarding the optimal level of sedation, with suboptimal sedation levels in many pediatric intensive care unit (PICU) patients.1

A multidisciplinary panel from the United Kingdom recommended the regular assessment and documentation of sedation level using a formal sedation assessment scale.8 The Richmond Agitation-sedation Scale (RASS) is a valid and reliable tool used for sedation assessments in adult patients.11–18 The RASS is becoming a common tool to evaluate sedation level in pediatric patients as supported by its inclusion as a component in 2 pediatric delirium assessment tools and a recent study by Kerson et al.,19 which demonstrated that it is valid and reliable in assessing sedation in critically ill pediatric patients.20 , 21 However, to date, no studies exist that explore the use of an educational intervention to improve and maintain the reliability of the scale over time in pediatric patients.

This investigation evaluates the inter-rater reliability (IRR) of the RASS in mechanically ventilated pediatric patients before and after an educational intervention in a PICU in which staff used the RASS as a sedation assessment for over 5 years. The research team chose the RASS, given its quick and easy use at the bedside compared with other scales; the staff’s prior familiarity with the scale; and its existing integration into the medical record for documentation. We hypothesized that the reliability would improve significantly with an educational intervention. Because the pediatric population is unique in that assessments of sedation and pain can be affected by young age and developmental delay,22–25 the reliability in these 2 groups was also specifically evaluated.

Back to Top | Article Outline

METHODS

This prospective interventional study was conducted in the PICU at North Carolina Children’s Hospital in Chapel Hill, NC, which is a 20-bed unit with 1,200 admissions annually that include medical, surgical, and cardiac patients. The institutional review board approved the study and waived consent for patients and staff involved. IRR data were collected in 4 phases from July 7, 2013, to July 16, 2014, on patients 18 years of age or younger requiring mechanical ventilation and receiving infusions of sedative or analgesic medications. At the time of the study, no sedation protocol was in place in the PICU; the ordering provider determined the infusions. Commonly used medications included fentanyl, morphine, midazolam, and dexmedetomidine; the study rarely included ketamine and propofol. Excluded patients included those with preexisting impaired hearing, impaired vision, and those receiving neuromuscular blockade. A core research team of 2 physicians and 2 nurses performed RASS assessments with PICU nurses and physicians. Assessments were performed using a slightly adapted version of the original RASS, which Ely et al.13 found to be valid and reliable in adult ICU patients (Table 1). The research team decided that the criteria of eye-opening in addition to eye contact for scores −1 to −3 enhances its use in neonates who do not make eye contact developmentally. Subgroup analysis was also performed on patients less than 1 year of age and those with developmental delay, as defined by the problem list during admission.

Table 1

Table 1

Back to Top | Article Outline

Baseline Phase

From July 7, 2013, to September 12, 2013, assessments of RASS IRR were performed in the PICU to ascertain a baseline among unit staff. The RASS was already utilized for sedation documentation by nurses in the electronic medical record; however, they had previously received no formal education on how to utilize the tool. A member of the research team combined with a PICU nurse or physician obtained IRR measurements by performing simultaneous independent RASS assessments of mechanically ventilated patients. The team first observed the patient for any signs of agitation. If none were present, the patient was then verbally stimulated, followed by physical stimulation if there was no response to voice. A RASS score was then assigned based on the scale in Table 1. Each participant was blinded to the scores of other participants. The majority of assessments were completed between 7 AM and 7 PM on weekdays due to the availability of research team members.

Back to Top | Article Outline

Intervention Phase

From October 1, 2013, to February 23, 2014, the research team implemented multi-modal education for nurses and physicians in the PICU. Education included a 4-pronged approach: independent online modules, one-on-one in-servicing with staff, group lectures, and visual aids/reminders. Staff completed an online learning module that described the RASS and included a posttest for completion. Each bedside nurse received individual education by a member of the research team with a written script that included how and when to document the RASS and reiterated the points demonstrated in the online module. Physicians attended educational conferences conducted by research team members. Visual aids included posters of the RASS in patient rooms proximal to the computer used for documentation, a copy of the RASS to attach to hospital badges, reminders throughout the unit and common staff areas to perform and document RASS assessments, and the addition of a “Goal RASS Score” to the daily goal sheet used during medical team rounds.

Back to Top | Article Outline

Postintervention Phase

From February 24, 2014, to March 31, 2014, paired observations were performed daily on mechanically ventilated patients receiving sedative or analgesic medications as detailed in the baseline phase. After completion, research team members shared the scores with participating staff and used the time for education and discussion.

Back to Top | Article Outline

Maintenance Phase

From May 4, 2014, to July 16, 2014, research team members and unit staff performed paired observation RASS assessments to evaluate the sustainability of the IRR.

Back to Top | Article Outline

Statistical Analysis

Normally distributed demographic characteristics were presented using means and SDs. Medians and interquartile ranges were used for non-normally distributed clinical variables. Weighted-κ statistics and 95% confidence intervals evaluated agreement between research staff and PICU staff. Based on standard guidelines for substantial agreement, the goal weighted-κ was greater than 0.7.

Back to Top | Article Outline

RESULTS

Data were collected on a total of 45 patients, some of whom were evaluated multiple times during their ICU hospitalization. Table 2 displays the characteristics of the patient population included. Over all phases of the study, 73 nurses and 6 physicians completed a total of 347 paired assessments. The 694 RASS scores obtained ranged from −5 to +3, with a median score of −3. Seventy-three percentage of RASS scores were in the sedated range (−5 to −1), 13% were 0, and 14% were in the agitated range (+1 to +3). Figure 1 displays the distribution of RASS scores.

Table 2

Table 2

Fig. 1

Fig. 1

Back to Top | Article Outline

Baseline Phase

During the baseline phase, 16 nurses and 2 physicians completed a total of 49 paired observations on 20 patients. IRR was lower than expected, with a weighted κ of 0.56 (95% confidence interval, 0.39–0.72; Table 3). Reliability was lowest for nurse-physician paired assessments with a weighted κ of 0.18 and highest for physician–physician paired assessments at 0.88.

Table 3

Table 3

Back to Top | Article Outline

Postintervention Phase

Over 90% of PICU staff completed the educational curriculum. During the postintervention phase, 36 nurses and 6 physicians completed a total of 228 paired observations on 15 patients. IRR improved significantly for all groups combined, with a weighted κ of 0.86 (95% confidence interval, 0.77–0.95; Table 3). The physician–physician weighted K decreased slightly (0.88–0.75).

Back to Top | Article Outline

Maintenance Phase

In the maintenance phase, 21 nurses and 2 physicians completed a total of 70 paired observations on 11 patients. IRR maintained above goal during this phase, with a κ of 0.78 for all pairs combined (95% confidence interval, 0.61–0.94). The physician–physician weighted κ decreased slightly but was still near goal at 0.67. Analysis of subgroups revealed that in patients less than 1 year of age, the IRR improved from a weighted κ of 0.41 before intervention to 0.87 after the intervention. Similarly, in patients with developmental delay, the IRR improved from a weighted κ of 0.49 before intervention to 0.84 after the intervention.

Back to Top | Article Outline

DISCUSSION

To our knowledge, this is the first investigation of an educational intervention to improve the IRR of the RASS in a pediatric population. The high weighted κ in the postintervention phase demonstrates the RASS to have the potential to be an extremely reliable tool for assessment of sedation status in mechanically ventilated, sedated PICU patients when a multi-modal educational curriculum is employed. Sustained IRR months after education and implementation ensure the feasibility of continued reliable use with time. Additionally, the IRR remained above goal despite an institutional transition to a new electronic medical record system between the postintervention and maintenance phases.

Previous studies demonstrate the utility of the RASS as an objective tool to assess sedation. However our experience did not reflect this as evidenced by poor reliability in the baseline phase despite the use of the RASS in daily sedation assessment by bedside nurses in the PICU for over 5 years. Several factors likely contributed. During the standard orientation, nurses received minimal training on how to correctly perform the RASS. Investigators noted the inconsistent use of the RASS by the medical team when describing sedation goals, varied nursing practice in the frequency of RASS assessment and documentation for mechanically ventilated patients, and difficulty in finding RASS documentation within the electronic medical record system. The implementation of a focused, multi-modal educational intervention by a multidisciplinary team was fundamental in improving the knowledge and utilization of the RASS with a subsequent significant improvement in IRR. Staff from multiple disciplines worked congruently to develop the various educational tools and to ensure each tool was appropriate for their colleagues. Multiple types of learning modules also accounted for different adult learning styles. One factor that was likely responsible for broad acceptance and utilization of the scale was the incorporation of after-evaluation feedback and debriefing sessions. These sessions identified opportunities for further education and discussion.

It is important that a sedation scale be reliable among differing intra- and inter-professional providers. Among nurse-to-nurse and nurse-to-physician pairs, there was a significant sustained improvement in RASS IRR after the educational intervention. However, the physician–physician pairs did not show similar improvement. Only the 2 physicians on the research team participated in the preintervention assessments, likely skewing the results and contributing to the high initial IRR compared with the postintervention results. Assessments in the postintervention and maintenance phases included both research and nonresearch physicians, thus contributing to slightly lower IRR scores. There were also only 2 physician–physician pairs in the maintenance phase assessments, potentially not demonstrating an accurate assessment of the IRR for this phase.

Importantly, the RASS can be a reliable tool in 2 unique patient populations frequently encountered in the PICU—children less than 1 year of age (weighted κ, 0.87) and developmentally delayed children (weighted κ, 0.84). The use of a slightly adapted version of the original RASS scale to include eye-opening, not just eye contact, for certain levels of sedation allowed the scale to be inclusive for patient populations in which the level of arousal is difficult to determine due to their developmental abilities.

There were several limitations to the study. First, the study evaluates use in a single institution. Also, several children required mechanical ventilation for a prolonged course during the postintervention phase. Thus, despite having over 200 RASS assessments, they were performed on a small number of patients. The timing of assessments on weekday shifts may have played a factor in the IRR. Many of the staff in the unit alternate between days and nights, but those working mainly nights represented a lower percentage of the RASS assessments obtained. Lastly, there was a paucity of assessments in the agitated range of the RASS. This result is largely due to the patient population included in the study as described previously. However, there were rare situations when research members were unable to complete RASS assessments on applicable patients due to too much activity in the room (eg, a patient undergoing a procedure or multiple visitors in the room).

Back to Top | Article Outline

CONCLUDING SUMMARY

The RASS is a reliable scale for assessing sedation in pediatric patients after introduction of a multi-modal educational curriculum. Without appropriate training and education, the RASS should not be presumed to be a reliable scale among users. Ongoing education at least annually is likely important to ensure maintenance of knowledge, especially in units with high staff turnover. In the future, the educational framework used can be applied to improve the education of other tools utilized in critically ill pediatric patients.

Back to Top | Article Outline

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

Back to Top | Article Outline

REFERENCES

1. Vet NJ, Ista E, de Wildt SN, et al. Optimal sedation in pediatric intensive care patients: a systematic review. Intensive Care Med. 2013;39:1524–1534.
2. Grant MJ, Balas MC, Curley MA; RESTORE Investigative Team. Defining sedation-related adverse events in the pediatric intensive care unit. Heart Lung. 2013;42:171–176.
3. Grant MJ, Scoppettuolo LA, Wypij D, et al; RESTORE Investigative Team. Prospective evaluation of sedation-related adverse events in pediatric patients ventilated for acute respiratory failure. Crit Care Med. 2012;40:1317–1323.
4. Fitzgerald RK, Davis AT, Hanson SJ; National Association of Children’s Hospitals and Related Institution PICU Focus Group Investigators. Multicenter analysis of the factors associated with unplanned extubation in the PICU. Pediatr Crit Care Med. 2015;16:e217–e223.
5. Lucas da Silva PS, de Carvalho WB. Unplanned extubation in pediatric critically ill patients: a systematic review and best practice recommendations. Pediatr Crit Care Med. 2010;11:287–294.
6. Payen V, Jouvet P, Lacroix J, et al. Risk factors associated with increased length of mechanical ventilation in children. Pediatr Crit Care Med. 2012;13:152–157.
7. Poh YN, Poh PF, Buang SN, et al. Sedation guidelines, protocols, and algorithms in PICUs: a systematic review. Pediatr Crit Care Med. 2014;15:885–892.
8. Playfor S, Jenkins I, Boyles C, et al; United Kingdom Paediatric Intensive Care Society Sedation; Analgesia and Neuromuscular Blockade Working Group. Consensus guidelines on sedation and analgesia in critically ill children. Intensive Care Med. 2006;32:1125–1136.
9. Jenkins IA, Playfor SD, Bevan C, et al. Current United Kingdom sedation practice in pediatric intensive care. Paediatr Anaesth. 2007;17:675–683.
10. Kudchadkar SR, Yaster M, Punjabi NM. Sedation, sleep promotion, and delirium screening practices in the care of mechanically ventilated children: a wake-up call for the pediatric critical care community. Crit Care Med. 2014;42:1592–1600.
11. Barr J, Fraser GL, Puntillo K, et al; American College of Critical Care Medicine. Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med. 2013;41:263–306.
12. Sessler CN, Gosnell MS, Grap MJ, et al. The Richmond Agitation-sedation Scale: validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med. 2002;166:1338–1344.
13. Ely EW, Truman B, Shintani A, et al. Monitoring sedation status over time in ICU patients: reliability and validity of the Richmond Agitation-sedation Scale (RASS). JAMA. 2003;289:2983–2991.
14. Pun BT, Gordon SM, Peterson JF, et al. Large-scale implementation of sedation and delirium monitoring in the intensive care unit: a report from two medical centers. Crit Care Med. 2005;33:1199–1205.
15. Yu A, Teitelbaum J, Scott J, et al. Evaluating pain, sedation, and delirium in the neurologically critically ill-feasibility and reliability of standardized tools: a multi-institutional study. Crit Care Med. 2013;41:2002–2007.
16. Nassar Junior AP, Pires Neto RC, de Figueiredo WB, et al. Validity, reliability and applicability of Portuguese versions of sedation-agitation scales among critically ill patients. Sao Paulo Med J. 2008;126:215–219.
17. Almgren M, Lundmark M, Samuelson K. The Richmond Agitation-sedation Scale: translation and reliability testing in a Swedish intensive care unit. Acta Anaesthesiol Scand. 2010;54:729–735.
18. Chanques G, Jaber S, Barbotte E, et al. [Validation of the French translated Richmond vigilance-agitation scale]. Ann Fr Anesth Reanim. 2006;25:696–701.
19. Kerson AG, DeMaria R, Mauer E, et al. Validity of the Richmond Agitation-sedation Scale (RASS) in critically ill children. J Intensive Care. 2016;4:65.
20. Smith HA, Boyd J, Fuchs DC, et al. Diagnosing delirium in critically ill children: validity and reliability of the Pediatric Confusion Assessment Method for the intensive care unit. Crit Care Med. 2011;39:150–157.
21. Traube C, Silver G, Kearney J, et al. Cornell Assessment of Pediatric Delirium: a valid, rapid, observational tool for screening delirium in the PICU. Crit Care Med. 2014;42:656–663.
22. Lawrence J, Alcock D, McGrath P, et al. The development of a tool to assess neonatal pain. Neonatal Netw. 1993;12:59–66.
23. Merkel SI, Voepel-Lewis T, Shayevitz JR, et al. The FLACC: a behavioral scale for scoring postoperative pain in young children. Pediatr Nurs. 1997;23:293–297.
24. Voepel-Lewis T, Merkel S, Tait AR, et al. The reliability and validity of the Face, Legs, Activity, Cry, Consolability observational tool as a measure of pain in children with cognitive impairment. Anesth Analg. 2002;95:1224–1229, table of contents.
25. Malviya S, Voepel-Lewis T, Burke C, et al. The revised FLACC observational pain tool: improved reliability and validity for pain assessment in children with cognitive impairment. Paediatr Anaesth. 2006;16:258–265.
Copyright © 2018 The Authors. Published by Wolters Kluwer Health, Inc. Health, Inc. All rights reserved.