Fleiss’ κ statistics with 95% CI were also used to describe the intrarater reliability for each rater (Table 4). The preference to use Fleiss’ κ over weighted κ were 2-fold: to eliminate the risk of falsely inflating our results, and to increase result reproducibility.
A κ score of 0 to 0.2 was deemed as poor, 0.21 to 0.4 as fair, 0.41 to 0.6 as good, 0.61 to 0.8 as very good, and 0.81 to 1.0 as almost perfect agreement.12,13 All calculations were performed in Stata version 14.1 (StataCorp, College Station, TX).
Rating 1 Interrater Reliability
The overall agreement for all raters on all MCD grades was very good (κ, 0.70; 95% CI, 0.61-0.80) (Table 2). The overall agreement varied among professional groups, ranging from κ 0.59 to 0.93. A difference in scores was noted between MCD I and II, compared with agreement among the other grades (Table 3). Overall agreement for MCD I and II was good (κ: 0.52, 0.47, respectively). Grading of MCD III, IV, and V AEs were near perfect (κ: 0.84, 0.97, 1.00, respectively).
Rating 2 Interrater Reliability
Reliability increased at rating 2, with very good overall agreement on MCD grades (κ, 0.75; 95% CI, 0.66-0.84) (Table 2). Grading of MCD I and II AEs slightly improved but remained good (κ: 0.57, 0.58, respectively) (Table 3). Grading of MCD III, IV, and V AEs were once again near perfect (κ: 0.90, 0.95, 1.00, respectively).
Overall intrarater reliability among raters was very good (average κ, 0.78). The highest level of agreement was seen among attending surgeons (κ, 0.80 to 1.00), orthopaedic clinical fellows (κ 0.71 to 0.90), and individuals with CP (κ, 0.61 to 0.93) (Table 4). The lowest level of agreement was observed among physical therapists (κ, 0.48 to 0.80).
Preventing surgical AEs is critically important to providing high-quality clinical care, minimizing harm, maximizing function, and containing health care costs.8,14,15 Historically, surgeons have reported short-term complications and longer term treatment outcomes that do not involve patients’ perceptions16–19 and is hindered by a lack of standardization and reproducibility.20 A classification system that is valid, reliable, and easy to use may permit transparent standardized reporting of AEs, improve the accuracy of audits, and lead to more objective understanding and comparison of surgical outcome studies in the literature. Use of such systems can promote early recognition of events that deviate from the normal postoperative course, to prevent a cascade effect that could cause permanent morbidity or mortality. Clear documentation and collection of data related to complications will also add information on the associated risks of surgery, guiding the shared or informed decision-making process with parents, caregivers, and young people with CP.7,8
This study shows a very good interrater and intrarater reliability of the MCD for lower limb surgery in children with CP. The system has good face validity and we have shown that it can be used reliably in a multidisciplinary team environment by surgeons, allied health and nursing professionals. Our data shows that a high level of agreement was consistent among orthopaedic attending surgeons and clinical fellows, which could reflect their experience and expertise in recognizing and managing AEs.
Raters had lower agreement with MCD grades I and II overall, and this may indicate some uncertainty in distinguishing typical postoperative sequelae from minor AEs. Subtle differences exist between a treatment considered “preexisting” compared with the addition of new treatment. We think that agreement between MCD grades I and II could be improved by a priori discussion, and the development of more detailed guidelines are now included in Appendix 1 (Supplemental Digital Content 1, http://links.lww.com/BPO/A170).
Near perfect agreement was achieved among MCD grades III, IV, and V, irrespective of professional background. This suggests that grading by treatments required to address the AE was clearly understood, and that the outcome worsened with increasing MCD grade.
Children who are most in need of major reconstructive lower limb surgery21–23 are often the most medically frail.8,18 MCD grades I and II events occurred in up to 60% of children in a recent prospective cohort study of hip surgery in nonambulant children with CP.7 The most common complications were constipation, cast-related or splint-related skin irritation, inadequate pain and spasm control from malfunctioning epidural or morphine infusions, and respiratory infections. Most were self-limiting or resolved with simple medical treatment.7 However, some children had multiple AEs and in other children, minor AEs escalated to more serious events. For example, children functioning at Gross Motor Function Classification System V have a high prevalence of preexisting respiratory disease. Attempts to manage postoperative pain and spasm with narcotic infusions and diazepam can lead to respiratory depression and pneumonia, which may require an intensive care unit admission and mechanical ventilation.7,8 DiFazio et al8 reported similar findings, in which 65% of children with CP suffered from postoperative AEs at a rate nearly twice that of typically developing children. This contrasts with a 10.5% surgical and a 29.8% medical AE reported in a large, retrospective cohort study of 168 hip reconstructions involving 121 children with CP.22 Constipation and inadequate analgesia were not reported, which may be due to the retrospective nature of the study.22
The difference between the reported rates of AEs in outcome studies of children with CP suggests that underreporting of AEs exists in the literature. This is an important issue that requires addressing. Surgeons readily accept that wound infections, hardware failure, or nonunion are surgical AEs that should be reported.6,8 As physicians who operate, surgeons should also take responsibility for reporting, preventing, and ameliorating “medical” AEs. If the child was not subjected to general anesthesia and the operation, the exacerbation of constipation, pain, hypertonia, aspiration, and chest infection would not have occurred. Medical and surgical AEs should be “owned,” prevented, and managed by all members of the clinical team. This is the principal reason that we involved all members of our multidisciplinary team in assessing the reliability of the MCD for CP, as well as the perspective of individuals with CP. We strongly encourage a proactive system of prospective documentation, to improve the transparency of recognizing and reporting events.
The key strengths of our study include the inclusion of participants from medical, nursing, and allied health professions, and of individuals with CP whom have experienced orthopaedic surgery. The diversity of our raters aimed to model holistic care provided by a multidisciplinary team, and included the perceptions of AEs in individuals with CP.
This study is the first to examine the reliability of the MCD in children with CP, and it has some limitations. First, raters were inexperienced in using the MCD, resulting in less than perfect agreement. Second, this study highlights the difficulty of distinguishing the severity of minor AEs (MCD I and II), with little comparative data in existing literature. With increasing utilization of this system in time, it is felt that reliability in grading minor AEs will also continue to improve, shown by the increased observed agreement from rating 1 to rating 2. Future reporting of all minor AEs should be encouraged in the literature to characterize the true frequency of events, identify risk factors and develop strategies for prevention in the postoperative period. Third, many of our complex care children experience more than 1 AE. The MCD system currently has no guidance on how to rate or summarize the effects of multiple complications.
In conclusion, this study has shown that the MCD is a system that can be utilized reliably for grading AEs in CP, following hip and lower limb surgery. The MCD is a useful tool that is easily understood and can be explained to any member of the multidisciplinary team. Future research will be directed on developing a system for grading multiple AEs and testing the validity of the MCD in lower limb surgery for CP in a real time, prospective clinical trial. We recommend the use of the MCD until there is consensus for a gold standard system in AE reporting.
1. Horton R. Surgical research or comic opera: questions, but few answers. Lancet. 1996;347:984–985.
2. Clavien P-A, Sanabria JR, Strasberg SM. Proposed classification of complications of surgery with examples of utility in cholecystectomy. Surgery. 1992;111:518–526.
3. Dindo D, Demartines N, Clavien P-A. Classification of surgical complications: a new proposal with evaluation in a cohort of 6336 patients and results of a survey. Ann Surg. 2004;240:205–213.
4. Clavien P-A, Barkun J, de Oliveira ML, et al. The Clavien-Dindo classification of surgical complications: five-year experience. Ann Surg. 2009;250:187–196.
5. Slankamenac K, Graf R, Puhan MA, et al. Perception of surgical complications among patients, nurs`es and physicians: a prospective cross-sectional survey. Patient Saf Surg. 2011;5:30–37.
6. Sink EL, Leunig M, Zaltz I, et al. Reliability of a complication
classification system for orthopaedic surgery
. Clin Orthop Relat Res. 2012;470:2220–2226.
7. Zhou L, Camp M, Gahukamble A, et al. Proximal femoral osteotomy in children with cerebral palsy
: the perspective of the trainee. J Child Orthop. 2017;11:6–14.
8. DiFazio R, Vessey JA, Miller P, et al. Postoperative complications after hip surgery in patients with cerebral palsy
: a retrospective matched cohort study. J Pediatr Orthop. 2016;36:56–62.
9. Dreher T, Thomason P, Svehlik M, et al. Long-term development of gait after multilevel surgery in children with cerebral palsy
: a multicentre cohort study. Dev Med Child Neurol. 2018;60:88–93.
10. Theologis T. Results of multilevel surgery in diplegic cerebral palsy
at skeletal maturity: new evidence. Dev Med Child Neurol. 2018;60:10–11.
11. Koch A, Jozwiak M, Idzior M, et al. Avascular necrosis as a complication
of the treatment of dislocation of the hip in children with cerebral palsy
. Bone Joint J. 2015;97-B:270–276.
12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174.
13. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85:257–268.
14. Krupka DC, Sandberg WS, Weeks WB. The impact on hospitals of reducing surgical complications suggests many will need shared savings programs with payers. Health Aff. 2012;31:2571–2578.
15. Shore BJ, Murphy RF, Hogue GD. Quality, safety, value: from theory to practice management what should we measure? J Pediatr Orthop. 2015;35:S61–S66.
16. Khalife R, Ghanem I, El Hage S, et al. Risk of recurrent dislocation and avascular necrosis after proximal femoral varus osteotomy in children with cerebral palsy
. J Pediatr Orthop B. 2010;19:32–37.
17. McNerney NP, Mubarak SJ, Wenger DR. One-stage correction of the dysplastic hip in cerebral palsy
with the San Diego acetabuloplasty: results and complications in 104 hips. J Pediatr Orthop. 2000;20:93–103.
18. Miller F, Girardi H, Lipton G, et al. Reconstruction of the dysplastic spastic hip with peri-ilial pelvic and femoral osteotomy followed by immediate mobilization. J Pediatr Orthop. 1997;17:592–602.
19. Stasikelis PJ, Lee DD, Sullivan CM. Complications of osteotomies in severe cerebral palsy
. J Pediatr Orthop. 1999;19:207–210.
20. Ergina PL, Cook JA, Blazeby JM, et al. Challenges in evaluating surgical innovation. Lancet. 2009;374:1097–1104.
21. Mallet C, Ilharreborde B, Presedo A, et al. One-stage hip reconstruction in children with cerebral palsy
: long-term results at skeletal maturity. J Child Orthop. 2014;8:221–228.
22. Rutz E, Vavken P, Camathias C, et al. Long-term results and outcome predictors in one-stage hip reconstruction in children with cerebral palsy
. J Bone Joint Surg Am. 2015;97:500–506.
23. Shore BJ, Zurakowski D, Dufreny C, et al. Proximal femoral varus derotation osteotomy in children with cerebral palsy
: the effect of age, gross motor function classification system level, and surgeon volume on surgical success. J Bone Joint Surg Am. 2015;97:2024–2031.
cerebral palsy; lower limb; orthopaedic surgery; adverse event; complication; modified Clavien-Dindo
Supplemental Digital Content
Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.