Reliability of a Complication Classification System for Orthopaedic Surgery

Abstract

Background 

Quality of health care and safety have been emphasized by various professional and governmental groups. However, no standardized method exists for grading and reporting complications in orthopaedic surgery. Conclusions regarding outcomes are incomplete without a standardized, objective complication grading scheme applied concurrently. The general surgery literature has the Clavien-Dindo classification that meets the above criteria.

Questions/purposes 

We asked whether a previously reported classification would show high intraobserver and interobserver reliabilities when modified for orthopaedic surgery specifically looking at hip preservation surgery. We therefore determined the interreader and intrareader reliabilities of the adapted classification scheme as applied to hip preservation surgery.

Methods 

We adapted the validated Clavien-Dindo complication classification system and tested its reliability for orthopaedic surgery, specifically hip preservation surgery. There are five grades based on the treatment required to manage the complication and the potential for long-term morbidity. Forty-four complication scenarios were created from a prospective multicenter database of hip preservation procedures and from the literature. Ten readers who perform hip surgery at eight centers in three countries graded the scenarios at two different times. Fleiss’ and Cohen’s κ statistics were performed for interobserver and intraobserver reliabilities, respectively.

Results 

The overall Fleiss’ κ value for interobserver reliability was 0.887 (95% CI, 0.855-0.891). The weighted κ was 0.925 (95% CI, 0.894-0.956) for Grade I, 0.838 (95% CI, 0.807-0.869) for Grade II, 0.87 (95% CI, 0.835-0.866) for Grade III, and 0.898 (95% CI, 0.866-0.929) for Grade IV. The Cohen’s κ value for intraobserver reliability was 0.891 (95% CI, 0.857-0.925).

Conclusions 

The adapted classification system shows high interobserver and intraobserver reliabilities for grading of complications when applied to orthopaedic surgery looking at complications of hip preservation surgery. This grading scheme may facilitate standardization of complication reporting and make outcome studies more comparable.

Introduction

There is an emphasis in orthopaedic surgery to provide quality health care through evidence-based medicine. The transparent and objective assessment of quality is increasingly demanded by patients, government agencies, payers, and healthcare providers [⁴]. Objective and reliable outcome reporting is imperative to assess the quality, effectiveness, and comparative value of different surgical interventions. Clinical outcome studies pertaining to hip preservation surgery (arthroscopy, surgical dislocation, osteotomies) report data that commonly include hip function, overall health, quality of life, and activity scores [^{1, 2, 13, 20, 21}]. Although recognized as a critical component of evidence-based outcome studies, a review of randomized controlled studies concluded that there was a lack of clear definitions of complications and classification of complications [¹⁵]. Further, no standardized method exists for complication grading and reporting in hip surgery outcome studies [⁷]. Conclusions regarding outcomes are incomplete without a standardized, objective complication grading scheme applied concurrently. In turn, the lack of an established basis for grading and reporting complications renders comparisons among studies ineffective.

Although reported data from short- to medium-term single-institution retrospective analyses indicate improvement in quality of life, pain, and function for most patients after hip preservation surgery [^{1, 16, 21}], a review of the literature on outcomes of surgery for acetabular dysplasia and femoroacetabular impingement (FAI) suggests the need for validated standardized outcome measures and a standardized measure of complications associated with surgical treatment [^{6, 7}]. A validated complication grading scheme for hip preservation surgery that can be used universally will allow easier interpretation of the literature and true comparisons of the risks of different hip procedures. For the purposes of reporting in a standardized manner, a complication grading scheme should be simple, objective, and reproducible, allowing orthopaedic surgeons to use it nationally and internationally. A well-defined grading scheme reduces the amount of interpretation by individual surgeons, ie, by eliminating classification terms such as minor, moderate, and severe, which often appear in the literature and are used inconsistently among studies owing to the subjective nature of the terminology. Moreover, it is essential to account for unplanned treatments necessary for management of a complication and the association with future morbidity or persistent disability.

The classification system of Clavien et al. was originally tested using 650 cholecystectomy cases [⁵] and contained four grades of increasing severity. In 2004, Dindo et al. proposed a modification that included a new five-grade classification system with the possibility of combining grades; “disability” was added to the system, “hospital stay” was eliminated as a complication and “organ failure” was changed to carry more weight in grading [¹²]. The so-called “Clavien-Dindo” classification has been adapted for use in general surgery [¹⁰] and for other surgical subspecialties such as gastroenterology/hepatology [^{3, 11, 17, 22-25, 28}], urology [¹⁹], and nephrology [^{18, 27}]. Given this system appears useful for general surgery and other specialties, we explored its use in orthopaedic surgery.

We therefore determined the interreader and intrareader reliabilities of the adapted classification scheme as applied to orthopaedic surgery, specifically hip preservation surgery.

Patients and Methods

The classification system of Dindo et al. [¹²] (Table 1) was adapted for application to orthopaedic surgery (Table 2). Our adapted scheme consists of five grades based on (1) the treatment required to manage the complication and (2) the associated long-term morbidity. As delineated in Table 2, complications requiring no change in the routine postoperative course are classified as Grade I, whereas those necessitating a change in outpatient management fall under Grade II. Complications requiring invasive surgical or radiographic management without any long-term morbidity are defined as Grade III. Grade IV has long-term morbidity or a life-threatening complication, and Grade V is death. For the purposes of illustration, a Grade IV complication after abdominal surgery indicates a complication that is life threatening, requires intensive care unit admission, or is associated with organ dysfunction. Our orthopaedic adaptation for a Grade IV surgical complication is defined as any intensive care unit admission, osteonecrosis of the femoral head or acetabulum, permanent nerve injury, major vascular injury, or pulmonary embolus, each representing the orthopaedic equivalent of organ dysfunction.

Table 1: Original classification of surgical complications [¹²]

Table 2: Adapted complication classification chart

To evaluate the reliability of our adapted system we identified 10 readers to evaluate 44 clinical scenarios related to hip preservation surgery. The 10 surgeons were from a multicenter study group with diverse backgrounds (fellowship training in sports medicine, joint arthroplasty, and pediatric orthopaedics), all of whom have an interest in hip disorders in the adolescent and young adult population. Training readers in the classification system was done at an in-person meeting, during which the adapted scheme was described to eight of the 10 readers. Two international readers were not present for the training meeting; however, a brief description of the study had been provided to them by e-mail. The training session consisted of a description of the classification scheme, followed by 10 sample scenarios (not included in the 44 study scenarios), which were graded and discussed; all questions were answered. Subsequently, an e-mail was sent to all 10 readers containing the classification scheme and the previously graded sample scenarios. In addition, the e-mail message contained a recapitulation of the guidelines described during the training session to focus on grading the clinical scenario using the definition, concentrating on the treatment required to manage the complication and the potential for long-term morbidity. This correspondence acted as the only formal training for the two international readers who did not attend the in-person meeting.

Clinical scenarios were derived from a prospective multicenter outcomes database with approximately 500 surgical procedures (at the time of scenario development) currently used for open and arthroscopic hip preservation surgery performed to treat hip dysplasia or impingement syndromes, from which we included all cases (n = 40) in the database with recorded complications. From these 40 cases, we found there was an underrepresentation of Grade IV cases. Therefore, an additional four scenarios were randomly selected from case reports in the literature to provide an adequate number of Grade IV complications for this analysis that more accurately represents the distribution of surgical cases in the hip preservation registry. These cases were similar to those in the multicenter database: hip preservation surgery (arthroscopy, surgical dislocation, osteotomies) associated with postoperative complications. Thus, each grade of complication was equally represented (11 scenarios for each of four grades; death [Grade V] was excluded as there were no cases in the database). Two of us (ELS, JC) summarized the 44 study cases to include age, sex, type of hip preservation surgery, complication presentation and course, type of management, and long-term outcome (Appendix 1; supplemental materials are available with the online version of CORR.)

The scenarios were sent to the readers by electronic format with a designated space for grading at the end of the specific scenario; all readers had access to the classification scheme while grading. The readers sent their graded scenarios to the coordinating center for evaluation by the first and senior authors (ELS, JC), who used a grading key that was created by the senior authors and recorded errors on a spreadsheet. Two weeks later, the same scenarios were sent again to the readers in a different order randomly performed and graded in the same manner. The graded data were collected and analyzed for reliability across the different readers for the same scenario and for individual readers’ accuracy across scenarios. The interobserver agreements were assessed for each grade of the complication.

We evaluated initial analysis of patient characteristics using means and SDs for continuous variables and frequencies and percentages for categorical variables. We then calculated interrater and intrarater correlations using Fleiss’ and Cohen’s κ statistics, respectively. To assess the precision of the κ statistics, we calculated 95% CIs for each correlation. Statistical analyses were conducted using SAS^® 9.2 (SAS Institute Inc, Cary, NC, USA).

We determined sample size to provide adequate variability to assess discrimination among complication grades and acceptably precise reliability estimates. Based on a simulation study of 5000 random samples, when the sample consisted of 44 subjects, each being assessed by nine raters on a four-category classification system, there would be a greater than 95% chance to reject the null hypothesis that Fleiss’ κ is less than 0.7, if the true Fleiss’ κ is 0.9. Chance-adjusted Fleiss’ and Cohen’s κ statistics with their 95% CIs were used to determine interobserver and intraobserver reliabilities, respectively [^{8, 14}].

Results

Differences in grading the scenarios were evenly distributed among the grades. The reliability of the two international graders was no different from that of the graders who attended the training meeting in which the classification system was first introduced.

The Fleiss’ κ values for the first reading were 0.909 (95% CI, 0.865-0.953) for Grade I, 0.85 (95% CI, 0.806-0.894) for Grade II, 0.87 (95% CI, 0.826-0.914) for Grade III, and 0.918 (95% CI, 0.874-0.962) for Grade IV (Table 3). Therefore, Grade II was the least well defined. The overall mean κ for the first reading was 0.886 (95% CI, 0.861-0.912). For the second reading, Fleiss’ κ values were 0.941 (95% CI, 0.897-0.985) for Grade I, 0.827 (95% CI, 0.783-0.871) for Grade II, 0.801 (95% CI, 0.757-0.845) for Grade III, and 0.876 (95% CI, 0.832-0.920) for Grade IV (Table 3). The overall mean κ for the second reading was 0.859 (95% CI, 0.833-0.884).

Table 3: Fleiss’ κ for interrater reliability

The results of the second reading were similar to the results of the first reading in terms of the range of observed κ values and means. In the first reading, Grades II and III had a slightly lower agreement rate with kappa chance adjusted agreements of 85% and 87% respectively versus 90.9% for Grade I and 91.8% for Grade IV in Reading 1. In the second reading, Grades II and III had kappa chance adjusted agreements of 82.7% and 80.1% versus 94.1% and 87.6% for Grades I and IV. Intraobserver testing yielded an overall Cohen’s κ value of 0.891 (95% CI, 0.857-0.925) (Table 4). Intraobserver testing yielded an overall Cohen’s κ value of 0.891 (95% CI, 0.857-0.925) (Table 4), indicating excellent repeatability.

Table 4: Cohen’s κ for intrarater reliability

Discussion

A standardized and reliable classification-grading scheme will allow complications to be assessed in an objective manner and allow complications to be compared among different outcome studies. The classification should emphasize the impact of the complication to the patient, the healthcare system, and the potential long-term morbidity. We adapted a validated general surgery grading system developed by Clavien et al. [⁵] for abdominal surgical complications, which relies on the magnitude of treatment needed to manage a complication and the potential for associated long-term morbidity [^{4, 12}]. In the original publication of this classification, Clavien et al. stated “lack of uniform reporting of negative outcomes makes interpretation of surgical literature difficult” [⁵], a statement particularly relevant to current orthopaedic outcome studies. Hip preservation surgery is a growing field in orthopaedics, yet surgeons have no universal method of measuring clinical outcome and reporting complications. Recent studies assert the reporting of complications is not homogeneous, well defined, or standardized in many orthopaedic studies [^{6, 7, 15}]. Terms such as minor, moderate, and severe have been used, but they are unreliable, subjective, and often are defined separately by each author [^{7, 9}]. Standardized complication reporting is critical for future preservation surgery outcome analysis so that data from different centers may be clearly compared, and risks involved with specific surgical preservation techniques may be objectively evaluated. We therefore determined the interreader and intrareader reliabilities of the adapted classification scheme as applied to orthopaedic surgery, specifically hip preservation surgery. Our adapted classification system had high interobserver and intraobserver reliabilities for grading of complications.

There are certain limitations to this study. First, inexperience using the adapted classification scheme likely accounts for agreement being less than 100%. Nevertheless, we found high reliability, and we presume continued use of this grading system would be associated with improved reliability with time. Second, certain complications change with time, as in the case of osteonecrosis, which may take months to develop; thus, its grade will change to accommodate the appropriate level of management. A potential change in grade with time is not unique to this classification system. Additionally, differences in complication management for procedures such as hematoma evacuation or intensive care unit admission may vary among institutions, resulting in a discrepancy in grading a specific complication. In addition, there are various complications that would be given a specific grade and not all of them are similar. For example, a procedure to evacuate a hematoma may be considered not as severe to a surgeon as a procedure to treat an infection. Yet, if the end result of both procedures is that the problem was treated and there is no long-term morbidity, it is classified as Grade III. The subjective argument regarding which complication is more severe is not part of the final outcome of the patient and the eventual grade for this classification. Despite these possible variances in grading, the adapted grading scheme is objective at its base and is a major improvement for standardized reporting of complications associated with hip preservation surgery. Third, in view of the fact that the readers are from the same study group, there may be bias in the data. However, we had 10 observers representing three countries (Canada, United States, and Switzerland), indicating a strong reliability in slightly different medical cultures. Many of the complications (infection, nonunion, deep venous thrombosis) were not necessarily specific to hip preservation surgery. This may be an indication for the use and testing of the adapted classification scheme in future studies of complications in other orthopaedic procedures. The reliability obtained with this investigation should be directly applicable to hip preservation surgery, as the scenarios represented actual surgical cases, but we cannot say whether the reliability would apply to other scenarios, specialties, or surgeons. Finally, there may be differences in the definition of a complication as described in the next paragraph.

In their original classification, Clavien et al. [⁵] first defined the term complication. When used in conjunction with other outcome parameters such as function, we believe there should be a distinction in what is defined as a complication and what is defined as a failure of treatment. The definition of a complication for this classification is any deviation from the normal postoperative course not inherent in the procedure. As espoused by Dindo et al. [¹²], a complication is different from a sequela, which is inherent to the procedure and inevitable (such as scar tissue), or failure to cure, which occurs when the original goal of the surgery has not been met (ie, continued hip pain after arthroscopy). Further, they argue sequelae and failure to cure are not part of a surgical complication classification [¹²]. We agree with the concept that sequelae and failure to cure are better-assessed using outcome measures rather than reported as complications. Outcome measures other than this complication classification will better define which indications and techniques are successful for hip preservation surgery to decrease failures of treatment. Poor patient selection for surgery, such as hip preservation in a patient with advanced osteoarthritis, would be evaluated by outcomes rather than by this complication classification. Used in conjunction with this complication classification the risk of the procedure versus the outcome can be assessed. If a complication of the surgical procedure (such as avascular necrosis) results in long-term morbidity this would be included in the complication classification. For avascular necrosis this would be classified as Grade IV with long-term morbidity resulting from the surgery.

In the original validation study, the Clavien-Dindo classification was tested in a cohort of 6336 patients who underwent elective general surgery [¹²]. They found strong correlations between length of hospital stay, surgical complexity, and classification system grades. Reproducibility of the classification scheme also was assessed by using clinical scenarios that were mailed to international surgeons. Analysis of the clinical scenarios showed the classification system to be reproducible, simple, and useful [¹²]. In a review of the 5-year experience with this classification, there were 161 citations that used the classification system to grade complications between 2004 and March 2009. These were all in general surgical subspecialties, including gynecology [⁴]. DeOliveira et al. [¹⁰] adapted the Clavien-Dindo classification system for complications associated with pancreatic surgery; they concluded the adaptation provided an objective and reproducible assessment that could enable comparisons among centers. An adapted scheme also was used in a recent orthopaedic study evaluating complications after surgical hip dislocation procedures [²⁶].

Based on our study results, we conclude the adapted Clavien-Dindo classification scheme for reported complications associated with hip preservation surgery has high interobserver and intraobserver reliabilities. Our adapted classification scheme endeavors to facilitate standardization of complication reporting and may be useful for all orthopaedic procedures. Additionally, it is applicable to future outcomes analysis of hip preservation surgery because of (1) any long-term morbidity from the complication, which is critical to the risk/benefit analysis of the procedure; and (2) the magnitude of treatment (inpatient, outpatient, reoperation) required to manage the complication, which is important to the overall cost to the patient, third-party insurer, and hospital.

Acknowledgments

We thank Amy Monreal BS, Joseph T. Nguyen MPH, and Pan Zhaoxing PhD for assistance with preparation of this manuscript.