Reliability of Surgeon-specific Reporting of Complications After Colectomy

Abstract

Objective: 

We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy.

Background: 

Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality.

Methods: 

We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment.

Results: 

A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%–25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold.

Conclusions: 

The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.

In Brief

We evaluate clinical registry data from 5033 colectomy patients and 345 surgeons to demonstrate that surgeon-specific complication rates are not statistically reliable. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume because statistical noise is a large determinant in estimating their complication rates.