We sought to determine the reliability of surgeon-specific postoperative complication rates after colectomy.
Conventional measures of surgeon-specific performance fail to acknowledge variation attributed to statistical noise, risking unreliable assessment of quality.
We examined all patients who underwent segmental colectomy with anastomosis from 2008 through 2010 participating in the Michigan Surgical Quality Collaborative Colectomy Project. Surgeon-specific complication rates were risk-adjusted according to patient characteristics with multiple logistic regression. Hierarchical modeling techniques were used to determine the reliability of surgeon-specific risk-adjusted complication rates. We then adjusted these rates for reliability. To evaluate the extent to which surgeon-level variation was reduced, surgeons were placed into quartiles based on performance and complication rates were compared before and after reliability adjustment.
A total of 5033 patients (n = 345 surgeons) undergoing partial colectomy reported a risk-adjusted complication rate of 24.5%. Approximately 86% of the variability of complication rates across surgeons was explained by measurement noise, whereas the remaining 14% represented true signal. Risk-adjusted complication rates varied from 0% to 55.1% across quartiles before adjusting for reliability. Reliability adjustment greatly diminished this variation, generating a 1.2-fold difference (21.4%–25.6%). A caseload of 168 colectomies across 3 years was required to achieve a reliability of more than 0.7, which is considered a proficient level. Only 1 surgeon surpassed this volume threshold.
The vast majority of surgeons do not perform enough colectomies to generate a reliable surgeon-specific complication rate. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume, as statistical noise is a large determinant in estimating their surgeon-specific complication rates.
We evaluate clinical registry data from 5033 colectomy patients and 345 surgeons to demonstrate that surgeon-specific complication rates are not statistically reliable. Risk-adjusted complication rates should be viewed with caution when evaluating surgeons with low operative volume because statistical noise is a large determinant in estimating their complication rates.
*Department of Surgery, Academic Surgeon Development Program
†Michigan Surgery Quality Collaborative, University of Michigan, Ann Arbor, MI.
Reprints: Terry Shih, MD, Department of Surgery, University of Michigan Hospitals, 1500 E Medical Center Dr, Ann Arbor, MI 48109. E-mail: tyshih@umich.edu.
Disclosure: This study was supported by a grant to Dr Shih from the National Institutes of Health (5T32HL07612309). The views expressed herein do not necessarily represent the views of the US Government. The authors declare no conflicts of interest.
