Background: Reliability adjustment, a novel technique for quantifying and removing statistical “noise” from quality rankings, is becoming more widely used outside surgery. We sought to evaluate its impact on hospital outcomes assessed with the American College of Surgeons' National Surgical Quality Improvement Program (ACS-NSQIP).
Methods: We used prospective, clinical data from the ACS-NSQIP to identify all patients undergoing colon resection in 2007 (n = 181 hospitals, n = 18,455 patients). We first used standard NSQIP techniques to generate risk-adjusted mortality and morbidity rates for each hospital. Using hierarchical logistic regression models, we then adjusted these for reliability using empirical Bayes techniques. To evaluate the impact of reliability adjustment, we first estimated the extent to which hospital-level variation was reduced. We then compared hospital mortality and morbidity rankings and outlier status before and after reliability adjustment.
Results: Reliability adjustment greatly diminished apparent variation in hospital outcomes. For risk-adjusted mortality, there was a 6-fold difference before (1.4%–7.8%) and less than a 2-fold difference (3.2% to 5.7%) after reliability adjustment. For risk-adjusted morbidity, there was a 2-fold difference (18.0%–38.2%) before and a 1.5-fold difference (20.8%–34.8%) after reliability adjustment. Reliability adjustment had a large impact on hospital mortality and morbidity rankings. For example, with rankings based on mortality, 44% (16 hospitals) of the “best” hospitals (top 20%) were reclassified after reliability adjustment. Similarly, 22% (8 hospitals) of the “worst” hospitals (bottom 20%) were reclassified after reliability adjustment.
Conclusions: Reliability adjustment reduces variation due to statistical noise and results in more accurate estimates of risk-adjusted hospital outcomes. Given the risk of misclassifying hospitals and surgeons using standard approaches, this technique should be considered when reporting surgical outcomes.
We evaluated the impact of reliability adjustment on hospital outcomes assessed with the American College of Surgeons' National Surgical Quality Improvement Program (ACS-NSQIP). We found that using reliability adjustment to account for statistical “noise” resulted in dramatic changes in apparent hospital performance for colectomy. This technique would be useful to avoid misjudging hospital quality.
*Center for Healthcare Outcomes and Policy (CHOP) and the Department of Surgery, University of Michigan, Ann Arbor, MI
†American College of Surgeons' Division of Research and Optimal Patient Care, Chicago, IL
‡Department of Surgery, Washington University, St. Louis, MO.
Reprints: Justin B. Dimick, MD, MPH, Center for Healthcare Outcomes and Policy (CHOP), 2800 Plymouth Road, Building 520, Office 3144, Ann Arbor, MI 48109. E-mail: email@example.com.
Disclosure: This study was supported by a career development award to Dr Dimick from the Agency for Healthcare Research and Quality (K08 HS017765) and a research grant to Dr Dimick from the National Institute of Diabetes and Digestive and Kidney Diseases (R21DK084397).