Institutional members access full text with Ovid®

The Relationship Between the C-Statistic of a Risk-adjustment Model and the Accuracy of Hospital Report Cards: A Monte Carlo Study

Austin, Peter C. PhD*,†,‡; Reeves, Mathew J. PhD§

doi: 10.1097/MLR.0b013e31827ff0dc
Original Articles

Background: Hospital report cards, in which outcomes following the provision of medical or surgical care are compared across health care providers, are being published with increasing frequency. Essential to the production of these reports is risk-adjustment, which allows investigators to account for differences in the distribution of patient illness severity across different hospitals. Logistic regression models are frequently used for risk adjustment in hospital report cards. Many applied researchers use the c-statistic (equivalent to the area under the receiver operating characteristic curve) of the logistic regression model as a measure of the credibility and accuracy of hospital report cards.

Objectives: To determine the relationship between the c-statistic of a risk-adjustment model and the accuracy of hospital report cards.

Research Design: Monte Carlo simulations were used to examine this issue. We examined the influence of 3 factors on the accuracy of hospital report cards: the c-statistic of the logistic regression model used for risk adjustment, the number of hospitals, and the number of patients treated at each hospital. The parameters used to generate the simulated datasets came from analyses of patients hospitalized with a diagnosis of acute myocardial infarction in Ontario, Canada.

Results: The c-statistic of the risk-adjustment model had, at most, a very modest impact on the accuracy of hospital report cards, whereas the number of patients treated at each hospital had a much greater impact.

Conclusions: The c-statistic of a risk-adjustment model should not be used to assess the accuracy of a hospital report card.

*Institute for Clinical Evaluative Sciences

Institute of Health Management, Policy and Evaluation

Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

§Department of Epidemiology and Biostatistics, Michigan State University, East Lansing, MI

The Institute for Clinical Evaluative Sciences is supported in part by a grant from the Ontario Ministry of Health and Long Term Care. The opinions, results, and conclusions are those of the authors and no endorsement by the Ministry of Health and Long Term Care or by the Institute for Clinical Evaluative Sciences is intended or should be inferred. Supported by operating Grant from the Canadian Institutes of Health Research (MOP 86508). P.C.A. is supported in part by a Career Investigator Award from the Heart and Stroke Foundation.

The authors declare no conflict of interest.

Reprints: Peter C. Austin, PhD, Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Avenue, Toronto, ON M4N 3M5, Canada. E-mail: peter.austin@ices.on.ca.

© 2013 Lippincott Williams & Wilkins, Inc.