Secondary Logo

Share this article on:

Physician Performance Rankings Based on Outcomes, Confirmed by the Same Outcomes: A Tautology

Myles, Paul S. MBBS, MPH, MD, FCAI, FANZCA, FRCA, FAHMS; Kasza, Jessica BSc, PhD

doi: 10.1213/ANE.0000000000001245
Letters to the Editor: Letter to the Editor

Department of Anaesthesia and Perioperative Medicine, Alfred Hospital and Monash University, Melbourne, Australia, p.myles@alfred.org.au

Department of Epidemiology and Preventive Medicine, Monash University, Melbourne, Australia

Back to Top | Article Outline

To the Editor

The recent article by Glance et al.1 suggests that anesthesiologist performance can be ranked according to rates of postoperative death and serious complications because such rates vary across anesthesiologists. This could lead to a misconception that this finding indicates how to improve perioperative management of high-risk surgical patients. These propositions were supported by 4 accompanying editorials.2–5 We believe such interpretations of the data are flawed in logic and practice.

Glance et al. analyzed the New York State Cardiac Surgery Reporting System, using clinical data from 7920 patients undergoing isolated coronary artery bypass graft surgery. Multivariable regression modeling was used to adjust for risk and other potential confounders, leading to a “performance” ranking based on patient outcome, in which patients managed by “low-performance” anesthesiologists experienced nearly twice the rate of death or serious complications as patients managed by “high-performance” anesthesiologists.

This is a tautological argument. If groups of anesthesiologists (or surgeons, or hospitals) are classified by the rate of adverse outcomes, then this classification will always be shown to predict adverse outcomes. This can be illustrated with a simple thought experiment: if a single anesthesiologist maintained steady, high-performing delivery of anesthesia care throughout the year, and then retrospectively ranked their annual caseload according to outcomes, he/she could then compare his/her patient outcomes in the lowest and highest quartiles. They will differ—not because the anesthesiologist was underperforming but because this merely represents the perioperative reality of variable risk status and multidisciplinary care of such a caseload.

Furthermore, performance comparisons based purely on ranks has been criticized because it does not indicate whether performance is actually poor because it is solely a relative comparison.6 Ohlssen et al.6 point out that with ranking, “one provider must be bottom (or top), not simply because of chance variation but because the null hypothesis that everyone is precisely the same rate is implausible.” As such, it may well be because of chance rather than performance that an anesthesiologist ends up in the bottom quartile of usual performers,7 and in this situation, it makes little sense to compare the performance of the top and bottom quartiles of usual performers.

In the caption of Figure 1, Glance et al. state that anesthesiologists with adjusted odds ratios significantly different from 1 are considered to be performance outliers. That the majority of anesthesiologists are thus considered to have outlying performance “appears a contradiction in terms,” as stated by Spiegelhalter8 in a similar situation. This points to excess variability or “overdispersion” in the adjusted odds ratios, which calls the appropriateness of Glance et al.s model for assessing anesthesiologist performance into question.9

Physician and hospital performance are extremely important issues in clinical medicine. Although it is very likely that there are high and low performers in anesthesiological practice, the methods used to identify and correct such practices require robust methods that adequately describe “usual” performance.

Paul S. Myles, MBBS, MPH, MD, FCAI, FANZCA, FRCA, FAHMS

Department of Anaesthesia and Perioperative Medicine

Alfred Hospital and Monash University

Melbourne, Australia

p.myles@alfred.org.au

Jessica Kasza, BSc, PhD

Department of Epidemiology and Preventive Medicine

Monash University

Melbourne, Australia

Back to Top | Article Outline

REFERENCES

1. Glance LG, Kellermann AL, Hannan EL, Fleisher LA, Eaton MP, Dutton RP, Lustik SJ, Li Y, Dick AW. The impact of anesthesiologists on coronary artery bypass graft surgery outcomes. Anesth Analg. 2015;120:526–33
2. Shafer SL. Anesthesiologists make a difference. Anesth Analg. 2015;120:497–8
3. Maxwell BG, Hogue CW Jr, Pronovost PJ. Does it matter who the anesthesiologist is for my heart surgery? Anesth Analg. 2015;120:499–501
4. Leslie K, Merry AF. Cardiac surgery: all for one and one for all. Anesth Analg. 2015;120:504–6
5. Wijeysundera DN, Beattie WS. Facing the uncomfortable truth: your choice of anesthesiologist does matter. Anesth Analg. 2015;120:502–3
6. Ohlssen DI, Sharples LD, Spiegelhalter DJ. A hierarchical modelling framework for identifying unusual performance in health care providers. J R Statist Soc A. 2007;170:865–90
7. Siregar S, Groenwold RH, Jansen EK, Bots ML, van der Graaf Y, van Herwerden LA. Limitations of ranking lists based on cardiac surgery mortality rates. Circ Cardiovasc Qual Outcomes. 2012;5:403–9
8. Spiegelhalter DJ. Handling over-dispersion of performance indicators. Qual Saf Health Care. 2005;14:347–51
9. Spiegelhalter DJ. Funnel plots for comparing institutional performance. Stat Med. 2005;24:1185–202
© 2016 International Anesthesia Research Society