Share this article on:

Making Sense of Surgical Risk When the Data Aren’t Perfect

Lane-Fall, Meghan B. MD, MSc; Neuman, Mark D. MD, MSc

doi: 10.1213/ANE.0000000000000324
Editorials: Editorial

From the Department of Anesthesiology and Critical Care, University of Pennsylvania, Philadelphia, Pennsylvania.

Accepted for publication May 1, 2014.

Funding: This work was unfunded. Mark D. Neuman receives salary support from the National Institutes of Health (grant number 5K08AG043548).

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Meghan B. Lane-Fall, MD, MSc, Department of Anesthesiology and Critical Care, University of Pennsylvania, 423 Guardian Dr., 1121-B Blockley Hall, Philadelphia, PA 19104. Address e-mail to

Informed consent discussions, performance measurement, and studies of patient outcomes all require accurate information regarding the likelihood of a good or a bad outcome after surgery for an individual patient. Statistical risk models, typically based on multivariate analyses of retrospective data, aim to help clinicians quantify the probability of adverse outcomes for individual patients. Beginning with Goldman et al.’s 1977 landmark publication of a multivariate risk index for postoperative cardiac events,1 statistical risk models have been used for >3 decades in various surgical contexts prospectively to predict the likelihood of adverse outcomes for patients considering surgical interventions.2 Predictions derived from such models can then be used to help decide whether to have surgery (e.g., medical versus surgical management of hip fracture), what kind of surgery to have (e.g., conventional versus transcatheter aortic valve replacement), or where to have surgery (e.g., outpatient surgicenter versus hospital).

In predicting risk before surgery, one size does not necessarily fit all; different risk factors might be more or less important for different groups of patients. Perhaps for this reason, different approaches to predicting risk have emerged that are specific to certain surgical specialties and specific procedures. For example, the Society of Thoracic Surgeons (STS) risk score is used to predict operative mortality for patients undergoing cardiothoracic surgery.3 In addition to being a helpful research tool for comparing patient populations, the STS score has emerged as a decision point in determining the management approach for symptomatic aortic stenosis. For instance, in the Placement of AoRTic TraNscathetER Valve (PARTNER) trial, an STS score of ≥10% was used to define a high-risk surgical group.4 Only those patients at or above this cutoff were considered for transcatheter aortic valve replacement. Other surgical specialties using procedure-specific risk models include orthopedics5 and transplant surgery.6

In this issue of Anesthesia & Analgesia, Reponen et al. 7 review evidence on preoperative risk stratification for patients undergoing cranial neurosurgery. The authors compared commonly used risk scores by conducting a qualitative systematic review. From a starting sample of >2,000 papers, Reponen et al. 7 analyzed 25 published reports describing the outcomes after craniotomy as a function of 5 different scales: American Society of Anesthesiologists physical status classification (ASA; 10 studies), the Karnofsky Performance Score (KPS; 16 studies), the Charlson Comorbidity Score (CCS; 3 studies), the modified Rankin Scale (mRS; 2 studies), and the Sex, KPS, ASA classification, meningioma Location, and peritumoral Edema score (SKALE; 2 studies). The authors found evidence of some association among 3 of these scores and postoperative outcomes: ASA physical status was associated with mortality and with infectious complications including surgical site infection and postoperative meningitis. The KPS was associated with mortality, functional impairment, and systemic complications, including pneumonia, heart failure, and deep venous thrombosis. The CCS was associated with mortality in patients who had elective treatment of intracranial aneurysms and in elderly patients with intracranial tumors.7 As Reponen et al. 7 note, important differences in study design, patient populations, and outcome definitions across studies limit the conclusions that can be drawn from this review. The 25 studies they reviewed included retrospective and prospective designs, sample sizes ranging from 30 to almost 5000 patients, and varying definitions of operative and nonoperative complications. Indeed, the authors note that they were unable to perform a quantitative meta-analysis (as opposed to a narrative review) because of marked differences in the methods and outcomes used across studies.

The work of Reponen et al. 7 offers a useful starting place for further research and discussion regarding prediction of outcomes after surgery. Importantly, it highlights the importance of critically examining the individual variables contained within any risk model; ideally, a plausible physiologic mechanism should link the individual elements of a risk score to the outcome of interest. In terms of postoperative mortality, for example, increasing age and comorbidity burden might predict increased mortality. In other cases, the relationship may be less apparent. For example, the data suggest that patients with a higher ASA physical status classification may be more likely to experience postoperative infections. While this association may be supported by observations in the data, further research is likely to be required to tease out the specific mechanisms that link poor physical status to infectious outcomes and to develop effective interventions to decrease risk among future patients.

The review by Reponen et al. 7 also highlights challenges in incorporating variables into risk scores that rely on individual clinician judgment. Indeed, the Goldman Index, and statistical risk modeling as a whole, grew out of a concern that existing tools for patient assessment, such as the ASA physical status classification, were prone to subjectivity due to their reliance on individual clinician’s judgments. Yet, when it has been included in statistical risk models, the ASA physical status classification turns out to be a tremendously useful predictor of postoperative outcomes specifically because it takes advantage of physicians’ abilities to judge how sick an individual patient is, above and beyond the comorbidities that appear in a patient’s chart. For example, a recent review of heterogeneous patient populations found that, despite poor interrater reliability, ASA physical status was an accurate predictor of postsurgical mortality.8 Nonetheless, reliance on human judgment in scoring patient risk raises important concerns regarding the potential for risk assessments, such as the ASA physical status classification, to vary across raters and to be overly influenced by individual biases.

Of the 5 scores that Reponen et al. 7 reviewed, 4 had components that require some degree of judgment on the part of the assessing clinician. As such, the paper highlights opportunities for further research on opportunities to increase the reliability of such assessments across clinicians and settings and how such clinical assessments can potentially be incorporated into large-scale prospective data collection efforts. Such efforts are ongoing outside of anesthesia through the Veterans Affairs and American College of Surgeons’ National Surgical Quality Improvement Programs (NSQIP9 and ACS-NSQIP10) and within anesthesiology through the Anesthesia Quality Institute’s National Anesthesia Clinical Outcomes Registry11 and the University of Michigan’s Multicenter Perioperative Outcomes Group.12 Ultimately, data from these and other registries may carry potential not only for the development of additional risk prediction models for neurosurgery and other specialties but also to shed light on how the collection and data on specific risk predictors can be more effectively standardized going forward.

The work of Reponen et al. 7 makes it clear that further work is needed to develop reliable and robust approaches to predicting the risk for neurosurgical patients. In the meantime, however, there are patients and clinicians who need guidance as they consider therapeutic options. For patients and clinicians, the work of Reponen et al.7 confirms that worse functional status and greater degrees of comorbidity are likely to be associated with perioperative morbidity and mortality in craniotomy patients7 and that further work is still needed to identify the specific combinations of factors that most accurately predict poor outcomes. In the meantime, then, perhaps we should consider the risk scores reviewed by Reponen et al. 7 to be places to start in our conversations about operative risk among neurosurgical patients, but not the ultimate arbiters of therapeutic choices.

Back to Top | Article Outline


Name: Meghan B. Lane-Fall, MD, MSc.

Contribution: This author helped write the manuscript.

Attestation: Meghan B. Lane-Fall approved the final manuscript.

Name: Mark D. Neuman, MD, MSc.

Contribution: This author helped write the manuscript.

Attestation: Mark D. Neuman approved the final manuscript.

This manuscript was handled by: Gregory J. Crosby, MD.

Back to Top | Article Outline


1. Goldman L, Caldera DL, Nussbaum SR, Southwick FS, Krogstad D, Murray B, Burke DS, O’Malley TA, Goroll AH, Caplan CH, Nolan J, Carabello B, Slater EE. Multifactorial index of cardiac risk in noncardiac surgical procedures. N Engl J Med. 1977;297:845–50
2. Neuman MD, Bosk CL. What we talk about when we talk about risk: refining surgery’s hazards in medical thought. Milbank Q. 2012;90:135–59
3. Shroyer AL, Coombs LP, Peterson ED, Eiken MC, DeLong ER, Chen A, Ferguson TB Jr, Grover FL, Edwards FHSociety of Thoracic Surgeons. . The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models. Ann Thorac Surg. 2003;75:1856–64
4. Leon MB, Smith CR, Mack M, Miller DC, Moses JW, Svensson LG, Tuzcu EM, Webb JG, Fontana GP, Makkar RR, Brown DL, Block PC, Guyton RA, Pichard AD, Bavaria JE, Herrmann HC, Douglas PS, Petersen JL, Akin JJ, Anderson WN, Wang D, Pocock SPARTNER Trial Investigators. . Transcatheter aortic-valve implantation for aortic stenosis in patients who cannot undergo surgery. N Engl J Med. 2010;363:1597–607
5. Maxwell MJ, Moran CG, Moppett IK. Development and validation of a preoperative scoring system to predict 30 day mortality in patients undergoing hip fracture surgery. Br J Anaesth. 2008;101:511–7
6. Onaca NN, Levy MF, Sanchez EQ, Chinnakotla S, Fasola CG, Thomas MJ, Weinstein JS, Murray NG, Goldstein RM, Klintmalm GB. A correlation between the pretransplantation MELD score and mortality in the first two years after liver transplantation. Liver Transpl. 2003;9:117–23
7. Reponen E, Tuominen H, Korja M. Evidence for the use of preoperative risk assessment scores in elective cranial neurosurgery: a systematic review of the literature. Anesth Analg. 2014;;119::420–32
8. Moonesinghe SR, Mythen MG, Das P, Rowan KM, Grocott MP. Risk stratification tools for predicting morbidity and mortality in adult patients undergoing major surgery: qualitative systematic review. Anesthesiology. 2013;119:959–81
9. Khuri SF, Daley J, Henderson W, Hur K, Demakis J, Aust JB, Chong V, Fabri PJ, Gibbs JO, Grover F, Hammermeister K, Irvin G 3rd, McDonald G, Passaro E Jr, Phillips L, Scamman F, Spencer J, Stremple JF. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg. 1998;228:491–507
10. Fink AS, Campbell DA Jr, Mentzer RM Jr, Henderson WG, Daley J, Bannister J, Hur K, Khuri SF. The National Surgical Quality Improvement Program in non-veterans administration hospitals: initial demonstration of feasibility. Ann Surg. 2002;236:344–53
11. Dutton RP, Dukatz A. Quality improvement using automated data sources: the anesthesia quality institute. Anesthesiol Clin. 2011;29:439–54
12. Kheterpal S. Clinical research using an information system: the multicenter perioperative outcomes group. Anesthesiol Clin. 2011;29:377–88
© 2014 International Anesthesia Research Society