Predictive Performance and Variability of the Cardiac Anesthesia Risk Evaluation Score
Ouattara, Alexandre M.D.*; Niculescu, Michaëla M.D.†; Ghazouani, Sarra M.D.†; Babolian, Ario M.D.†; Landi, Marc M.D.*; Lecomte, Patrick M.D.†; Boccara, Gilles M.D., Ph.D.*; Varnous, Sheida M.D.‡; Leprince, Pascal M.D.‡; Riou, Bruno M.D., Ph.D.§; Coriat, Pierre M.D.‖
Background: The Cardiac Anesthesia Risk Evaluation (CARE) score, a simple Canadian classification for predicting outcome after cardiac surgery, was evaluated in 556 consecutive patients in Paris, France. The authors compared its performance to those of two multifactorial risk indexes (European System for Cardiac Operative Risk Evaluation [EuroSCORE] and Tu score) and tested its variability between groups of physicians (anesthesiologists, surgeons, and cardiologists).
Methods: Each patient was simultaneously assessed using the three scores by an attending anesthesiologist in the immediate preoperative period. In a blinded study, the CARE score category was also determined by a cardiologist the day before surgery, by a surgeon in the operating room, and by a second anesthesiologist at arrival in intensive care unit. Calibration and discrimination for predicting outcomes were assessed by goodness-of-fit test and area under the receiver operating characteristic curve, respectively. The level of agreement of the CARE scoring between the three physicians was then assessed.
Results: The calibration analysis revealed no significant difference between expected and observed outcomes for the three classifications. The areas under the receiver operating characteristic curves for mortality were 0.77 with the CARE score, 0.78 with the EuroSCORE, and 0.73 with the Tu score (not significant). The agreement rate of the CARE scoring between two anesthesiologists, between anesthesiologists and surgeons, and between anesthesiologists and cardiologists were 90%, 83%, and 77%, respectively.
Conclusions: Despite its simplicity, the CARE score predicts mortality and major morbidity as well the EuroSCORE. In addition, it remains devoid of significant variability when used by groups of physicians of different specialties.
NUMEROUS multifactorial risk indexes have been constructed to predict outcome after cardiac surgery. 1–10
Despite their usefulness, their relative complexity may explain why few physicians use these scores in clinical practice. Recently, Dupuis et al.11
have described the Cardiac Anesthesia Risk Evaluation (CARE) score, a simple intuitive risk ranking system which is similar to the American Society of Anesthesiologists (ASA) physical status classification. These authors 11
reported reliable performances of this score in predicting mortality and major morbidity with a discrimination and a calibration comparable with those obtained by three popular multifactorial risk indexes. 1,2,5
However, because these interesting findings result only from a single institution, more evaluations are required to confirm the apparently good predictive accuracy of this new classification. In addition, the predictive performance of this simple stratification was not compared with the most recent multi-factorial risk index, the European System for Cardiac Operative Risk Evaluation (EuroSCORE). 9
Therefore, the main objectives of the current study were (1) to determine the predictive performance of the CARE score for predicting mortality and major postoperative morbidity in cardiac surgical adult patients from a center other than where it was initially developed; (2) to compare its predictive performance with those of two multifactorial risk indexes (the EuroSCORE 9
and the Tu score 5
); and (3) to evaluate variability of CARE scoring among cardiac anesthesiologists, surgeons, and cardiologists.
Materials and Methods
This prospective study was conducted at the Institute of Cardiology in the Pitié-Salpêtrière Hospital, Paris, France, from January 2002 to May 2002, and was approved by our local institutional medical ethics committee. Because data were collected while care of patients conformed to standard procedures currently used in our institute, authorization was granted to waive informed consent for the study. During a 16-week period, all adult patients undergoing cardiac surgical procedures with or without cardiopulmonary bypass at our institute were enrolled. We excluded patients scheduled to undergo cardiac transplantation, implantation of ventricular assistance devices, and peripheral vascular surgery. Patients with congenital heart disease were not included in this study.
Scoring of Patients
In the immediate preoperative period, the CARE score risk category of each patient was determined by an attending anesthesiologist (the definitions of each category are summarized in table 1
). Simultaneously, the Tu score and the EuroSCORE were calculated by the same attending anesthesiologist (the components of these two multifactorial risk indexes are summarized in table 2
). Because the CARE score is similar to the ASA physical status scale, a model familiar to surgeons, a CARE score category was also determined by the surgeon in the operating room. All CARE score stratifications were only performed by senior physicians. The day before the surgery, an attending cardiologist also determined the CARE score category. The intraoperative care was left to the discretion of both the surgeon and the anesthesiologist. At the arrival in the cardiac intensive care unit, a new CARE score category was given by a second anesthesiologist in charge of the patient. After each assessment of the CARE score by a physician, the value was enclosed in an envelope. Consequently, each physician was unaware of the result previously assessed. The results of the scoring by the different physicians were only analyzed when the patient left the hospital or died. We evaluated the variability of the CARE score between two anesthesiologists, between anesthesiologists and surgeons in the immediate preoperative period, and between anesthesiologists and cardiologists in the preoperative period. To eliminate small differences of the CARE score assessment with the fewest clinical implications, we evaluated the variability of the score between different specialist groups with the score divided into only four categories: a low-risk group (CARE 1), an intermediate-risk group (CARE 2 and 3), a high-risk group (CARE 4), and a very-high-risk group (CARE 5).
The two main outcomes of our study were the inhospital mortality and major postoperative morbidity. In-hospital mortality
was defined as a death occurring during the hospital stay. Major morbidity
was defined using criteria previously reported by Dupuis et al.11
The performance of the CARE score for predicting in-hospital mortality and major morbidity in our population was assessed by determining the calibration and discrimination of the score. The calibration was assessed using the Pearson chi-square goodness-of-fit test, 12
which compared the predicted outcomes (in-hospital mortality and severe morbidity) with those observed in the population studied. Expected outcome values for three classifications were calculated from the reference group of the studies of Nashef et al.9
and Dupuis et al.11
Because the EuroSCORE was defined only as a multifactorial risk index for predicting mortality, the calibration of this score was only assessed using this outcome. The discrimination of the three classifications was assessed by building the receiver operating characteristic (ROC) curve. 13
The ROC curves were obtained using the whole range of scores of each classification. The areas under the ROC curves and their 95% confidence intervals (CIs) were calculated and compared by using a nonparametric test from a method previously described by Hanley et al.14
The ROC curve obtained with the CARE score rating by the attending anesthesiologists was used as the reference for comparisons with all the other curves.
The interrater variability of the CARE scoring was assessed by calculating the agreement rate between groups of physicians. In addition, the κ measure of agreement between the scoring by the attending anesthesiologist and the other groups of physicians (e.g., surgeons and cardiologists) was calculated. Data are expressed as mean ± SD or percentages and their 95% CIs. All P values are two-tailed, and a P value of less than 0.05 was considered significant.
During a 16-week period, 556 consecutive patients were included prospectively in our study. The characteristics of these patients are presented in table 4
. The overall in-hospital mortality and major morbidity rates were 5.8% (95% CI, 3.9–7.7%) and 24.5% (95% CI, 20.9–28.1%), respectively.
Predictive Performances of the CARE Score
The calibration analysis for mortality and severe morbidity in our population is summarized in table 5
. No significant difference was found between expected and observed outcomes. The areas under the ROC curves with the CARE score for predicting mortality (fig. 1
) and major morbidity (fig. 2
) were 0.77 (95% CI, 0.69–0.85) and 0.71 (95% CI, 0.65–0.77).
Comparison of Predictive Performances of the CARE Score with Two Multifactorial Risk Indexes (EuroSCORE and Tu Score)
In the population studied, the calibration analysis showed no significant difference between the expected and observed outcomes for both multifactorial risk indexes (tables 6 and 7
). The area under the ROC curve for mortality (fig. 1
) was 0.78 (95% CI, 0.71–0.85) for the EuroSCORE, 0.77 (95% CI, 0.69–0.85) for the CARE score, and 0.73 (95% CI, 0.63–0.83) for the Tu score (not significant [NS]). Similarly, the areas under the ROC curves for morbidity were also comparable among the three classifications (fig. 2
). For all classifications, the discrimination was significantly lower for major morbidity than for mortality.
Variability of the CARE Score among Anesthesiologists, Surgeons, and Cardiologists
All patients (n = 556) were scored at arrival in the cardiac intensive care unit by a second anesthesiologist who was unaware of the CARE score category given in the preoperative period. The level of agreement of the CARE score between two different anesthesiologists was 90% (95% CI, 87.5–92.5%), and the κ value was 0.739 (SE = 0.033; P < 0.0001). The goodness-of-fit test when the patient was scored by a second anesthesiologist showed a significant difference between expected and observed rates of mortality (chi-square = 20.4; P = 0.009) and but not for morbidity (chi-square = 13.3; P = 0.102). The CARE score, determined by the second anesthesiologist at the arrival of patient in the intensive care unit, had an area under the ROC curve of 0.78 (95% CI, 0.70–0.86) for mortality and 0.72 (95% CI, 0.66–0.78) for morbidity. These values were not significantly different from those obtained by the anesthesiologist in the preoperative period.
The preoperative determination of the CARE score category by surgeons was available in 463 patients. The level of agreement of the CARE score between anesthesiologists and surgeons in preoperative period was 83% (95% CI, 79.7–86.4%), and the κ value was 0.613 (SE = 0.038; P < 0.0001). The goodness-of-fit test when the patient was stratified by surgeons showed no difference between expected and observed rates of mortality (chi-square = 10.7; P = 0.219) and morbidity (chi-square = 3.2; P = 0.921). The CARE score determined by surgeons had areas under the ROC curves of 0.86 (95% CI, 0.78–0.94) for predicting mortality and 0.75 (95% CI, 0.69–0.81) for predicting major morbidity. A total of 463 patients were scored both by the surgeon and the attending anesthesiologist. In these patients, the areas under the ROC curves were not significantly different for predicting mortality (0.86 [95% CI, 0.78–0.94]vs. 0.84 [95% CI, 0.76–0.92]; NS) or predicting major morbidity (0.75 [95% CI, 0.69–0.81]vs. 0.73 [95% CI, 0.66–0.79]; NS).
A total of 367 patients were also scored by a cardiologist using the CARE score. The overall rate of agreement was 77% (95% CI, 72.7–81.3%), and the κ value was 0.453 (SE = 0.049; P < 0.0001). The goodness-of-fit test when the patient was scored by a cardiologist showed a significant difference between expected and observed rate of mortality (chi-square = 23.3; P = 0.003) but not for morbidity (chi-square = 13.7; P = 0.09). The CARE score determined by the cardiologist had areas under the ROC curves of 0.77 (95% CI, 0.63–0.91) for predicting mortality and 0.70 (95% CI, 0.64–0.76) for predicting major morbidity. A total of 367 patients were scored both by the cardiologist and the attending anesthesiologist. In these patients, the areas under the ROC curves were not significantly different for predicting mortality (0.77 [95% CI, 0.63–0.91]vs. 0.79 [95% CI, 0.69–0.89]; NS) or predicting major morbidity (0.70 [95% CI, 0.64–0.76]vs. 0.69 [95% CI, 0.63–0.75]; NS).
The main findings of the current study are that (1) the CARE score, a simple scoring system recently initially described in a group of Canadian patients, is equally applicable in a cardiac surgical population from another center; (2) its performance is similar to those obtained with two popular multifactorial risk indexes, including the EuroSCORE; and (3) the stratification by the CARE score is devoid of significant variability when used by different group of physicians, including surgeons and cardiologists.
Up to the current time, the risk-adjusted prediction of outcome after cardiac surgery has been preferentially assessed by multifactorial risk indexes. 1–10,15
These scores are obtained from multiple regression logistic analysis, which allows for identification of risk factors of morbidity, mortality, or both. Because these scores are not easily calculated at the bedside, few clinicians use them routinely. Recently, Canadian authors have reported reliable performance of the CARE score for predicting duration of stay in the hospital, morbidity, and mortality after cardiac surgery. 11
In contrast to multifactorial risk indexes, the CARE score is an intuitive risk ranking similar to the ASA physical status classification. The score system was constructed on both clinical judgment and three classic risk factors, comorbidity, surgical complexity, and operative priority. Although simple, the CARE score has a predictive performance similar to three popular multifactorial risk indexes. Because the CARE score has only been evaluated in a single institute, further investigations were required to confirm the reliability of this new simple classification at other centers. Therefore, we assessed both the calibration and the discrimination of the CARE score at our institute. We confirmed that CARE score remains a reliable outcome predictor in a cardiac surgical population from a center other than where it was initially developed. We found no significant difference between expected and observed outcomes when the CARE score was evaluated preoperatively by an anesthesiologist. In our population, the CARE score provided an acceptable discrimination for predicting mortality and major postoperative morbidity. It is worth noting, as recently reported by Heijmans et al.
that an area under the ROC curve between 0.7 and 0.9 means that the classification has acceptable discrimination. In addition, the performance of the scores obtained in our population is consistent with those reported by Canadian authors. As previously reported by Dupuis et al.
we found that the CARE score was better for predicting mortality than for predicting morbidity. To our knowledge, the current study is the first study to validate the predictive performance of the CARE score in a cardiac surgical population with different preoperative characteristics.
In the population studied, the overall mortality rate was higher than that observed by Dupuis et al.11
Two principal reasons could explain this divergence. First, our study included a larger proportion of patients with advanced age: 26% of our patients were aged 75 yr or older, whereas Dupuis et al.11
included only 16.5% of this age group. Several studies 17–19
reported that elderly patients undergoing cardiac surgery have a higher inhospital mortality rate. However, the overall mortality rate found in the current study is consistent with those reported in previous European studies. 9,20
Second, 22% of the patients studied (only 14.5% reported by Dupuis et al.11
) underwent complex or combined surgical procedures, which are universally known to be a important risk factor of mortality in cardiac surgery. 1,5,9
On other hand, the overall morbidity rate obtained in the current study was comparable with that observed by Dupuis et al.11
The EuroSCORE has been reported by numerous authors 21–24
as a robust multifactorial index. Initially described in European population, 9
this score has been more recently validated in North American patients. 23
To date, no previous study has compared the predictive performances of this score with those of the CARE score. In our population, both scores had an acceptable calibration. In addition, the area under the ROC curve for predicting mortality and morbidity were comparable between these two scores. It should be noted that the performances of these two scores were always better than that obtained from the Tu score. These findings are consistent with those previously reported by several authors. 11,21
Despite its relative subjectivity, a good concordance was found between CARE scoring by different groups of physicians. The level of agreement of the CARE score between anesthesiologists was 90%, with a κ value of 0.739. These results are consistent with those previously reported by Dupuis et al.11
We also evaluated the concordance of the CARE score between anesthesiologists and surgeons. The agreement rate of the CARE scoring was 83%, with a κ value of 0.613. The lowest agreement rate (77%) was obtained between anesthesiologists and cardiologists. However, the κ value of 0.453 (SE = 0.049; P
< 0.0001) suggested that this variability is not significant.
The following points must to be considered in the assessment of the clinical relevance of our study. First, we included only 556 patients, and consequently, few patients with high risk were enrolled. However, the aim of our study was only to evaluate the predictive performances of a score that has been previously validated in a Canadian population. Second, we evaluated the performances of the EuroSCORE for predicting in-hospital mortality. Nevertheless, the EuroSCORE was initially described for predicting 30-day mortality. Third, we evaluated the variability between physicians by arbitrarily choosing the immediate preoperative scoring by the anesthesiologist as the reference point. Fourth, we did not evaluate the performances of the CARE score for predicting the duration of stay in hospital as previously reported by Dupuis et al.11
However, these authors have already emphasized the modest performance of this score for predicting this outcome. Fifth, we did not assess the variability associated with the use of the multifactorial risk index. However, it has been previously shown that the use of risk scoring systems may be associated with errors in data entering and computing. 25
In conclusion, our study confirms that the CARE score remains a simple but efficient tool for predicting mortality and major morbidity after cardiac surgery at a center other than that where it was initially developed. Its predictive performance is similar to multifactorial risk indexes, including the EuroSCORE. In addition, the CARE score remains devoid of significant variability when used by different groups of physicians, including surgeons and cardiologists.
The authors thank David Baker, M.D., F.R.C.A. (Staff Anesthesiologist, Department of Anesthesiology, Centre Hospitalier Universitaire Necker-Enfants Malades, Paris, France), for reviewing the manuscript.
1. Parsonnet V, Dean D, Bernstein AD: A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989; 79(suppl I):3–12
2. Tuman KJ, McCarthy RJ, March RJ, Hassan N, Ivankovich AD: Morbidity and duration of ICU stay after cardiac surgery: A model for preoperative risk assessment. Chest 1992; 102:36–44
3. Higgins TL, Estafanous FG, Loop FD, Beck GJ, Blum JM, Paranandi L: Stratification of morbidity and mortality outcome by preoperative risk factors in coronary artery bypass patients: A clinical severity score. JAMA 1992; 267:2344–8
4. O'Connor GT, Plume SK, Olmstead EM, Coffin LH, Morton JR, Maloney CT, Nowicki ER, Levy DG, Tryzelaar JF, Hernandez F, Adrian L, Casey KJ, Bundy D, Soule DN, Marrin CAS, Nugent WC, Charlesworth DC, Clough R, Katz S, Leavitt BJ, Wennberg JE, for the Northern New England Cardiovascular Disease Study Group: Multivariate prediction of in-hospital mortality associated with coronary artery bypass graft surgery. Circulation 1992; 85:2110–18
5. Tu JV, Jaglal SB, Naylor D, the steering Committee of the Provincial Adult Cardiac Care Network of Ontario: Multicenter validation of a risk index for mortality, intensive care unit stay, and overall hospital length of stay after cardiac surgery. Circulation 1995; 91:677–84
6. Roques F, Gabrielle F, Michel P, De Vincentiis C, David M, Baudet E: Quality of care in adult heart surgery: Proposal for a self-assessment approach based on a French multicenter study. Eur J Cardiothorac Surg 1995; 9:433–9
7. Magovern JA, Sakert T, Magovern GJ Jr., Benckart DH, Burkholder JA, Liebler GA, Magovern GJ Sr: A model that predicts morbidity and mortality after coronary artery bypass graft surgery. J Am Coll Cardiol 1996; 28:1147–53
8. Higgins TL, Estafanous FG, Loop FD, Beck GJ, Lee J, Starr NJ, Knaus WA, Cosgrove DM: ICU admission score for predicting morbidity and mortality risk after coronary artery bypass grafting. Ann Thorac Surg 1997; 64:1050–8
9. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R, the Euro
SCORE study group: European system for cardiac operative risk evaluation (EuroScore). Eur J Cardiothorac Surg 1999; 16:9–13
10. Wong DT, Cheng DCH, Kustra R, Tibshirani R, Karski J, Carroll-Munro J, Sandler A: Risk factors of delayed extubation, prolonged length of stay in the intensive care unit, and mortality in patients undergoing artery bypass graft with fast-track anesthesia. ANESTHESIOLOGY 1999; 91:936–44
11. Dupuis JY, Wang F, Nathan H, Lam M, Grimes S, Bourke M: The cardiac anesthesia risk evaluation score: A clinically useful predictor of mortality and morbidity after cardiac surgery. ANESTHESIOLOGY 2001; 94:194–204
12. Plackett RL: Karl Pearson and the chi-squared test. Int Stat Rev 1983; 51:59–72
13. Sweets JA: Measuring the accuracy of diagnostic systems. Science 1988; 240:1285–93
14. Hanley JA, McNeil BJ: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148:839–43
15. Kurki TS, Kataja MJ, Reich DL: Validation of a preoperative risk index as a predictor of perioperative morbidity and hospital costs in coronary artery bypass graft surgery. J Cardiothorac Vasc Anesth 2002; 16:401–4
16. Heijmans JH, Maessen JG, Roekaerts PMHJ: Risk stratification for adverse outcome in cardiac surgery (review article). Eur J Anaesthesiol 2003; 20:515–27
17. Mehta RH, Eagle KA, Coombs LP, Peterson E, Edwards FH, Pagani FD, Deeb GM, Bolling SF, Prager RL, Society of thoracic surgeons national cardiac registry: Influence of age on outcomes in patients undergoing mitral valve replacement. Ann Thorac Surg 2002; 74:1459–67
18. Peterson ED, Cowper PA, Jollis JG, Bebchuk JD, DeLong ER, Muhlbaier LH, Mark DB, Pryor DB: Outcomes of coronary artery bypass graft surgery in 24,461 patients aged 80 years or older. Circulation 1995; 92(9 suppl):II85–91
19. Craver JM, Puskas JD, Weintraub WW, Shen Y, Guyton RA, Gott JP, Jones EL: 601 octogenarians undergoing cardiac surgery: Outcome and comparison with younger age groups. Ann Thorac Surg 1999; 67:1104–10
20. Lasocki S, Provenchère S, Bénessiano J, Vicaut E, Lecharny JB, Desmonts JM, Dehoux M, Philip I: Cardiac troponin I is an independent predictor of in-hospital death after adult cardiac surgery. ANESTHESIOLOGY 2002; 97:405–11
21. Geissler HJ, Hölzl P, Marohl S, Kuhn-Régnier F, Mehlhorn U, Südkamp M, de Vivie ER: Risk stratification in heart surgery: Comparison of six score systems. Eur J Cardiothorac Surg 2000; 17:400–6
22. Al-Ruzzeh S, Asimakopoulos G, Ambler G, Omar R, Hasan R, Fabri B, El-Gamel A, DeSouza A, Zamvar V, Griffin S, Keenan D, Trivedi U, Pullan M, Cale A, Cowen M, Taylor K, Amrani M: Validation of four different risk stratification systems in patients undergoing off-pump coronary artery bypass surgery: A UK multicentre analysis of 2223 patients. Heart 2003; 89:432–5
23. Nashef SA, Roques F, Hammill BG, Peterson ED, Michel P, Grober FL, Wyse RKH, Ferguson TB: Validation of European system for cardiac operative risk evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg 2002; 22:101–5
24. Kurki TS, Järvinen O, Kataja MJ, Laurikka J, Tarkka M: Performance of three preoperative risk indices; CABDEAL, EuroSCORE and Cleveland models in a prospective coronary bypass database. Eur J Cardiothorac Surg 2002; 21:406–10
25. Burack JH, Impellizzeri P, Homel P, Cunningham JN: Public reporting of surgical mortality: A survey of New York state cardiothoracic surgeons. Ann Thorac Surg 1999; 68:1195–202
This article has been cited 5 time(s).
European Journal of Cardio-Thoracic SurgeryAccuracy, calibration and clinical performance of the EuroSCORE: can we reduce the number of variables?European Journal of Cardio-Thoracic Surgery
CirculationRisk of Assessing Mortality Risk in Elective Cardiac Operations Age, Creatinine, Ejection Fraction, and the Law of ParsimonyCirculation
Journal of Cardiothoracic and Vascular AnesthesiaPerioperative Statin Therapy Is Associated With a Significant and Dose-Dependent Reduction of Adverse Cardiovascular Outcomes After Coronary Artery Bypass Graft SurgeryJournal of Cardiothoracic and Vascular Anesthesia
Current Opinion in CardiologyPredicting outcomes in cardiac surgery: risk stratification matters?Current Opinion in Cardiology
© 2004 American Society of Anesthesiologists, Inc.
Publication of an advertisement in Anesthesiology Online does not constitute endorsement by the American Society of Anesthesiologists, Inc. or Lippincott Williams & Wilkins, Inc. of the product or service being advertised.