A Clinical Score to Predict Acute Renal Failure after Cardiac Surgery

Abstract

 

The risk of mortality associated with acute renal failure (ARF) after open-heart surgery continues to be distressingly high. Accurate prediction of ARF provides an opportunity to develop strategies for early diagnosis and treatment. The aim of this study was to develop a clinical score to predict postoperative ARF by incorporating the effect of all of its major risk factors. A total of 33,217 patients underwent open-heart surgery at the Cleveland Clinic Foundation (1993 to 2002). The primary outcome was ARF that required dialysis. The scoring model was developed in a randomly selected test set (n= 15,838) and was validated on the remaining patients. Its predictive accuracy was compared by area under the receiver operating characteristic curve. The score ranges between 0 and 17 points. The ARF frequency at each score level in the validation set fell within the 95% confidence intervals (CI) of the corresponding frequency in the test set. Four risk categories of increasing severity (scores 0 to 2, 3 to 5, 6 to 8, and 9 to 13) were formed arbitrarily. The frequency of ARF across these categories in the test set ranged between 0.5 and 22.1%. The score was also valid in predicting ARF across all risk categories. The area under the receiver operating characteristic curve for the score in the test set was 0.81 (95% CI 0.78 to 0.83) and was similar to that in the validation set (0.82; 95% CI 0.80 to 0.85;P= 0.39). In conclusion, a score is valid and accurate in predicting ARF after open-heart surgery; along with increasing its clinical utility, the score can help in planning future clinical trials of ARF.

The overall mortality after open-heart surgery ranges between 2 and 8% (^1–3). The risk for mortality, however, increases exponentially among patients who develop postoperative acute renal failure (ARF), with mortality rates in excess of 60% (^4–8). When defined in its most severe form, as requiring dialysis, postoperative ARF is an independent risk factor of death (⁹). The survival rate associated with ARF has remained dismal over the past few decades; multiple attempts at therapeutic interventions have failed to demonstrate clear benefits in either amelioration of renal injury or improved survival. The promise indicated by successful interventions in experimental models suggests that the proposed intervention should come early, possibly within 24 to 48 h after inducing renal injury (^10,11). It is extremely difficult to translate early interventions into clinical trials because (1) it is difficult to anticipate renal dysfunction, and (2) use of surrogate markers of GFR as current indicators of acute renal dysfunction leads to a significant delay in the diagnosis of ARF.

Prognostic risk stratification is used to predict renal dysfunction and identify patients who are at a greater risk for developing ARF; ARF after open-heart surgery is one of the clinical settings commonly studied in this regard. Although many analyses have identified independent predictors of ARF, relatively few have addressed the issue of preoperative risk stratification (⁵). These studies, however, either were not well represented by differences in demographic characteristics such as gender and race (⁵) or excluded high-risk patients (¹²), thus limiting their clinical utility. Moreover, the studies did not evaluate the simultaneous effect of multiple risk factors on the outcome, by either their order of importance (e.g., odds of outcome) or their degree of stability (e.g., confidence intervals [CI]), which may affect the accuracy of prediction.

The purpose of this analysis was to develop and validate a clinical score that predicts ARF after open-heart surgery. We aimed to develop a scoring model that accurately predicts ARF by accounting for the effect of all of its major risk factors. To achieve adequate power and generalizability, the study aimed to analyze a large cohort of patients that is well represented by differences in gender and race and includes cardiac surgeries with varying degrees of risk, thus making it more clinically applicable. The data indicate that a clinical score is valid in predicting ARF after open-heart surgery and that incorporating the effect of multiple risk factors in the model improves the accuracy of prediction.

Materials and Methods

Study Population

We studied 33,217 patients who underwent open-heart surgery at the Cleveland Clinic Foundation between April 1993 and December 2002, as recorded in the database of the Department of Cardiothoracic Anesthesiology. The database is approved by the Institutional Review Board to record information in cardiac surgery patients. There were 34,562 surgeries performed on 33,217 patients; for the purpose of this analysis, only the first surgical episode was considered. We excluded 1540 patients from the analysis, including heart transplant recipients; those who required preoperative dialysis, preoperative extracorporeal membrane oxygenation, preoperative tracheostomy, or mechanical ventilation; patients who underwent procedures for automated implantable cardioverter-defibrillator, left ventricular assist devices, or sternal work; and those with missing data. A total of 164 patients met more than one criterion for exclusion. The final study population included 31,677 patients, 69.5% of whom were men (n = 22,012) and 31.5% of whom were women (n = 9665). Racial categories, as recorded in the database, included white (89.1%; n = 28,230), black (4%; n = 1264), and others (6.9%; n = 2183). To generate a scoring system, we randomly selected one half of the total number of patients in our sample to be used as a test set (n = 15,838); the remaining 15,839 patients served as a data set for the validation of the score.

Definitions

The primary outcome was ARF that required dialysis during the postoperative period. The indications for dialysis included uremia, volume overload, or biochemical abnormalities and were based on clinical judgment. We examined the following variables as possible predictors of ARF to develop the scoring model: Age; gender; race; weight; history of congestive heart failure; severe left ventricular dysfunction (ejection fraction <35%); preoperative use of intra-aortic balloon pump; emergency surgery; previous open-heart surgery; history of chronic obstructive pulmonary disease requiring medical therapy; diabetes mellitus; preoperative serum creatinine (mg/dl); cardiopulmonary bypass (CPB) time; and types of cardiac surgery, including coronary artery bypass graft, valve surgery, combined coronary artery bypass graft and valve procedures, and other cardiac surgeries such as ventricular aneurysm repair, pericardiectomy, etc. The rationale for using these variables in the scoring model was based on our previous validation of this database and clinical relevance; a portion of this database (cases from 1993 to 2000) has been validated to identify independent predictors of ARF (^8,13). The details regarding the variables included in the database, criteria for exclusion, and definitions of the risk variables and outcomes have been reported previously (^8,13). To maintain internal consistency regarding the definitions of risk factors and outcomes for the present study, we chose to use the same definitions as used in our previous analyses.

Statistical Analyses

The scoring model was developed on the test data set (n = 15,838). We compared patients with and without ARF univariably on the risk factors considered for the score using χ², Fisher exact, and t test as appropriate. To select the variables that would be used in the score, we fit logistic regression models for ARF, using stepwise selection to choose the predictors to include in each model in 1000 bootstrapped samples from the test data set (¹⁴). We selected variables that were significant predictors of ARF in >50% of the bootstrap runs for the final model. We chose to categorize the continuous predictors of ARF using no more than two cut points for each predictor. To choose these cut points, we used the WinBUGS Gibbs sampling program to estimate and plot the posterior distributions of each cut point. Although CPB time appeared in >50% of the models, we chose not to include it in this scoring model because it is a function of intraoperative course and cannot be determined preoperatively. However, because history of previous open-heart surgery is associated with longer bypass times (data not shown), we chose it as a surrogate marker to represent CPB time in the scoring model. This allows the score to be based exclusively on preoperative risk factors. We then performed a final logistic regression analysis using this reduced set of risk factors. We assigned score points to each risk factor using the model parameter estimates, multiplied by 2 and rounded to the nearest integer. The logistic estimates for the risk variables, corresponding score points, and the contributed area under the curve (AUC) for each variable are shown in Table 1.

Table 1: Logistic model parameter estimates, score points, and AUC for each risk factor included in the scoring model^a

We then applied the scoring model generated on the test data set to the validation data set to assess the accuracy of the score in predicting ARF. To measure and compare the predictive accuracy of the model in the test and validation data sets, we generated receiver operating characteristics (ROC) curves and compared their C-statistics (AUC) (¹⁵). Statistical analyses were performed using SAS version 8.0 (SAS Institute Inc., Cary, NC) and WinBUGS 1.3 (MRC and Imperial College of Science, Technology and Medicine, Cambridge, UK). P = 0.05 was used for all tests.

Results

The observed overall frequency of ARF requiring dialysis in the test data set was 1.7% (n = 269). Tables 2 and 3 show the distribution of the risk factors used to develop the scoring model according to the occurrence of ARF. By univariate comparison, patients with ARF were more likely to have preoperative risk factors such higher serum creatinine level, presence of diabetes, chronic obstructive pulmonary disease, previous cardiac surgery, markers of severe cardiovascular disease, and female gender, as well as intraoperative risk factors such as longer CPB time.

Table 2: Categorical risk factors in patients with or without ARF that required dialysis^a

Table 3: Risk variables (continuous) in patients with or without ARF that required dialysis

Table 1 shows those variables that were selected for the final scoring model. Age, weight, race, peripheral vascular disease, and cerebrovascular disease were excluded for the final scoring model on the basis of the results of bootstrap analysis. Table 1 also shows the logistic estimates, corresponding score points, and associated AUC for all of the risk factors. The resulting ARF scores (Table 4) range between a minimum of 0 to a maximum of 17 points (patients can belong to only one category of type of surgery or level of creatinine).

Table 4: ARF score^a

Table 5 shows the number of patients at each ARF score level and the corresponding frequency of ARF in both the test and the validation data sets. The frequency of ARF at each score level in the validation data set fell within the 95% CI of the ARF frequency at the corresponding score level in the test data set. This demonstrates the validity of the predictive model at all levels of the ARF score. Figure 1 illustrates frequency of ARF and the 95% CI at each score level in the two data sets.

Table 5: Frequency of ARF that required dialysis at various score levels in test and validation sets

Figure 1.: Frequency of acute renal failure (ARF) that requires dialysis corresponding to ARF score in test and validation data sets.

There were fewer patients at higher score levels; no patients had a score of >13. We arbitrarily formed four risk categories of increasing severity (scores 0 to 2, 3 to 5, 6 to 8, and 9 to 13) to increase the number of patients in each risk category and to enhance the clinical utility of the score. The frequency of ARF across the four categories of severity in the test data set ranged between 0.5 and 22.1% (Figure 2). The frequency of ARF in each risk category of the validation data set also fell within the 95% CI of the corresponding ARF frequency in the test data set, as shown in Table 4.

Figure 2.: Frequency of ARF across risk categories in test and validation sets.

Figure 3 demonstrates the ROC curves generated to test and compare the diagnostic accuracy of the scoring model in the two data sets. The AUC for the score in the test data set was 0.81 (95% CI 0.78 to 0.83), whereas it was 0.82 (95% CI 0.80 to 0.85) in the validation data set; the area under the ROC curves was not significantly different (P = 0.39). We also calculated the AUC for the ROC curves for the score by excluding the total number of deaths in the overall cohort of patients. This allowed us to avoid the bias in prediction that may be introduced by differential survival among nondialyzed patients. The AUC for the test and validation sets were 0.82 (95% CI 0.79 to 0.86) and 0.84 (95% CI 0.80 to 0.87), respectively, and were statistically similar (P = 0.65).

Figure 3.: Receiver operating characteristic curve analysis for the test and validation sets.

Discussion

ARF is one of the most serious complications after cardiac surgery; the occurrence of ARF strikingly magnifies the risk for postoperative death (^5,8,9). There has been little improvement in our ability to treat renal injury and to demonstrate a survival benefit among this subgroup of patients (¹⁶). Preoperative renal risk stratification provides an opportunity to develop strategies of early diagnosis and intervention. The present study provides a clinical score to predict ARF after open-heart surgery and validates that score in a comparable population.

Few studies have addressed this issue. Chertow et al. (⁵) were among the first to develop a risk algorithm to predict postoperative ARF. This analysis involved a large multicenter cohort of patients (n = 43,642) who underwent cardiac surgery from the Veterans Administration health system. The population was predominantly men (99% men). The risk algorithm was subsequently assessed by Fortescue et al. (¹²) in a smaller cohort of patients (n = 8797). Although it included a larger proportion of women, this analysis did not include high-risk cardiac surgical procedures such as valve surgery. Both of these analyses excluded patients with severe preoperative renal dysfunction (creatinine >3 mg/dl). We also have analyzed a similar nonparametric recursive partitioning model in our cohort of patients (¹⁷). Although these were among the few analyses aimed at risk stratification involving large patient cohorts, there was a limitation in the statistical model that was used. It did not allow testing of the effect of various risk factors by order of their magnitude of association with ARF.

The present analysis includes all of the major risk factors of ARF. The estimates of prediction were subsequently converted into a clinically applicable score. The area under the ROC curve for the ARF score was 0.81 in the test data set and 0.82 in the validation data set. The ROC curve traditionally has been used as a method to describe the intrinsic accuracy of a diagnostic test as well as to compare various diagnostic tests; it can be interpreted as the probability that a randomly selected patient with a disease has a test result indicating greater suspicion of the disease than that of a randomly chosen patient without the disease (¹⁵). In other words, when applied to evaluate the accuracy of the ARF score in two randomly selected patients—one with and one without ARF—the probability is 0.82 that the patient with ARF will have a higher score than the patient without ARF. Thus, the present study enhances the accuracy of prediction by accounting for the interaction of all major risk factors by order of their degree of association with postoperative ARF.

In an earlier study, we identified the independent predictors of ARF after open-heart surgery. In addition to confirming the traditional risk factors in a large cohort of patients (n = 24,660), our earlier study identified female gender as an independent risk factor of postoperative ARF (⁸). The present analysis supplements earlier observations by developing a clinical ARF score that is validated on a randomly selected cohort of patients that was well represented by differences in gender and race. As indicated in Table 4, the frequency of ARF corresponding with each level of the score in the validation data set fell within the 95% CI of the ARF frequency in the test data set. Said in another way, at all levels of risk, the score was valid in predicting ARF when applied to a randomly selected cohort of patients.

A number of studies address the risk for ARF after open-heart surgery. When defined in its most severe form, as requiring dialysis, the reported frequency of postoperative ARF is usually <5% (^5,6,18,19). This low event rate translates into two major limitations related to clinical research. First, it limits the ability of most of the epidemiologic studies to identify independent predictors of ARF, as a result of inadequate power (i.e., number of patients). Second, it impedes the development of clinical studies that pertain to early diagnosis and intervention in ARF by necessitating the enrollment of a large number of patients. This underscores the importance of risk stratification in clinical settings where renal injury can be anticipated.

The present study involves a large cohort of patients, sufficient to generate and validate a score that incorporates multiple independent risk factors, despite the low event rate of ARF. Furthermore, we arbitrarily divided the score into four risk categories (Table 4). This allowed optimizing of the number of patients who are at a moderate to high risk for developing ARF and improvement in the clinical utility of the score. The validity of the score was maintained even after the score levels were condensed into four risk categories. The frequency of ARF ranged between 0.4 and 22% across the risk categories. It should be noted that the overall frequency of ARF was 1.8%, so the risk categories allow identification of subgroups of patients who have lower- as well as higher-than-average risk for developing ARF. This can be a valuable tool used to randomize patients in clinical trials of ARF.

A weakness of the present study is that the data are derived from a single center. The model needs to be tested prospectively at multiple centers to substantiate its broad applicability. However, unlike other studies, it does include the single largest cohort of patients that is well represented by differences in gender, race, and all types of cardiac surgical procedures. Inherent to the observational study design is the limitation that it establishes association and not causality. Thus, it would be incorrect to justify a change in clinical decision making regarding open-heart surgery as a result of the risk for ARF on the basis of any such analyses. Nevertheless, it can improve individual patient care by allowing us to identify accurately patients who have a greater likelihood of developing ARF. It should be noted that the proposed scoring model predicts severe form of ARF defined by requirement of dialysis. Although less severe degrees of renal dysfunction (defined arbitrarily) may portend a risk for worse outcomes, the clinically relevant threshold of renal dysfunction after cardiac surgery remains unclear. Our rationale to choose this definition was based on clinical relevance and, more important, because ARF that requires dialysis after cardiac surgery has been unequivocally associated with mortality.

In conclusion, we provide a clinical score validated in our population of patients that predicts ARF after open-heart surgery. The score enhances the accuracy of prediction by accounting for the effect of all major risk factors of ARF. In addition, the score identifies patients who have a lower- as well as a higher-than-average risk for ARF. This increases the clinical utility of the score in improving both individual patient care and by providing a vital tool in planning future clinical trials of early diagnosis and intervention in ARF.

Acknowledgments

We are grateful for the contribution of the research personnel from the Department of Cardiothoracic Anesthesiology at the Cleveland Clinic Foundation in maintaining the database and for the contributions of Marion Piedmonte and David Nelson in the initial validation of this database.