Acute kidney injury (AKI) is a frequent complication of cardiac surgery that produces high incidences of morbidity and mortality.1 The incidence of cardiac surgery-associated AKI (CSA-AKI) has been described in several large clinical databases as being between 0.3 and 45%, depending on the definition of AKI and the duration of the postoperative period studied. Between 1.2 and 5% of the CSA-AKI patients require some type of renal replacement therapy (RRT), either continuous or intermittent. In these cases, mortality can be greater than 50%.2,3
The pathogenesis of CSA-AKI is complex and multifactorial. Over the past two decades, multiple preoperative and intraoperative factors for defining the risk of AKI in cardiac surgery patients have been studied. Predisposing factors include age and comorbidities such as the elderly patient with chronic kidney disease, diabetes or atherosclerosis. The prognosis of CSA-AKI also depends on the type of cardiac surgery, surgical time and other patient-related factors such as haemodynamic stability, anaemia and intraoperative diuresis. Postoperative factors such as haemodynamic instability, use of nephrotoxic drugs or anaemia may also contribute to CSA-AKI.1,2
The early diagnosis of AKI after cardiac surgery is not easy to establish. The current basic parameters used in the definition of AKI are oliguria and an increase in serum creatinine concentration, but the latter often occurs late. This is an important limitation because AKI is a continuous process and therefore renal damage may have occurred already in the presence of a small, or even an undetected, increase of serum creatinine concentration. Early changes in a patient's physiology that would indicate renal injury after cardiac surgery have not been defined sufficiently to allow the establishment of a therapeutic approach within the first 24 to 48 h of the onset of AKI.4 However, early detection of CSA-AKI is important to limit kidney damage and improve patient prognosis by enabling prompt treatment of reversible factors that contribute to AKI.3
To facilitate early treatment of AKI, predictive models or prognostic scores for CSA-AKI and for the risk of dialysis after cardiac surgery have been developed. The application of predictive models helps professionals make decisions and take an earlier and more effective clinical approach.5 Over the past two decades, various predictive models for CSA-AKI have been developed, such as those of Chertow et al.,6 Thakar et al.,7 Mehta et al.,8 Brown et al.9 and Wijeysundera et al.10
In 2012, Demirjian et al.11 developed four predictive models of AKI after cardiac surgery in Cleveland (United States), using presurgical and intrasurgical variables. The four predictive models included presurgical variables only (for dialysis), pre- and intrasurgical variables (for dialysis), presurgical variables only (for the composite outcome of either a doubling of serum creatinine level or a requirement for dialysis), and pre- and intrasurgical variables (for the composite outcome of either a doubling of serum creatinine level or a requirement for dialysis). These new predictive models for dialysis showed good calibration and discrimination in the sample population used for validation. Predictive models for the composite endpoint also showed good discrimination whether presurgical or pre- and intrasurgical variables were used, but using only the preoperative variables, the predictive model for the composite end-point showed poor calibration.
The applicability of these four new predictive models11 to clinical practice depends on their performance in a patient population different from that in which they were developed. The aim of this study was to perform an external validation of the four predictive models11 for AKI developing in the postoperative period after cardiac surgery, using data prospectively collected from such a different patient population.
Materials and methods
This was a prospective, observational, multicentre study designed by the Renal Dysfunction in Cardiac Surgery Spanish Group (GEDRCC2). Several hospitals performing cardiac surgery in Spain were invited to participate in the study. The goal for each hospital was to collect a total of 50 consecutive patients or, if not possible to reach that number, to gather data from all consecutive patients over a period of 1 year (September 2012 to September 2013). A total of 23 hospitals took part in this multicentre study (Appendix 1, http://links.lww.com/EJA/A109). Each participating centre was responsible for obtaining approval from its local ethics committee. In the case of the coordinating centre, the approval was obtained from the Clinical Research Ethics Committee of the Government of Navarra (Jesús Arteaga Coloma, MD) for the ‘Study GEDRCC_2: Spanish Renal Dysfunction in Cardiac Surgery Group. Prospective observational study on renal dysfunction after cardiac surgery’, with internal code: EO 3/2012. The date of approval for the study was 2 March 2012.
Consecutive adult patients (>18 years old) who underwent cardiac surgery with sternotomy were included in the study. Patients who required preoperative dialysis were excluded. Each patient gave informed consent to participate in this study.
Two outcomes were studied in patients who developed AKI after cardiac surgery: development of AKI that required dialysis; and a composite end-point of dialysis or a doubling of serum creatinine level in the postoperative period (up to 2 weeks after surgery or upon being discharged from the hospital). In short, we named them as dialysis or composite outcome, respectively.
The diagnosis of AKI in the postoperative period was based on serum creatinine and diuresis, according to the KDIGO definition.12 Initiation of dialysis was based on analytical criteria (hyperkalaemia, metabolic acidosis, overt uraemia), oliguria and/or volume overload.13 In each case, the decision to initiate dialysis was made by the medical team responsible for the patient.
Data were collected from 1067 patients who had cardiac surgery from September 2012 to September 2013. The postoperative observation period was up to 2 weeks after cardiac surgery, or until hospital discharge. We also recorded the length of stay in the ICU, the hospital length of stay and mortality (defined as death in hospital or up to 30 days after surgery).14 We collected all variables for the four risk calculators: presurgical variables to predict dialysis (Calc1), presurgical and intrasurgical variables to predict dialysis (Calc2), presurgical variables to predict the composite outcomes of doubling of serum creatinine level or dialysis therapy (Calc3), and presurgical and intrasurgical variables to predict the same composite outcomes (Calc4), (Appendix 2, http://links.lww.com/EJA/A109). Using the variables shown in Appendix 2, http://links.lww.com/EJA/A109 and outlined in Table 1, the four predictive models were calculated for each of the patients.
The estimated glomerular filtration rate (eGFR) was calculated using the equation four-variable Modification of Diet in Renal Disease (MDRD)15 as in the original publication.11 Laboratory data corresponding to the closest date of surgery were used.
Missing data were handled using a two-stage strategy: a complete-case analysis for covariates with few missing data (14 covariates; from 0.09 to 3.1% missing observations), followed by replacement of missing data by median values for four continuous variables (with >3.1% missing observations), according to the approach followed by Demirjian et al.11
Appropriate descriptive statistics were computed for all variables based on the level of measurement and the empirical data distribution. Differences between demographic and clinical characteristics in the nonoutcome and outcome groups were evaluated using two-sample t tests, Mann–Whitney U tests, Pearson χ2 tests or Fisher exact test as determined by variable type and its distribution. The Kolmogorov–Smirnov test was used to check for normality.
The ability of the models to discriminate between patients with and without dialysis therapy (or with and without the composite outcomes of doubling of serum creatinine level or dialysis therapy) was evaluated with the area under the receiver operating characteristics curve (AUC). The comparison between the AUC was carried out using a nonparametric approach.
Calibration was assessed by calibration plots, calibration intercepts (α) and slopes (β) and Hosmer–Lemeshow goodness-of-fit (H-L) tests.
A type I error rate of 0.05 was assumed. All analyses were conducted using Stata 14 (StataCorp. 2015. Stata Statistical Software: Release 14; StataCorp LP, College Station, Texas, USA) and IBM SPSS Statistics 20. (IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp).
A total of 1067 consecutive cardiac patients were recruited for this study. After excluding 53 patients due to missing data in important covariates, 1014 patients were available for the final analysis.
Table 1 summarises the pre- and intraoperative characteristics of the patients who participated in the study, grouped by those who required dialysis therapy in the postoperative period and by those with the composite outcome of a doubling of the serum creatinine level or who required dialysis therapy.
Median age was 69 years (25th to 75th percentiles, 60 to 75.2) and 65.4% were men. In the postoperative period, only 34 (3.4%) required dialysis therapy and 95 (9.4%) had the composite outcome of either a doubling of the serum creatinine level or dialysis therapy.
ICU median length of stay was 3 days (25th to 75th percentiles, 2 to 5), hospital median length of stay was 10 days (25th to 75th percentiles, 8 to 15) and hospital mortality was 3.6% (n = 37); the mortality rate of patients requiring dialysis therapy was 41.2% (14 deaths occurred in the group of 34 patients undergoing dialysis therapy).
The calibration plots of the predicted probabilities for patients with and without dialysis therapy according to Calc1 (Fig. 1) and Calc2 (Fig. 2) indicate miscalibration across the probability ranges for both models. In agreement with the results from visual inspection of the calibration plots, estimates of the calibration slopes and intercepts, and the H-L tests indicate poor calibration of both Calc1 [α (95% confidence interval, 95% CI) = −3.6 (−3.2 to −4.0); β (95% CI) = 5.7 (1.7 to 9.8); H-L Chi-square = 16.6, P = 0.035] and Calc2 [α (95% CI) = −3.7 (−3.3 to −4.1); β (95% CI) = 6.2 (3.3 to 9.2); H-L Chi-square = 20.0, P < 0.001].
Models Calc3 and Calc4 were also poorly calibrated. The calibration plots of the predicted probabilities for patients with and without the composite outcomes of a doubling of the serum creatinine level or dialysis therapy according to Calc3 and Calc4 are shown in Figs. 3 and 4, respectively. The calibration intercept and slope for Calc3 were −2.7 (95% CI; −2.4, −3.1) and 4.7 (95% CI 2.2 to 7.2), respectively. For Calc4, the calibration intercept and slope were −2.9 (95% CI −2.6 to −3.2) and 5.6 (95% CI, 3.9 to 7.2), respectively. Results from the H-L tests also suggest poor calibration of both Calc3 (H-L Chi-square = 13.9, P = 0.085) and Calc4 (H-L Chi-square = 20.5, P = 0.009).
Figure 5 shows the receiver operating characteristic curves of Calc1 and Calc2 for dialysis therapy, which reflected a fair discriminative ability of the Calc1 model (AUC = 0.79; 95% CI = 0.72 to 0.86) and a good discriminative ability of the Calc2 model (AUC = 0.80, 95% CI = 0.72 to 0.88). There was no significant difference between these two areas (P = 0.78).
The receiver operating characteristic curves of Calc3 and Calc4 for the composite outcome of doubling of serum creatinine level or dialysis therapy are shown in Fig. 6. Both Calc3 and Calc4 models showed a fair discriminative ability [AUC of Calc3 (95% CI) = 0.70 (0.64 to 0.75), and AUC of Calc4 (95% CI) = 0.76 (0.71 to 0.81)], although the discriminative capacity of Calc4 was significantly higher than that of Calc3 (P = 0.002).
To our knowledge, this is the first external validation study of the four predictive models for AKI in postoperative cardiac surgery proposed by Demirjian et al.11
These models were developed by Demirjian et al.11 in an attempt to predict which patients would develop a composite of dialysis therapy or doubling of serum creatinine level after cardiac surgery. The objective of these models was to expand and improve the performance of the Thakar score,7 previously developed by the Cleveland Clinic group for predicting the risk of postoperative dialysis therapy, and validated by Englberger et al.16 and other European research groups.17–19 Our validation study consisted of an evaluation of the performance of the new predictive models in a Spanish population.
A general application of the predictive models requires that they be tested in a population that is different from that used to construct the model, as we have done in this study.
The predictive models for dialysis therapy (Calc1 and Calc2) showed a good capacity to distinguish between the patients who will require dialysis therapy and those who will not. The model based on pre- or intrasurgical variables (Calc4) predicting the composite outcome of either dialysis therapy or a doubling of serum creatinine level also showed fair discriminative capacity. Although the model based only on presurgical variables (Calc3) predicting the composite outcomes also showed fair discriminative ability, its magnitude was lower than Calc4. Unlike Calc4, which uses pre- and intrasurgical variables for the composite outcomes, Calc3 uses only presurgical variables, including sex, type of surgery, prior cardiac surgery and the bilirubin values.
The incidence of dialysis therapy in our cohort (3.4%) was greater than that found in the population studied by Demirjian et al.11 in which 1.7% of patients required dialysis in the postoperative period. The incidence of composite outcomes (a doubling of the serum creatinine or requiring dialysis) in our cohort was 9.4%, also greater than that found in the population used to develop the models (4.3%).11
Several differences in the clinical characteristics between our population and the population used in the study by Demirjian et al. are worth mentioning. With regard to the type of surgery, in our population, 51% of the cases were valvular surgery in contrast to 32% in the study by Demirjian et al.11 In this latter study, 32% of the patients had artery revascularisation. In our current study, 10% of the patients had undergone previous cardiac surgery compared with 22% in the population used by Demirjian et al.11
A lack of calibration was observed not only in the two predictive models for dialysis therapy (Calc 1, based on preoperative variables and Calc2, based on both preoperative and intrasurgical variables) but also in Calc 4, the model predictive of the composite endpoint based on pre- and intrasurgical variables. Various factors could have contributed to the lack of calibration in our study. We observed differences in both the pre- and the intraoperative characteristics of our population compared with the population described in the study by Demirjian et al.11 In our cohort, only 10.9% of the patients presented with chronic kidney disease, while in the study by Demirjian et al.11 it was 22%. Other differences included the number of patients with a greater risk of AKI due to previous cardiac surgery, valvular surgery or emergency surgery. Some 6.7% of our cases involved emergency surgery compared with 3.5% for the Cleveland Clinic. With regard to the intrasurgical variables, 49.2% of our patients required packed red blood cell transfusion in the intraoperative period compared with 31% of the patients at the Cleveland Clinic. Despite similar surgical duration, the mean intraoperative diuresis was 600 ml in our population, while that at the Cleveland clinic was 950 ml.
Validation of a score depends on the characteristics of the population under study, especially if the outcome has a low incidence and its cause is multifactorial.20 These differences in our population (fewer coronary revascularisation procedures, more emergency surgery, more packed red blood cell transfusions and lower intraoperative diuresis)21–23 could provide a partial explanation for the greater incidence of AKI associated with cardiac surgery in our population, and thereby contribute to the lack of calibration we observed with the models in our study.
The fact that our study shows adequate discrimination supports the generalisation of the prognostic scores to enable the preoperative prediction of patients whose kidney function may worsen and require dialysis therapy after cardiac surgery. Having this knowledge at an early stage gives physicians the opportunity to inform patients and families of the perioperative risks regarding renal function, risk of dialysis therapy and mortality. In addition, these predictive models may help physicians in the early detection and treatment of postoperative kidney injury. However, the scores proposed by Demirjian et al.11 require a large number of variables, including demographic data, comorbidities, laboratory data and surgical procedure. The high number of variables required for the predictive calculations limits the applicability of the models.
In recent years, several biomarkers of AKI have been proposed,24,25 although their routine use has not been established in regular clinical practice. Possibly in the near future, such biomarkers may change current diagnostic criteria of AKI and may lead to an improvement in the predictive value of the models.
There are potential limitations of our study. First, due to its observational nature, there was no protocol governing various aspects of the patients’ clinical management, such as fluid management, use of vasopressors or diuretics, the analytical criteria required for dialysis and the actual dialysis technique itself.26–29 However, with a common training pathway, these clinical management entities are sufficiently well established in our country that it is unlikely we should expect important differences between the participating hospitals. Second, some studies have recommended at least 100 events and 100 nonevents for external validation studies.30 While a reasonable power was reached to carry out the external validation of the predictive models for the composite outcome of either doubling of serum creatinine level or dialysis therapy (95 events and 919 nonevents), the evaluation of the predictive models for dialysis therapy (34 events and 980 nonevents) deserves a more cautious interpretation. In addition, both a possible bias in the results due to suboptimal handling of missing data, and our relatively old data (from 2012 to 2013) could be considered as potential limitations.
The major strength of this study is the relatively large size of the sample, formed by patients recruited from more than 20 hospitals all over Spain, which should ensure that the target population is representative of the country as a whole. Other important strengths of the study include its prospective design, and detailed records of demographic, clinical and surgical data.
In conclusion, our results show that the new models developed by Demirjian et al.11 have acceptable discriminative ability to predict patients who will need dialysis therapy (or the composite outcome of either a doubling of the serum creatinine level or dialysis therapy) after cardiac surgery. Nevertheless, the lack of calibration of these models and the difficulty in using the models in clinical practice due to the large number of variables included, limit their practicality. A recalibration of the models in a larger sample would be required to establish their applicability and clinical usefulness in other patient populations.
Acknowledgements relating to this article
Assistance with the study: we are grateful to Laura Stokes (University Scientific Translation Service, Research Management Service, University of Navarra, Pamplona, Spain) for her contribution to the translation of the manuscript.
Financial support and sponsorship: none.
Conflict of interest: none.
1. Noyez L. Influence of the definition of acute renal failure postcardiac surgery on incidence, patient identification, and identification of risk factors. Eur J Cardiothorac Surg
2. Thakar CV, Worley S, Arrigain S, et al. Influence of renal dysfunction on mortality after cardiac surgery: modifying effect of preoperative renal function. Kidney Int
3. Demirkilic U, Kuralay E, Yenicesu M, et al. Timing of replacement therapy for acute renal failure after cardiac surgery. J Card Surg
4. Ho J, Tangri N, Komenda P, et al. Urinary, plasma, and serum biomarkers’ utility for predicting acute kidney injury associated with cardiac surgery in adults: a meta-analysis. Am J Kidney Dis
5. Berg KS, Stenseth R, Wahba A, et al. How can we best predict acute kidney injury following cardiac surgery? A prospective observational study. Eur J Anaesthesiol
6. Chertow GM, Lazarus JM, Christiansen CL, et al. Preoperative renal risk stratification. Circulation
7. Thakar CV, Arrigain S, Worley S, et al. A clinical score to predict acute renal failure after cardiac surgery. J Am Soc Nephrol
8. Mehta RH, Grab JD, O’Brien SM, et al. Bedside tool for predicting the risk of postoperative dialysis in patients undergoing cardiac surgery. Circulation
9. Brown JR, Cochran RP, Leavitt BJ, et al. Multivariable prediction of renal insufficiency developing after cardiac surgery. Circulation
10. Wijeysundera DN, Karkouti K, Dupuis JY, et al. Derivation and validation of a simplified predictive index for renal replacement therapy after cardiac surgery. JAMA
11. Demirjian S, Schold JD, Navia J, et al. Predictive models for acute kidney injury following cardiac surgery. Am J Kidney Dis
12. Khwaja A. KDIGO clinical practice guidelines for acute kidney injury. Nephron Clin Pract
13. Mirhosseini SM, Fakhri M, Asadollahi S, et al. Continuous renal replacement therapy versus furosemide for management of kidney impairment in heart transplant recipients with volume overload. Interact Cardiovasc Thorac Surg
14. Nashef SA, Roques F, Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg
15. Levey AS, Bosch JP, Lewis JB, et al. A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation. Modification of Diet in Renal Disease Study Group. Ann Intern Med
16. Englberger L, Suri RM, Li Z, et al. Validation of clinical scores predicting severe acute kidney injury after cardiac surgery. Am J Kidney Dis
17. Candela-Toha A, Elias-Martin E, Abraira V, et al. Predicting acute renal failure after cardiac surgery: external validation of two new clinical scores. Clin J Am Soc Nephrol
18. Vives M, Monedero P, Perez-Valdivieso JR, et al. External validation and comparison of three scores to predict renal replacement therapy after cardiac surgery: a multicenter cohort. Int J Artif Organs
19. Heise D, Sundermann D, Braeuer A, Quintel M. Validation of a clinical score to determine the risk of acute renal failure after cardiac surgery. Eur J Cardiothorac Surg
20. Ranucci M, Castelvecchio S, Menicanti L, et al. Risk of assessing mortality risk in elective cardiac operations: age, creatinine, ejection fraction, and the law of parsimony. Circulation
21. Antunes PE, Prieto D, Ferrao de Oliveira J, Antunes MJ. Renal dysfunction after myocardial revascularization. Eur J Cardiothorac Surg
22. McIlroy DR, Argenziano M, Farkas D, et al. Incorporating oliguria into the diagnostic criteria for acute kidney injury after on-pump cardiac surgery: impact on incidence and outcomes. J Cardiothorac Vasc Anesth
23. Karkouti K, Grocott HP, Hall R, et al. Interrelationship of preoperative anemia, intraoperative anemia, and red blood cell transfusion as potentially modifiable risk factors for acute kidney injury in cardiac surgery: a historical multicentre cohort study. Can J Anaesth
24. De Geus HR, Ronco C, Haase M, et al. The cardiac surgery-associated neutrophil gelatinase-associated lipocalin (CSA-NGAL) score: a potential tool to monitor acute tubular damage. J Thorac Cardiovasc Surg
25. Jongman RM, van Klarenbosch J, Molema G, et al. Angiopoietin/Tie2 dysbalance is associated with acute kidney injury after cardiac surgery assisted by cardiopulmonary bypass. PLoS One
26. Crescenzi G, Torracca L, Pierri MD, et al. ’Early’ and ’late’ timing for renal replacement therapy in acute kidney injury after cardiac surgery: a prospective, interventional, controlled, single-centre trial. Interact Cardiovasc Thorac Surg
27. Garcia-Fernandez N, Perez-Valdivieso JR, Bes-Rastrollo M, et al. Timing of renal replacement therapy after cardiac surgery: a retrospective multicenter Spanish cohort study. Blood Purif
28. Iyem H, Tavli M, Akcicek F, Buket S. Importance of early dialysis for acute renal failure after an open-heart surgery. Hemodial Int
29. Kellum JA, Mehta RL, Angus DC, et al. ADQI Workgroup. The first international consensus conference on continuous renal replacement therapy. Kidney Int
30. Vergouwe Y, Steyerberg EW, Eijkemans MJ, Habbema JD. Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol