Secondary Logo

Journal Logo

Low Accuracy of the HeartMate Risk Score for Predicting Mortality Using the INTERMACS Registry Data

Kanwar, Manreet K.*; Lohmueller, Lisa C.; Kormos, Robert L.; Loghmanpour, Natasha A.; Benza, Raymond L.*; Mentz, Robert J.§; Bailey, Stephen H.*; Murali, Srinivas*; Antaki, James F.

doi: 10.1097/MAT.0000000000000494
Adult Circulatory Support
Free

Selection is a key determinant of clinical outcomes after left ventricular assist device (LVAD) placement in patients with end-stage heart failure. The HeartMate II risk score (HMRS) has been proposed to facilitate risk stratification and patient selection for continuous flow pumps. This study retrospectively assessed the performance of HMRS in predicting 90 day and 1 year mortality in patients within the Interagency Registry for Mechanically Assisted Circulatory Support (INTERMACS). A total of 11,523 INTERMACS patients who received a continuous flow LVAD between 2010 and 2015 were retrospectively categorized per their calculated HMRS to predict their 90 day and 1 year risk of mortality. The performance of the score was evaluated by the area under curve (AUC) of the receiver operator characteristic. We also performed multiple regression analysis using variables from the HMRS calculation on the INTERMACS data. The HMRS model showed moderate discrimination for both 90 day and 1 year mortality prediction with AUCs of 61% and 59%, respectively. The predictions had similar accuracy irrespective of whether the pump was axial or centrifugal flow. Multivariable analysis using independent variables used in the original HMRS analysis revealed different set of variables to be predictive of 90 day mortality than those used to calculate HMRS. HMRS predicts both 90 day and 1 year mortality with poor discrimination when applied to a large cohort of LVAD patients. Newer risk prediction models are therefore needed to optimize the therapeutic application of LVAD therapy. Patient selection for appropriate use of LVADs is critical. Currently available risk stratification tools (HMRS) continue to be limited in their ability to accurately predict mortality after LVAD. This study highlights these limitations when applied to a large, comprehensive, multicenter database. HMRS predicts mortality with only modest discrimination when applied to a large cohort of LVAD patients. Better risk stratification tools are needed to optimize outcomes.

From the *Cardiovascular Institute, Allegheny Health Network, Pittsburgh, Pennsylvania; Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania; Heart and Vascular Institute, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania; and §Division of Cardiology, Department of Medicine, Duke University Medical Center, Durham, North Carolina.

Submitted for consideration August 2016; accepted for publication in revised form November 2016.

Disclosure: The authors have no conflicts of interest to report.

Funding for this study was provided by the National Institute of Health Division of National Heart, Lung, and Blood Institute Grant NIH R01 HL122639 CORA: A Personalized Cardiac Counselor for Optimal Therapy. Data for this study were provided by the International Registry for Mechanical Circulatory Support (INTERMACS), funded by the National Heart, Lung and Blood Institute, National Institutes of Health, under Contract No. HHSN268201100025C.

Correspondence: Manreet K. Kanwar, Cardiovascular Institute, Allegheny Health Network, 320 E North Ave, Pittsburgh, PA 15212. Email: mkanwar@wpahs.org.

The use of durable mechanical assistance with implantable left ventricular assist devices (LVADs) in patients with end-stage heart failure (HF) has risen dramatically in recent years.1 Patient selection is a key determinant of clinical outcomes with this therapy, but there is limited guidance for appropriate risk stratification.2 Current guidelines from the International Society of Heart and Lung Transplantation (ISHLT) recommend that HF patients who are at “high-risk for 1 year mortality using prognostic models” should be referred for advanced therapies as appropriate (Class II a, level of evidence C).3 Although various risk stratification models have been proposed, the guidelines do not specify any particular model. Thus, at present time, clinicians primarily rely on prior experience when evaluating the suitability of a potential LVAD candidate.4

Over the past 30 years, multiple variables have been identified as predictors of increased mortality in patients undergoing consideration for an LVAD. The Destination Therapy Risk Score was originally derived from patients in the REMATCH trial undergoing durable pulsatile flow pump implantation.5 More recently, and in response to the evolving device technology and patient heterogeneity, new risk models have emerged.6,7 The HeartMate II risk score (HMRS) was introduced as a measure to predict 90 day and 1 year mortality in patients receiving a continuous flow HeartMate II LVAD.8 It was derived from and validated within 1122 patients enrolled in the original HeartMate II Bridge to Transplant (BTT) and Destination Therapy (DT) clinical trials. The score takes into account patient age, albumin, creatinine, international normalized ratio (INR) and the medical center implant volume to stratify patients as high, medium, and low risk for mortality. Although simplistic and easy to calculate, its applicability in routine clinical practice has not been systematically evaluated in the broad population of patients with contemporary LVADs. Previously published single-center studies that assessed the HMRS performance have also questioned the validity and utility of this score when applied to individual patients.9 In this article, we present the first retrospective analysis of the application of HMRS on the comprehensive, large, multicenter Interagency Registry of Mechanically Assisted Circulatory Support (INTERMACS). Patients who received both axial and centrifugal flow pumps were included in this analysis.

Back to Top | Article Outline

Methods

Patient Cohort

INTERMACS maintains a database of pre- and postimplant variables for patients in the United States who receive mechanical circulatory support devices that are approved by the Food and Drug Administration (FDA). Retrospective, de-identified patient data from the INTERMACS registry were provided by the Data Coordinating Center at University of Alabama at Birmingham, which in turn were collected under institutional review board approval from over 150 participating hospitals. The INTERMACS Data, Access, Analysis and Publication (DAAP) Committee approved the study described in this article. We included adult patients (age ≥ 18 years) who received a CF-LVAD as the primary implant between years 2010 and 2015 (n = 14,792; Figure 1). The year 2010 was chosen to assure that only latest pump technology currently in practice was included, and patients with implant after June 2015 were excluded (n = 527) to ensure a minimum of 90 day follow-up. Patients with missing data points (n = 1739) that did not allow us to calculate HMRS were excluded, as were those where the pump flow was not categorized as continuous flow (n = 526; pulsatile pumps n = 504, total artificial hearts n = 22). Patients who ultimately received a right ventricular assist device were included as long as the initial implant was primarily an LVAD. Data from patients whose LVAD was electively removed (e.g., because of transplantation or recovery, n = 324) were included for postoperative outcomes, but censored at the time of explant.

Figure 1

Figure 1

Back to Top | Article Outline

Statistical Analysis

Baseline characteristics from both the patient cohort in the original HMRS publication and the INTERMACS cohort are compared in Table 1. Student’s t-test was used to compare means of continuous variables, and z-test was used for categorical variables. Cohen’s d was used to measure effect size between groups to mitigate the large difference in cohort size. Because the INTERMACS data for age were recorded by decade (e.g., 40–49 or 50–59 years), it was coded by the median value (e.g., 55 for 50–59) for the purpose of computing HMRS. The implant volume of the clinical site for each patient was provided in ranges of 10 per year and was also coded by median value (e.g., 15 for 10–20 LVADs/year). Using the numerical value of HMRS, patients were categorized as low (<1.58), mid (1.58–2.48), and high (>2.48) risk. The performance of the HMRS was assessed by the area under curve (AUC) of the receiver operator characteristic (ROC), Kaplan–Meier curves, and the mortality distribution of predicted low-, medium-, and high-risk patients.

Table 1

Table 1

For HMRS calculation, clinically relevant risk correlates from prior LVAD risk modeling studies (age, female sex, LVAD indication, preoperative vasodilator or vasopressor, preoperative ventilator or intraaortic balloon pump support, inotrope support, platelet count, total bilirubin, aspartate aminotransferase, serum creatinine, hematocrit, INR, albumin, center implant volume (<15), implant year, and right ventricular stroke work index) were entered into stepwise multivariable logistic analysis, with exit criteria of p ≥ 0.05. We entered the same set of variables into a similar multivariate analysis using SPSS to assess their significance in predicting 90 day mortality in the INTERMACS cohort. Hematocrit and use of preoperative vasopressors/vasodilators were omitted from the multiple regression in this analysis because of lack of data in the INTERMACS dataset.

Back to Top | Article Outline

Results

The average age of patients in the INTERMACS cohort was 57.3 years (range 50–68 years), with 79% (n = 6327) male preponderance (Table 1). In total, 5187 (45%) patients received the LVAD as DT and 6248 (54%) patients were categorized as BTT. When compared with the baseline characteristics of patients included in the calculation of HMRS,8 patients in INTERMACS tended to be slightly younger, spent less time on cardiac bypass, and had lower platelet counts. INTERMACS patients included in this analysis had a higher likelihood to be implanted as BTT, with lesser days on pump support (186 days, range 17–237). Vast majority of these patients (n = 9723, 84%) received an axial flow pump, whereas the remaining received a continuous flow centrifugal pump. Mortality was reported as 11.5% at 90 days and 27.3% at 1 year.

The majority of patients (42%, n = 4861) were categorized as low risk according to the HMRS score, followed by medium (35%, n = 3985) and high (23%, n = 2677) risk. At 90 days, 18% (n = 409) patients with high HMRS died, compared with 13% (n = 465) medium-risk and 7% (n = 338) low-risk patients. The ROC in Figure 2B illustrates the predictive performance of these models for 90 day and 1 year mortality, with an AUC of 61% and 59%, respectively. This is considerably lower than reported in the original published studies (71% for 90 day mortality).8Figure 3 shows the Kaplan–Meier survival curves stratified by the HMRS at the 90 day and 1 year time points.

Figure 2

Figure 2

Figure 3

Figure 3

Back to Top | Article Outline

Pump Type and Strategy

At 90 days, the score predicted mortality similarly for axial (AUC 61%) and centrifugal pumps (AUC 62%) (Figure 2A). For 1 year mortality, the prediction for centrifugal pumps (AUC 61%) was slightly better than axial (AUC 59%) pumps. The prediction performance at 90 days was comparable for BTT (AUC 61%) versus DT (AUC 60%) indications and was slightly lower at 1 year with AUC for BTT and DT at 59% and 58%, respectively.

Back to Top | Article Outline

Multiple Regression Analysis of HMRS Variables on INTERMACS Data

When we entered the variables used for multiple regression analysis for calculating HMRS into a similar analysis on INTERMACS variables, a significantly different list of variables emerged as being statistically significant (Table 2). For example, both INR and center volume were very significant (p < 0.001) in HMRS data but not very significant (p = 0.94 and 0.45, respectively) in the INTERMACS cohort. Alternately, the INTERMACS dataset identified five additional significant variables from the same list that were significantly predictive of 90 day mortality (gender, LVAD indication, preoperative ventilator or intraaortic balloon pump, platelet count, and total bilirubin, all p < 0.05).

Table 2

Table 2

Back to Top | Article Outline

Discussion

Our data clearly demonstrates that the HMRS performance is poor at best when applied to a contemporary, comprehensive, multicenter LVAD database. Originally derived from analyzing outcomes in patients enrolled in the HeartMate clinical trial, HMRS initially gained traction for clinical use given the relative ease of obtaining its five composite variables. However, it should be noted that several potential assumptions were made during its production, which explain its limited applicability that was observed in this study.

Back to Top | Article Outline

Relation Between Variables and Outcomes

HeartMate II risk score was derived from multivariate analysis, and thus assumes a linear and independent relationship between each of the clinical variables and mortality. In other words, it is a weighted sum of individual variables reflecting the cumulative effect on the outcome, and fails to take into account the interdependency among the individual variables and the relationship to mortality. For example, in clinical practice, it is well recognized that preoperative right ventricular dysfunction is associated with a higher mortality risk after LVAD implantation. This risk association is not only based on derangement of hemodynamic variables, but also from the resulting liver congestion and renal dysfunction, all of which are interrelated, and potentially synergistic. However, if right ventricular flow is optimized preoperatively by measures such as diuretic, inotropic, or pulmonary vasodilator therapy, it may result in improvement in measures of liver and renal function, which in turn will improve the risk score, without necessarily affecting the patient’s risk for an adverse post-LVAD outcome. These dynamics play a considerable role in clinical/bed-side decision-making but cannot be captured by a traditional risk score that assume linear relationships between variables and outcomes.

Back to Top | Article Outline

Number of Variables in HMRS

The limited number of variables that go into calculating HMRS fail to capture the complex, heterogeneous risk profile of patients with end-stage HF. For example, INR is one of the five predictive variables in HMRS, with high levels reflective of liver dysfunction and contributing to a high-risk categorization. In routine practice, end-stage HF patients may have normal liver function but have a high INR by virtue of being on anticoagulation. Similarly, the absolute value of creatinine used to define HMRS may be low (suggesting low-risk category), but the patient may be on dialysis (a high-risk feature). Other confounding factors can be attributed to the derivation from clinical trial data and the limited number of variables considered, which fails to account for factors such as nutrition, frailty, socioeconomic dynamics, and psychosocial parameters.

To provide a real-life example using a representative case studies, patient A, a 60 year old male who is INTERMACS level 1, with New York Heart Association class IV symptoms, in acute renal failure with a creatinine level of 2.0 mg/dl, on a ventilator and intraaortic balloon pump would be given a low risk (8% 90 day mortality) by HMRS categorization. Similarly, patient B, a frail 70 year old female with chronic kidney disease (creatinine of 3.3 mg/dl), INTERMACS level 3, with New York Heart Association class IV symptoms would have a medium risk of mortality (11% at 90 days). In clinical practice, both these patients would be considered fairly high risk for an LVAD implantation because of their multiple medical comorbidities that were not captured by HMRS. It is also to be noted that the predictors of 1 year survival when conditioned to a 90 day survival in HMRS calculator were limited to only two variables: age and implant center experience. This is one of the main reasons why clinical use of HMRS in routine practice is limited in application.

Back to Top | Article Outline

Data Sampling

We must remember that HMRS was calculated from the HeartMate II clinical trial data, which had specific inclusion and exclusion criteria. The clinical and surgical experience across institutions implanting VADs has evolved and improved since then. When we applied the variables that were entered into a multivariate risk model to calculate HMRS in the much larger patient cohort in INTERMACS, we found different set of variables to be predictive of 90 day mortality. The main reason for this observed difference is that INTERMACS is a much larger, heterogeneous, and comprehensive database with differences in baseline patient characteristics from the HeartMate II clinical trial, and represents the “real life” advanced HF population encountered in clinical practices in the United States. This highlights the fact that risk scores, when derived from specific data sets, may not be applicable to a wider patient population encountered in clinical practice.

Back to Top | Article Outline

Application to Single Centers

The lack of robustness of these scores is corroborated by other single-center validation studies. Thomas et al.9 applied the HMRS to 205 patients who received a HeartMate II pump at their center to report a poor prediction of 90 day mortality by ROC (AUC = 0.56). They concluded that the score’s generalizability as a universal prognostic score may be limited. Another single-center study retrospectively validated the score’s validity to predict short-term mortality in the critically sick INTERMACS class 1 patients.10 However, for a score to be of value to clinicians, it needs to not only predict poor outcomes in critical patients but also be applicable across the heterogeneous HF population at different stages of their disease. In addition, the score’s applicability becomes even more limited to single-center analysis given that one of the five variables (center volume) would remain constant when applied to an individual center.

It could be argued that the HRMS discrimination for predicting mortality at 90 day with an AUC of 0.61, although considerably lower than noted in the original study, is still modest. However, given that an LVAD implantation is a high-risk, long-term, and expensive intervention associated with significant morbidity and mortality,11 a truly predictive risk score needs to be better than ‘modest’ and complement human decision making.

Recently, our group has reported the use of Bayesian methods of analysis to overcome some of these limitations in predicting post-LVAD implant mortality.12 The Bayesian method clearly outperforms conventional risk scores in predictive accuracy, and we believe that this method could allow clinicians to reliably select patients who are likely to achieve “optimal” clinical outcomes with LVAD implantation. This undoubtedly will go a long way toward cost-effectively using this technology in the management of advanced HF.

Back to Top | Article Outline

Limitations

We acknowledge several limitations of this study, including those that apply to the premise of a retrospective cohort analysis. Age and center experience were calculated as a median from the range of options provided to us. However, our study used the most comprehensive and robust registry of LVAD recipients that is currently available.

Back to Top | Article Outline

Conclusions

The decision to implant an LVAD in an ambulatory patient with advanced HF may be a daunting task for both patients and physicians for multiple reasons. Prognostic risk models have the potential to enhance a clinician’s ability to appropriately select patients for this advanced HF therapy and reduce the likelihood of adverse events. Yet, information gleaned from presently available risk scores models such as HMRS is inconsistent and may be of limited utility for day-to-day clinical practice. There is a critical need for accurate, flexible, and improved predictive model to account for the heterogeneity of end-stage HF patients.

Back to Top | Article Outline

Acknowledgments

The authors thank the Data Access, Analysis, and Publications Committee of INTERMACS for allowing us to use their registry for the study, and we specifically thank Dr. James Kirklin, Dr. Francis Pagani, and Dr. David Naftel. The authors also thank Susan Meyers and Grant Studdard for administrative, database, and statistical assistance with INTERMACS.

Back to Top | Article Outline

References

1. Kirklin JK, Naftel DC, Pagani FD, et al. Seventh INTERMACS annual report: 15,000 patients and counting. J Heart Lung Transplant 2015.34: 1495–1504.
2. Miller LW, Guglin M. Patient selection for ventricular assist devices: A moving target. J Am Coll Cardiol 2013.61: 1209–1221.
3. Feldman D, Pamboukian SV, Teuteberg JJ, et al; International Society for Heart and Lung Transplantation: The 2013 International Society for Heart and Lung Transplantation Guidelines for mechanical circulatory support: executive summary. J Heart Lung Transplant 2013.32: 157–187.
4. Slaughter MS, Pagani FD, Rogers JG, et al; HeartMate II Clinical Investigators: Clinical management of continuous-flow left ventricular assist devices in advanced heart failure. J Heart Lung Transplant 2010.29 (suppl 4): S1–39.
5. Lietz K, Long JW, Kfoury AG, et al. Outcomes of left ventricular assist device implantation as destination therapy in the post-REMATCH era: Implications for patient selection. Circulation 2007.116: 497–505.
6. Imamura T, Kinugawa K, Shiga T, et al. Novel risk scoring system with preoperative objective parameters gives a good prediction of 1-year mortality in patients with a left ventricular assist device. Circ J 2012.76: 1895–1903.
7. Boyle AJ, Ascheim DD, Russo MJ, et al. Clinical outcomes for continuous-flow left ventricular assist device patients stratified by pre-operative INTERMACS classification. J Heart Lung Transplant 2011.30: 402–407.
8. Cowger J, Sundareswaran K, Rogers JG, et al. Predicting survival in patients receiving continuous flow left ventricular assist devices: The HeartMate II risk score. J Am Coll Cardiol 2013.61: 313–321.
9. Thomas SS, Nahumi N, Han J, et al. Pre-operative mortality risk assessment in patients with continuous-flow left ventricular assist devices: Application of the HeartMate II risk score. J Heart Lung Transplant 2014.33: 675–681.
10. Adamo L, Nassif M, Tibrewala A, et al. The HeartMate Risk Score predicts morbidity and mortality in unselected left ventricular assist device recipients and risk stratifies INTERMACS class 1 patients. JACC Heart Fail 2015.3: 283–290.
11. Long EF, Swain GW, Mangi AA. Comparative survival and cost-effectiveness of advanced therapies for end-stage heart failure. Circ Heart Fail 2014.7: 470–478.
12. Loghmanpour NA, Kanwar MK, Druzdzel MJ, Benza RL, Murali S, Antaki JF. A new Bayesian network-based risk stratification model for prediction of short-term and long-term LVAD mortality. ASAIO J 2015.61: 313–323.
Keywords:

risk score; INTERMACS; HMRS

Copyright © 2017 by the American Society for Artificial Internal Organs