A point-based model created using the weight of each independent variable identified by CTA is summarized in Table 3. Points were assigned according to their weight measured by ANN. One point was given for categorical risk factors. The maximum number of points that each patient could score was capped at 6, representing the value at which observed postoperative mortality risk reached its plateau. A total of 11 620 patients (38.2%) scored 0 points, 7631 (25.1%) scored 1 point, 4284 (14.1%) scored 2 points, 3597 (11.8%) scored 3 points, 2450 (8.0%) scored 4 points, 739 (2.4%) scored 5 points, and 137 (0.4%) scored 6 points.
Observed 90-day Mortality
The observed 90-day mortalities were 6%, 8.7%, 10.4%, 11.9%, 15.7%, 16%, and 19.7% for patients with 0, 1, 2, 3, 4, 5, and 6 points, respectively (Figure 2). For each additional point of the scoring system, the observed 90-day mortality increased on average by 2.3% (SD 1.37).
A positive linear relationship was observed between the number of points of the scoring system and observed 90-day mortality with a Pearson correlation coefficient R2 of 0.9795 (P ≤ 0.001). There was also a linear correlation between the OR of 90-day mortality and number of points of the scoring system (Pearson correlation coefficient R2 = 0.9724; P ≤ 0.001).
In comparison to patients with 0 points (reference group), the OR for 90-day mortality was 1.47 (95% CI, 1.32-1.64; P = 0.000) for patients with 1 point, 1.79 (95% CI, 1.58-2.03; P < 0.001) for patients with 2 points, 2.09 (95% CI, 1.84-2.38; P < 0.001) for patients with 3 points, 2.88 (95% CI, 2.52-3.29; P < 0.001) for patients with 4 points, 2.95 (95% CI, 2.38-3.64; P ≤ 0.001) for patients with 5 points, and 3.81 (95% CI, 2.48-5.84; P < 0.001) for patient with 6 points (Figure S2, SDC, http://links.lww.com/TP/B753).
Predicted 90-day Mortality
The predicted 90-day mortality by ANN stratified by the preoperative scoring system is summarized in Table 4. Predicted mortality was categorized as low (≤5%), average (5.1%–10%), increased (10.1%–14.9%), high (15%–19.9%), and very high (≥20%) probability. All patients with 0 score had predicted mortality ≤10%. The proportion of patients with risk of 90-day mortality >10% was 7.8% for patients with 1 point, 50.2% for patients with 2 points, 81.9% for patients with 3 points, 94.5% for patients with 4 points, 84.5% for patients with 5 points, and 96.3% for patients with 6 points.
Performance of the Model
The Hosmer-Lemeshow goodness-of-fit test used to assess the calibration of the model was not statistically significant (P = 0.67), indicating that the model was correctly specified with a linear correlation between the predicted and observed 90-day mortality (Pearson R2 coefficient of 0.99; P ≤ 0.001) (Figure 3). The Brier score of the model was 0.002, denoting that the model was informative with statistically significant differences between the mean value of the Z scores of patients who died within 90 days in comparison to patients who survived beyond 90 days (−2.23 versus −2.37; P < 0.001).
A linear correlation with Pearson R2 coefficient of 0.91 (P ≤ 0.001) was found between the probabilities of 90-day mortality estimated by ANN and logistic regression (Figure S3, SDC, http://links.lww.com/TP/B753).
For the entire cohort, the model’s discriminative performance in identifying patients who died within 90 days after LT had an AUC of 0.601 (95% CI, 0.590-0.613; P < 0.001) (Figure 4A). The AUC increased to 0.952 (95% CI, 0.950-0.954; P < 0.001) for the discrimination of patients with a 90-day mortality risk of ≥10% (Figure 4B). The value of the AUC was 0.930 (95% CI, 0.926-0.935; P < 0.001) for patients with 90-day mortality risk ≥15% (Figure 4C) and 0.866 (95% CI, 0.854-0.877; P < 0.001) for patients with 90-day mortality risk ≥20% (Figure 4D). The sensitivity and specificity of the model for the prediction of 90-day mortality for the entire cohort and patients with higher predicted risks are reported in Figure S4 (Panel A to D; SDC, http://links.lww.com/TP/B753).
In patients with 0–1 points, the net reclassification index of the model for perioperative risk <10% was 9.1%. In patients with 2–3 points, NRI for perioperative risk of 10%–15% was 3.2%, and in patients with 4–6 points, NRI for perioperative risk of ≥15% was 4.8%. Using the cutoff value of 2 or more points, the NRI of the model for patients at increased risk of 90-day mortality (≥10%) was 7.6%.
Within 1 year after LT, 4257 patients (14.0% of the cohort) had died. Mortality affected 9.8% of patients with 0 points, 13.4% of patients with 1 point, 15.8% of patients with 2 points, 17.2% of patients with 3 points, 23.0% of patients with 4 points, 25.2% of patients with 5 points, and 35.8% of patients with 6 points (P ≤ 0.001). For each additional point of the scoring system, the observed 1-year mortality increased in average by 4.3% (Figure 5).
Overall 5-year survival for the cohort was 74% with a median follow-up of 11.1 years (95% CI, 10.7-11.4). During the study period, 8244 patients had died (27.1%) and 22 214 (72.9%) were censored. Five-year patient survival was 78% for patients with 0 points, 73% for patients with 1 point, 72% for patients with 2 points, 71% for patients with 3 points, 65% for patients with 4 points, 59% for patients with 5 points, and 48% for patients with 6 points (P = 0.001) (Figure 6).
The most significant finding of this study is that a predictive model that can identify patients at increased risk of perioperative mortality after LT is feasible using clinical variables attainable during the early phase of their evaluations. Because LT is a life-saving procedure requiring advanced clinical and technical skills, this predicting model is not meant to be used in isolation or as a substitute for good clinical judgment. Yet, it could be a valuable instrument for clinicians, administrators, and investigators as an instrument that can provide an objective estimate of the risk of suboptimal outcomes after LT.25
The allocation of liver grafts based on MELD score26,27 prioritizes the sickest patients on the waitlist27-29 and has changed the characteristics of recipients undergoing LT in the United States and other parts of the world.30,31 Compared to patients who underwent transplantation before the MELD score was implemented, current candidates are older, with more comorbidities32 and higher acuity of liver disease.1,9,33,34 Recent studies have shown that the average perioperative mortality after LT ranges between 5% and 10%,11 but the risk is more significant in patients with high MELD scores,35 several comorbidities,9 advanced age,36 abnormal BMI,37,38 and low performance status.39
One of the common challenges for transplant specialists dealing with the current allocation system is the selection of appropriate surgical candidates. By selecting only patients at low perioperative risk, transplant programs would decline life-saving operations to many individuals who benefit from LT. On the other hand, the selection of very high-risk patients reduces the number of grafts that could be allocated to recipients with a better chance of survival. Finding the balance between these 2 scenarios can be difficult without an objective instrument to stratify patients during the early phases of their evaluations.25
For our model, the C statistics of perioperative mortality risk ≥10%, ≥15%, and ≥20% were 0.95, 0.93, and 0.86, respectively, and for patients with 3 or more points, sensitivity and specificity were 91% and 82%, respectively. These findings are relevant because patients with irreversible liver diseases can only be cured by LT unless unsuitable for surgery. Consequently, the most critical decision to be made is whether or not LT should be performed based on the probability that the patient would survive the operation or not. In many circumstances, this decision is rather straightforward, but for marginal recipients it can be difficult, and despite the best clinical acumen, it can be biased and inconsistent over time. Consequently, a scoring system like the one we are proposing could assist healthcare providers in making more objective decisions during patient selection or in allocating appropriate resources to patients at high perioperative risk.
Several other investigators2,4,5,9,12,13,15,16,35,40-43 have proposed predictive models to identify LT candidates at increased risk of postoperative death. These existing scoring systems include characteristics that are pertinent to recipients, donors, and quality of the graft and require operative variables that become available only once surgery is completed.5 Therefore, most transplant centers do not rely on these models for the listing potential candidates. Another reason is that the predictive performance of current models is only modest with C statistics ranging from 0.63,7 to 0.7.5,8,10,31 Compared with existing models, ours has higher C statistics for the identification of patients at the increased risk of 90-day mortality and the advantage of being usable when patients are initially referred for LT. Therefore, it can be used to counsel patients and their families before surgery regarding their specific probabilities of short-term outcomes and expected survival up to 5 years after LT. Last, our scoring system was developed using a large national dataset that makes it more generalizable than other models developed using single-center datasets.
Despite all these advantages, the results of our study should be interpreted with some caution, because the scoring system has not been tested and validated using other cohorts of patients yet. Although the model performed well in identifying high-risk patients, several limitations are worth mentioning in addition to its retrospective design. First, we could only analyze and subsequently develop the model using variables collected in the STAR files. Because the risk of postoperative death depends on other factors that are not collected in the STAR files, we could not study the impact of malnutrition,44 sarcopenia,45 or frailty,46 which were identified as negative prognostic factors by other groups.
Another limitation is that the STAR files did not provide enough information to determine the sequence of events that led to postoperative death. Therefore, we could not assess whether patients expired from complications directly related to technical problems during surgery or had complications caused by preexisting conditions.
From the methodological point of view, our model might be less accurate when used in other populations.47 To address this issue, we are currently evaluating its validity and performance in a cohort of patients who underwent transplantation between June 30, 2013, and December 31, 2017. We also acknowledge that this model was developed using data from patients who underwent transplant surgery. Therefore, our findings might be mitigated by the inevitable selection bias, because only patients who were deemed surgical candidates were included. In addition, the model includes only preoperative variables and consequently does not incorporate the role of decisions made by surgeons and physicians before proceeding to each transplant. These complex decisions could plausibly modify the risks of perioperative mortality as many transplant programs commonly allocate the best grafts to patients with the highest perioperative risk and vice versa.
Also, the primary diagnosis of liver disease and recipient sex were intentionally excluded from the variables used to develop the predictive model, because the inclusion of these characteristics would disadvantage some groups of patients due to their gender or cause of liver failure. Therefore, it is possible that our model might perform differently in females versus males and for different causes of liver disease.
Finally, it is important to point out that this model was not developed for the stratification of patients who are known to be at an increased risk of perioperative death, such as patients who require a redo LT, or patients who undergo split livers or multivisceral transplant surgeries.
In conclusion, using machine learning techniques, we were able to develop a model to stratify the risk of 90-day postoperative mortality of patients referred for cadaveric LT. This model can also predict the risk of 1-year mortality and 5-year survival based only on pretransplant recipients’ clinical and demographic characteristics. Although the model has good discrimination for high-risk recipients, a validation study will be necessary to test its performance in a different cohort of LT recipients.
1. Rana A, Ackah RL, Webb GJ, et al. No gains in long-term survival after liver transplantation over the past three decades.Ann Surg201926920–27
2. Dutkowski P, Oberkofler CE, Slankamenac K, et al. Are there better guidelines for allocation in liver transplantation? A novel score targeting justice and utility in the model for end-stage liver disease era.Ann Surg2011254745–53Discussion 753
3. Feng S, Goodrich NP, Bragg-Gresham JL, et al. Characteristics associated with liver graft failure: the concept of a donor risk index.Am J Transplant20066783–790
4. Halldorson JB, Bakthavatsalam R, Fix O, et al. D-MELD, a simple predictor of post liver transplant mortality for optimization of donor/recipient matching.Am J Transplant20099318–326
5. Rana A, Hardy MA, Halazun KJ, et al. Survival outcomes following liver transplantation (SOFT) score: a novel method to predict patient survival following liver transplantation.Am J Transplant200882537–2546
6. Schaubel DE, Guidinger MK, Biggins SW, et al. Survival benefit-based deceased-donor liver allocation.Am J Transplant200994 Pt 2970–981
7. Braat AE, Blok JJ, Putter H, et al; European Liver and Intestine Transplant Association (ELITA) and Eurotransplant Liver Intestine Advisory Committee (ELIAC)The Eurotransplant donor risk index in liver transplantation: ET-DRI.Am J Transplant2012122789–2796
8. Rana A, Pallister ZS, Guiteau JJ, et al. Survival outcomes following pediatric liver transplantation (pedi-SOFT) score: a novel predictive index.Am J Transplant2015151855–1863
9. Petrowsky H, Rana A, Kaldas FM, et al. Liver transplantation in highest acuity recipients: identifying factors to avoid futility.Ann Surg20142591186–1194
10. Levesque E, Winter A, Noorah Z, et al. Impact of acute-on-chronic liver failure on 90-day mortality following a first liver transplantation.Liver Int201737684–693
11. Burroughs AK, Sabin CA, Rolles K, et al; European Liver Transplant Association3-month and 12-month mortality after first liver transplant in adults in Europe: predictive models for outcome.Lancet2006367225–232
12. Weismüller TJ, Prokein J, Becker T, et al. Prediction of survival after liver transplantation by pre-transplant parameters.Scand J Gastroenterol200843736–746
13. Schrem H, Reichert B, Frühauf N, et al. The donor-risk-index, ECD-score and D-MELD-score all fail to predict short-term outcome after liver transplantation with acceptable sensitivity and specificity.Ann Transplant2012175–13
14. Luca A, Angermayr B, Bertolini G, et al. An integrated MELD model including serum sodium and age improves the prediction of early mortality in patients with cirrhosis.Liver Transpl2007131174–1180
15. Györi GP, Silberhumer GR, Zehetmayer S, et al. Dynamic changes in MELD score not only predict survival on the waiting list but also overall survival after liver transplantation.Transpl Int201225935–940
16. Lau L, Kankanige Y, Rubinstein B, et al. Machine-learning algorithms predict graft failure after liver transplantation.Transplantation2017101e125–e132
17. Ayloo S, Hurton S, Cwinn M, et al. Impact of body mass index on outcomes of 48281 patients undergoing first time cadaveric liver transplantation.World J Transplant20166356–369
18. Vandenbroucke JP. The making of STROBE.Epidemiology200718797–799
19. Fei Y, Gao K, Hu J, et al. Predicting the incidence of portosplenomesenteric vein thrombosis in patients with acute pancreatitis using classification and regression tree algorithm.J Crit Care201739124–130
20. Hayes T, Usami S, Jacobucci R, et al. Using classification and regression trees (CART) and random forests to analyze attrition: results from two simulations.Psychol Aging201530911–929
21. Brier GW. Verification of forecasts expressed in terms of probability.Mon Weather Rev1950781–3
22. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, et al. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.Stat Med200827157–172. Discussion 207
23. Hosmer DW, Hosmer T, Le Cessie S, et al. A comparison of goodness-of-fit tests for the logistic regression model.Stat Med199716965–980
24. Strauss D, Shavelle R. An extended Kaplan-Meier estimator and its applications.Stat Med199817971–982
25. Fayek SA, Quintini C, Chavin KD, et al. The current state of liver transplantation in the United States: perspective from American Society of Transplant Surgeons (ASTS) scientific studies committee and endorsed by ASTS council.Am J Transplant2016163093–3104
26. Malinchoc M, Kamath PS, Gordon FD, et al. A model to predict poor survival in patients undergoing transjugular intrahepatic portosystemic shunts.Hepatology200031864–871
27. Wiesner R, Edwards E, Freeman R, et al; United Network for Organ Sharing Liver Disease Severity Score CommitteeModel for end-stage liver disease (MELD) and allocation of donor livers.Gastroenterology200312491–96
28. Kamath PS, Wiesner RH, Malinchoc M, et al. A model to predict survival in patients with end-stage liver disease.Hepatology200133464–470
29. Botta F, Giannini E, Romagnoli P, et al. MELD scoring system is useful for predicting prognosis in patients with liver cirrhosis and is correlated with residual liver function: a European study.Gut200352134–139
30. Thuluvath PJ, Guidinger MK, Fung JJ, et al. Liver transplantation in the United States, 1999-2008.Am J Transplant2010104 Pt 21003–1019
31. Dutkowski P, Oberkofler CE, Béchir M, et al. The model for end-stage liver disease allocation system for liver transplantation saves lives, but increases morbidity and cost: a prospective outcome analysis.Liver Transpl201117674–684
32. Yi Z, Mayorga ME, Orman ES, et al. Trends in characteristics of patients listed for liver transplantation will lead to higher rates of waitlist removal due to clinical deterioration.Transplantation20171012368–2374
33. Raval Z, Harinstein ME, Skaro AI, et al. Cardiovascular risk assessment of the liver transplant candidate.J Am Coll Cardiol201158223–231
34. Xia VW, Taniguchi M, Steadman RH. The changing face of patients presenting for liver transplantation.Curr Opin Organ Transplant200813280–284
35. Schlegel A, Linecker M, Kron P, et al. Risk assessment in high- and low-MELD liver transplantation.Am J Transplant2017171050–1063
36. Waits SA, Kim EK, Terjimanian MN, et al. Morphometric age and mortality after liver transplant.JAMA Surg2014149335–340
37. Bambha KM, Dodge JL, Gralla J, et al. Low, rather than high, body mass index confers increased risk for post-liver transplant death and graft loss: risk modulated by model for end-stage liver disease.Liver Transpl2015211286–1294
38. Barone M, Viggiani MT, Losurdo G, et al. Systematic review with meta-analysis: post-operative complications and mortality risk in liver transplant candidates with obesity.Aliment Pharmacol Ther201746236–245
39. Dolgin NH, Martins PN, Movahedi B, et al. Functional status predicts postoperative mortality after liver transplantation.Clin Transplant2016301403–1410
40. Klein KB, Stafinski TD, Menon D. Predicting survival after liver transplantation based on pre-transplant MELD score: a systematic review of the literature.PLOS One20138e80661
41. Ghobrial RM, Gornbein J, Steadman R, et al. Pretransplant model to predict posttransplant survival in liver transplant patients.Ann Surg2002236315–322. Discussion 322
42. Bilbao I, Armadans L, Lazaro JL, et al. Predictive factors for early mortality following liver transplantation.Clin Transplant200317401–411
43. Bilbao I, Figueras J, Grande L, et al. Risk factors for death following liver retransplantation.Transplant Proc2003351871–1873
44. Stephenson GR, Moretti EW, El-Moalem H, et al. Malnutrition in liver transplant patients: preoperative subjective global assessment is predictive of outcome after liver transplantation.Transplantation200172666–670
45. Englesbe MJ, Patel SP, He K, et al. Sarcopenia and mortality after liver transplantation.J Am Coll Surg2010211271–278
46. Guaraldi G, Dolci G, Zona S, et al. A frailty index predicts post-liver transplant morbidity and mortality in HIV-positive patients.AIDS Res Ther20171437
47. Bleeker SE, Moll HA, Steyerberg EW, et al. External validation is necessary in prediction research: a clinical example.J Clin Epidemiol200356826–832
Supplemental Digital Content
Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.