Secondary Logo

Journal Logo

Liver Simulated Allocation Modeling

Were the Predictions Accurate for Share 35?

Goel, Aparna MD1; Kim, W. Ray MD1,2; Pyke, Joshua PhD2; Schladt, David P. MS2; Kasiske, Bertram L. MD2,3; Snyder, Jon J. PhD2,4; Lake, John R. MD2,5; Israni, Ajay K. MD, MS2,3,4

doi: 10.1097/TP.0000000000002079
Original Clinical Science—Liver
Free

Background The liver simulated allocation model (LSAM) can be used to study likely effects of liver transplant allocation policy changes on organ offers, acceptance, waitlist survival, and posttransplant survival. Implementation of Share 35 in June 2013 allowed for testing how well LSAM predicted actual changes.

Methods LSAM projections for 1 year of liver transplants before and after the Share 35 policy change were compared with observed data during the same period. Numbers of organs recovered, organ sharing, transplant rates, and waitlist mortality rates (per 100 waitlist years) were evaluated by LSAM and compared with observed data.

Results Candidate, recipient, and donor characteristics in the LSAM cohorts were similar to those in the observed population before and after Share 35. LSAM correctly predicted more accepted organs and fewer discarded organs with Share 35. LSAM also predicted increased regional and national sharing, consistent with observed data, although the magnitude was overestimated. Transplant rates were correctly projected to increase and waitlist death rates to decrease.

Conclusions Although the absolute number of transplants was underestimated and waitlist deaths overestimated, the direction of change was consistent with observed data. LSAM correctly predicted change in discarded organs, regional and national sharing, waitlist mortality, and transplants after Share 35 implementation.

LSAM has been used as a modeling tool for policy development by the OPTN since 2004. The performance of LSAM was evaluated to study the results of Share 35. Although transplants and waitlist mortality are underestimated, the direction of change is correctly predicted.

1 Division of Gastroenterology and Hepatology, Department of Medicine, Stanford University School of Medicine, Stanford, CA.

2 Scientific Registry of Transplant Recipients, Minneapolis Medical Research Foundation, Minneapolis, MN.

3 Department of Medicine, Hennepin County Medical Center, University of Minnesota, Minneapolis, MN.

4 Department of Epidemiology and Community Health, University of Minnesota, Minneapolis, MN.

5 Division of Gastroenterology, Hepatology, and Nutrition, University of Minnesota, Minneapolis, MN.

Received 18 July 2017. Revision received 25 October 2017.

Accepted 26 November 2017.

This work was conducted under the support of the Minneapolis Medical Research Foundation, contractor for SRTR, as a deliverable under contract no. HHSH250201500009C (US Department of Health and Human Services, Health Resources and Services Administration, Healthcare Systems Bureau, Division of Transplantation). As a US government-sponsored work, there are no restrictions on its use. The views expressed herein are those of the authors and not necessarily those of the US Government. W.R.K. was partially supported by DK 34238/9 and AKI was partially supported by R01 HS 24527.

The authors have no conflicts of interest to disclose.

A.G. was responsible for research design, data analysis, interpretation of data, writing of the article and critical revision of the article. W.R.K. was responsible for research design, interpretation of the data, and critical revision of the article for important intellectual content. J.P. was responsible for research design, data analysis, interpretation of the data and critical revision of the article for important intellectual content and statistical analysis. D.P.S. was responsible for research design and critical revision of the article for important intellectual content. B.L.K. was responsible for research design and critical revision of the article for important intellectual content. J.J.S. was responsible for data analysis and critical revision of the article for important intellectual content and statistical analysis. J.R.L. was responsible for research design and critical revision of the article for important intellectual content. A.K.I. was responsible for research design and critical revision of the article for important intellectual content.

Correspondence: W. Ray Kim, MD, Division of Gastroenterology and Hepatology, Stanford University, Rm M211, Alway Bldg, 300 Pasteur Dr, Stanford, CA 94305. (wrkim@stanford.edu).

Liver transplantation invokes a longstanding, complex debate regarding how best to allocate a scarce resource in an equitable and efficient manner.1,2 As the demand for organs far outstrips supply, patients often die while waiting for transplant. In 2012, more than 15 000 patients were on the liver transplant waiting list, nearly 6000 transplants were performed, and more than 2000 patients died on the waiting list.3

Policies related to organ allocation and distribution have evolved over 3 decades. Many of these policies are intended to improve the survival of waitlisted candidates, reduce waitlist time, and minimize subjectivity with regard to status assignment.1,4 After the Final Rule was issued by the US Department of Health and Human Services in 1998, liver allocation shifted from “first-come, first-served” to a “sickest first” prioritization.5 Initially, the Child-Turcotte-Pugh score was used to define illness severity, but due to its subjective elements, it was replaced by the model for end-stage liver disease (MELD) score in 2002.6-10 Analysis of data following MELD implementation indicated no net survival benefit to performing transplants in patients with MELD scores below 15, prompting the Share 15 regional policy.11

Sharma et al12 suggested that mortality of candidates with MELD scores 36 to 40 is similar to mortality of candidates listed as status 1a, arguing for a more urgent need for transplant. The so-called Share 35 policy, in recognition of the urgent need for transplant in high-MELD candidates with advanced hepatic decompensation, provides for regional organ sharing for patients with MELD scores above 35, similar to the way status 1a candidates were prioritized. The policy was implemented in June 2013.

As with prior policy changes, the Scientific Registry of Transplant Recipients (SRTR) was asked to study the likely effects of the Share 35 policy by using liver simulated allocation modeling (LSAM).13,14 LSAM is a discrete event-based computer simulation program that uses historical data to model organ offers, acceptance, MELD changes over time, waitlist survival, and posttransplant survival, and the uncertainty associated with these events.13 Although LSAM predictions have been used frequently to inform proposed policies in the past, opportunities to test these predictions against actual results after significant organ allocation and distribution policy changes are rare in liver transplantation. Such fact checking of LSAM is important in light of recent heavily debated policy changes regarding redistricting.

In this study, we aimed to describe the extent to which LSAM predictions reflect actual effects of policy implementation by comparing LSAM projections 1 year before and after Share 35 with observed data. The study period was limited to 1 year before and after Share 35 to mitigate the effects of other longitudinal trends in various aspects of transplantation that are independent of the policy change. The discrete period allowed us to more accurately determine whether LSAM can estimate the effect of an allocation policy change.

Back to Top | Article Outline

MATERIALS AND METHODS

Study Design

This study used data from SRTR. The SRTR data system includes data on all donors, waitlisted candidates, and transplant recipients in the United States, submitted by the members of the Organ Procurement and Transplantation Network. LSAM was implemented to simulate 1 year of liver transplants before and after implementation of the Share 35 policy. LSAM projections were compared with observed outcomes before and after the policy change. Observed data were extracted from SRTR standard analysis files for 2 periods, each covering 1 year pre- and post-Share 35 implementation on June 18, 2013. Although several years have passed since the policy change, observed data were limited to 1 year after Share 35 implementation to provide the most reliable estimation of the effects of the policy change. The Health Resources and Services Administration, US Department of Health and Human Services, provides oversight of the activities of the Organ Procurement and Transplantation Network and SRTR contractors.

Details of the simulated modeling process using the LSAM software have been described.13 Briefly, LSAM produces simulation results based on 3 components: (1) input data (ie, liver transplant candidates [existing candidates and new arrivals for a given year], donor organ arrivals, and changes in candidate status) and (2) rules and definitions (ie, allocation rules [current or proposed], ABO compatibility, and geographic definitions), to which (3) probability models (ie, organ acceptance and posttransplant survival) are applied. Candidate status is incorporated using a time-ordered input stream to describe changes to each candidate’s medical conditions and listing statuses that would occur if the candidate did not undergo transplant. Candidates’ medical condition can improve or deteriorate (also reflected as changes in MELD), or a candidate may die, leave the system, become temporarily inactive, or resume active status before undergoing transplant. Because the time of graft offer cannot be predicted, each candidate is included in the model from the time of listing until the first of death or the end of the simulated period.

The input data set used for this analysis consisted of liver transplant candidates on the liver transplant waiting list at any time between January 1, 2010, and December 31, 2010, and all organs offered during that same period. Using the same input file and probability models, pre- and post-Share 35 simulations used organ allocation priorities depicted in Table 1. LSAM simulations were repeated 10 times, each time randomly varying donor and candidate arrivals for each scenario; the mean and range of values are reported.

TABLE 1

TABLE 1

Back to Top | Article Outline

Data Analysis

The outcome parameters evaluated in the comparison between LSAM projections and the observed data included (1) number of organs recovered (transplanted and discarded), (2) proportions of organs transplanted within local, regional, or national geographic distributions, and (3) transplant rates and waitlist mortality rates (per 100 waitlist years). The transplant and waitlist mortality rates were stratified by status and MELD. Although additional outcome variables can be considered with LSAM, the study focuses on transplant rates and waitlist mortality rates, as these were the motivating factors for the Share 35 policy change. LSAM estimates in the observed direction of change and good agreement with observed values, subject to random variation, were considered accurate. We understand that this definition is subjective, but we believe it provides an appropriate degree of clinical meaningfulness. Unfortunately, traditional hypothesis tests cannot be applied directly to Monte Carlo simulation results, so we are unable to provide P values comparing simulated and observed data.15

Back to Top | Article Outline

RESULTS

Observed Data: 2010 Input and Pre- and Post-Share 35

The candidate and donor characteristics of the LSAM input data set and the observed characteristics 1 year before and after Share 35 were similar (Table 2). Candidate age, sex, and etiology of liver disease, and donor age, sex, height, and cause of death were comparable between the LSAM input data set and observed values. The proportion of donation after circulatory death donors did not differ significantly between cohorts. Given the stability in candidate and donor characteristics over the span of a few years, using the 2010 input data in LSAM is reasonable.

TABLE 2

TABLE 2

Back to Top | Article Outline

Organs Recovered, Transplanted, and Discarded

In the LSAM projection, the number of organs recovered was predetermined based on the input data. LSAM predicted an increase in the number of accepted organs and a decrease in the number of discarded organs (Figure 1). The acceptance rate would increase from 90.7% to 91.1% and the discard rate would decrease from 9.3% to 8.9% pre- and post-Share 35 (P = 0.43).

FIGURE 1

FIGURE 1

The observed data also showed an increase in organ acceptance and a decrease in organ discards. The acceptance rate increased from 89.9% to 90.5% and the discard rate decreased from 10.1% (678/6706) to 9.5% (665/7026) with Share 35 (Figure 1).

Back to Top | Article Outline

Regional Sharing

LSAM projected an increase in regional sharing with Share 35 from 28.6% to 33.5%; observed data showed an increase from 20.4% to 31.8% (Figure 2). LSAM also projected an increase in national sharing from 4.3% to 6.9%. This was in the same direction as the observed change, although the magnitude of observed change was smaller than predicted. Finally, with increases in regional and national sharing, LSAM projected decreases in the proportions of organs locally transplanted. The actual impact of Share 35 on local organ use was somewhat larger than LSAM’s prediction; the predicted reduction was 7.6 percentage points and the observed reduction was 11.6 percentage points.

FIGURE 2

FIGURE 2

Back to Top | Article Outline

Transplant Rates

LSAM projected a 46% increase in transplant rates for candidates with MELD scores 35 or above, from 862 to 1258 transplants per 100 waitlist years; a 36% increase from 1086 to 1478 was observed (Table 3). In general, LSAM underestimated transplant rates both before and after Share 35 across all MELD categories. However, the projected direction of change in transplant rates in each MELD category was consistent with observed data.

TABLE 3

TABLE 3

Back to Top | Article Outline

Waitlist Death Rates

LSAM projected a slight decrease in waitlist death rates for candidates with MELD scores 35 or above, from 105 to 100 deaths per 100 waitlist years; a decrease from 157 to 143 was observed (Table 4). In general, LSAM overestimated death rates in candidates with listed MELD scores 25 to 34. LSAM did not predict a notable change in death rates for candidates with MELD scores 15 to 24, consistent with observed data.

TABLE 4

TABLE 4

Back to Top | Article Outline

DISCUSSION

In this study, we compared LSAM projections with actual results of Share 35 policy implementation. Although the projections did not exactly match the observed results, LSAM correctly predicted the direction of change for virtually all parameters considered, including number of discarded organs, regional sharing, waitlist mortality, and transplants. Our results validate LSAM predictions with an observed cohort after a major organ distribution policy change.

Predicting the effect of organ allocation policies can be challenging given the complexity of transplantation. At any given moment, there are numerous changes in inputs (waitlisted candidates, new candidates, donated organs), candidate status (removal from the waiting list, inactivation, death, etc.), and transplant practices (organ allocation and acceptance).13,14,16 Allocation simulation offers a practical way to represent this dynamic system and inform much-debated policy changes. Such simulations have been performed for years, first with the University of Pittsburgh’s contract with CONSAD Research Corporation in 1993 to demonstrate the importance of broader sharing, subsequently with the United Network for Organ Sharing liver allocation model from 1995 to 2001, and most recently with LSAM.1,14,16

In its initial application, LSAM correctly predicted changes in transplant rates and waitlist mortality with the inception of MELD in 2002. In that study, Thompson et al13 compared simulation models with historical data from 6 months after MELD implementation and found that the direction of projected change was the same as observed, but LSAM estimates were more conservative, similar to our findings. Importantly, LSAM cannot predict changes in organ supply and human behavior resulting from policy changes. For example, LSAM predicted an increase in regional sharing of organs with Share 15; however, this was not observed, as transplant centers increased applications for MELD exception points and organs remained local.17 More recently, LSAM has been used to predict the effects of national Share 15 and regional sharing for high MELD scores at various cutoff points. The greatest reduction in waitlist mortality with the least distance traveled between donor hospital and transplant center was found with a national Share 15 and regional 35-32-29 policy.18

In the present study, LSAM correctly predicted the direction of change in most outcome categories of interest, although the magnitude of change was smaller than observed; that is, LSAM predictions were conservative in most cases. For example, LSAM underestimated transplant rates across all MELD categories with Share 35 but correctly predicted approximately 400 more transplants per 100 waitlist years among candidates with MELD scores 35 or above. The overestimation in transplant rates predicted by LSAM is likely due to the fixed organ supply built into the simulation. In the observed data, an increase in organ supply was noted after Share 35, which affected the overall magnitude of change in transplant rates (Figure 1). This change in organ supply is not possible to predict. Similarly, LSAM predicted that waitlist mortality would decrease in candidates with MELD scores 35 or above, and the observed mortality reduction was greater than predicted. LSAM also overestimated waitlist death rates across all MELD categories. Due to numerical differences in projected and expected rates, comparing LSAM simulations with other LSAM simulations (ie, before and after Share 35 simulations) is often more informative than comparing a simulation with observed data, as projected trends are more reliable than exact rates.

LSAM is a Monte Carlo simulation of the liver transplant allocation system. Two main classes of prediction error are made by this type of simulator: Monte Carlo error resulting from the random variation in inputs and random number generation, and bias resulting from the assumptions required to simplify a complex system and make it tractable for simulation. We addressed Monte Carlo error by running each simulation 10 times and taking the average result; this gives us confidence that our reported results are not the product of a single unlikely set of random number draws. Simplification bias is more difficult to address because many of the simplifications are important features of the simulation. For instance, we assume that offer acceptance behavior is the same at every transplant program in the country. This is certainly not true in practice, but we believe it would be a mistake to attempt to model each program’s acceptance practices individually; practices may change year to year as staff and circumstances change, and data to construct such a set of models would be quite limited in most cases. Instead, we use a single average model of acceptance behavior trained on match runs from across the country. These types of simplifying assumptions introduce some bias into the model predictions but they make the model better suited to simulate general outcomes under arbitrary proposed allocation rules.

Our study has several limitations. First, as noted above, LSAM does not account for changes in listing or acceptance behavior, both of which can significantly affect mortality and transplant rates. This is an important part of targeting the simulation to the effects of allocation policy change. Recent studies have evaluated changes in organ acceptance patterns and transplant characteristics with the implementation of Share 35.19-22 These studies indicate an increase in organ offers for patients with MELD scores above 35 but with lower acceptance rates, possibly indicating the desire for improved donor-recipient matching. Despite the conceivably more aggressive behavior in accessing organs, donor liver quality assessed by the donor risk index remained the same before and after Share 35. Although national transplant rates increased for patients with MELD scores above 35, less than one quarter of transplant programs accounted for nearly two thirds of this increase. Importantly, these programs were not concentrated in regions where the median MELD at transplant is above 35. The programs that experienced a marked increase in transplants saw a concurrent increase in new listings of candidates with MELD above 35, predominantly young patients with alcoholic liver disease.22 These program-specific changes in listing patterns that undoubtedly occur with allocation policy changes cannot be predicted with even the most mathematically sound simulation model. In the future, LSAM findings in conjunction with findings from behavioral economists, who have expertise in decision making under risk, may better inform these effects of policy change.23 Second, LSAM does not predict outcomes on a geographic basis because key model components are based on national probabilities. This helps LSAM model potential changes in the areas of organ distribution. Additionally, LSAM will never be able to predict future changes in organ supply, but variability in the supply is added through the donor generator. Lastly, LSAM predictions are based on data from 2010, which was the data cohort used for the original modeling of Share 35 during the policy development process. The durability of our results based on this data set is limited, especially in light of changes in donor pool characteristics from the opioid epidemic, listing practices, and emergence of liver transplant programs.24 An update to the LSAM input is currently underway to include data up to June 2016 with a 5-year cohort. This updated version of LSAM is expected to be publically available in 2017.

As noted by Washburn et al,20 potential behavioral changes should be a part of the discussion during policy development but should not necessarily determine the fate of a good policy. While LSAM is not perfect at predicting the magnitude of change a policy brings about, it can help guide well-intentioned policy proposals. In conclusion, our results indicate that LSAM offers insightful information on the potential impact of organ allocation policies and support its use to test proposed strategies for changing organ allocation and distribution.

Back to Top | Article Outline

ACKNOWLEDGMENTS

The authors thank SRTR colleague Nan Booth, MSW, MPH, ELS, for article editing.

Back to Top | Article Outline

REFERENCES

1. Hughes CB. The history of trying to fix liver allocation: why a consensus approach will never work. Clin Transplant. 2015;29:477–483.
2. Schwartz A, Schiano T, Kim-Schluger L, et al. Geographic disparity: the dilemma of lower socioeconomic status, multiple listing, and death on the liver transplant waiting list. Clin Transplant. 2014;28:1075–1079.
3. Israni AK, Zaun D, Rosendale JD, et al. OPTN/SRTR 2012 Annual Data Report: deceased organ donation. Am J Transplant. 2014;14(Suppl 1):167–183.
4. Olthoff KM, Brown RS Jr, Delmonico FL, et al. Summary report of a national conference: evolving concepts in liver allocation in the MELD and PELD era. December 8, 2003, Washington, DC, USA. Liver Transpl. 2004;10(10 Suppl 2):A6–A22.
5. Weimer DL. Public and private regulation of organ transplantation: liver allocation and the final rule. J Health Polit Policy Law. 2007;32:9–49.
6. Bernardi M, Gitto S, Biselli M. The MELD score in patients awaiting liver transplant: strengths and weaknesses. J Hepatol. 2011;54:1297–1306.
7. Kamath PS, Wiesner RH, Malinchoc M, et al. A model to predict survival in patients with end-stage liver disease. Hepatology. 2001;33:464–470.
8. Pritsker AAB. Life and death decisions: organ transplantation allocation policy analysis. OR/MS Today. 1998;25:22–28.
9. Kamath PS, McKusick MA. Transvenous intrahepatic portosystemic shunts. Gastroenterology. 1996;111:1700–1705.
10. Freeman RB, Wiesner RH, Edwards E, et al. Results of the first year of the new liver allocation plan. Liver Transpl. 2004;10:7–15.
11. Merion RM, Schaubel DE, Dykstra DM, et al. The survival benefit of liver transplantation. Am J Transplant. 2005;5:307–313.
12. Sharma P, Schaubel DE, Gong Q, et al. End-stage liver disease candidates at the highest model for end-stage liver disease scores have higher wait-list mortality than status-1A candidates. Hepatology. 2012;55:192–198.
13. Thompson D, Waisanen L, Wolfe R, et al. Simulating the allocation of organs for transplantation. Health Care Manag Sci. 2004;7:331–338.
14. Harper AM, Taranto SE, Edwards EB, et al. An update on a successful simulation project: The UNOS Liver Allocation Model. In. 2000 Winter Simulation Conference Proceedings (Cat. No.00CH37165); Orlando, FL. 2000 p. 1955–1962.
15. Kleijnen JPC. Verification and validation of simulation models. Eur J Operational Res. 1995;82:145–162.
16. Pritsker AAB, Martin DL, Reust JS, et al. Organ transplantation policy evaluation. In: Proceedings of the Winter Simulation Conference, WSC 2007, December 9–12, 2007. Washington, DC, USA: 2007:10.
17. Bittermann T, Makar G, Goldberg D. Exception point applications for 15 points: an unintended consequence of the share 15 policy. Liver Transpl. 2012;18:1302–1309.
18. Washburn K, Pomfret E, Roberts J. Liver allocation and distribution: possible next steps. Liver Transpl. 2011;17:1005–1012.
19. Edwards E, Harper A, Hirose R, et al. The impact of broader regional sharing of livers: 2-year results of “Share 35.” Liver Transpl. 2016;22:399–409.
20. Washburn K, Harper A, Baker T, et al. Changes in liver acceptance patterns after implementation of Share 35. Liver Transpl. 2016;22:171–177.
21. Goldberg DS, Levine M, Karp S, et al. Share 35 changes in center-level liver acceptance practices. Liver Transpl. 2017;23:604–613.
22. Murken DR, Peng AW, Aufhauser DD Jr, et al. Same policy, different impact: center-level effects of share 35 liver allocation. Liver Transpl. 2017;23:741–750.
23. Schnier KE, Cox JC, McIntyre C, et al. Transplantation at the nexus of behavioral economics and health care delivery. Am J Transplant. 2013;13:31–35.
24. Goldberg DS, Blumberg E, McCauley M, et al. Improving organ utilization to help overcome the tragedies of the opioid epidemic. Am J Transplant. 2016;16:2836–2841.
Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved.