Secondary Logo

Journal Logo

Developing CUSUM Charts for Monitoring Transplant Outcomes

Varied Goals and Many Possible Paths to Success

Snyder, Jon J. PhD1,2; Salkowski, Nicholas PhD1,2; Wey, Andrew PhD1,2

doi: 10.1097/TP.0000000000002590
Commentaries
Free

1 Chronic Disease Research Group, Hennepin Healthcare Research Institute, Minneapolis, MN.

2 Scientific Registry of Transplant Recipients, Hennepin Healthcare Research Institute, Minneapolis, MN.

Received 30 November 2018. Revision received 17 December 2018.

Accepted 19 December 2018.

This study was conducted under the auspices of the Hennepin Healthcare Research Institute, contractor for the Scientific Registry of Transplant Recipients, as a deliverable under contract number HHSH250201500009C (US Department of Health and Human Services, Health Resources and Services Administration, Healthcare Systems Bureau, Division of Transplantation). As a US Government-sponsored study, there are no restrictions on its use. The views expressed herein are those of the authors and not necessarily those of the US Government.

The authors declare no conflicts of interest.

J.J.S., N.S., and A.W. participated in the writing of the article.

Correspondence: Jon J. Snyder, PhD, Hennepin Healthcare Research Institute, 701 Park Avenue, Suite S-4.100, Minneapolis, MN 55415. (jsnyder@cdrg.org).

The field of organ transplantation, which is often regulated within each country, is fairly unique as a medical specialty in the amount of standardized data available for nearly all procedures performed. As such, we can observe vast improvements in allograft outcomes, including an 80% or greater reduction in kidney allograft failure rates from 1988 to 2014 across Australia, New Zealand, the United Kingdom, and the United States.1 Even with these improvements, unexplained variation remains among countries, suggesting that improvements can and should be sought. Standardized data collection across multiple transplant hospitals allows for detailed monitoring of patient and allograft outcomes, with the ability to risk-adjust for varying case mix across programs. In the United States, the Organ Procurement and Transplantation Network/Scientific Registry of Transplant Recipients (SRTR) Annual Data Report publishes up to 10-year trends in patient and allograft outcomes,2 and program-specific reports are published semiannually showing risk-adjusted outcomes for every transplant program.3

While national or program-level statistics are helpful for tracking progress in a broad sense, programs need as near real-time data as feasible to support quality improvement efforts and seek continued gains. Statistical process control charting methodologies, originally developed in the field of manufacturing, have recently been applied in transplantation to support near-real-time outcomes monitoring.4 In the United States, since 2014, SRTR has produced program-specific cumulative sum (CUSUM) charts allowing programs to monitor patient and allograft survival,5 following a recommendation stemming from a consensus conference to “provide transplant centers … with tools such as the cumulative sum (CUSUM) technique and tools to allow subgroup analysis to facilitate quality assessment and performance improvement.”6 SRTR has also begun to supply CUSUMs for organ offer acceptance to allow programs to benchmark their offer accept/decline practices relative to their peers. In the United Kingdom, CUSUM charts are supplied to all transplant programs to monitor 30-day allograft failure rates.7

In this edition of the journal, Alexandrine et al8 present a proposed CUSUM methodology to be implemented in the French transplantation system. Many variations of CUSUM methodology can be used, so critical thinking about each decision is imperative. Critical questions to guide the choice of appropriate methodology include:

  • Will charts be used by regulators/payers? If so, how?
  • How are signal thresholds determined? What are the consequences of a false-positive signal? What are the consequences of a false-negative?
  • Should the timescale be the transplant number or calendar time?
  • How is risk adjustment performed to account for case-mix?•What statistical methodology should be used? Will the risk adjustment gain acceptance among the providers being monitored?
  • Are programs compared with their peers nationally or with their own historical standard?
  • Upon a signal, should charts reset to resume monitoring from scratch, or reset to a “head-start”value?

This list presents only examples of the types of questions chart sponsors should consider. Table 1 summarizes differences in how these questions have been addressed in the United States, United Kingdom, and proposed French systems.

TABLE 1

TABLE 1

Perhaps the most important question to be addressed when choosing the appropriate methodology is how the reports will be used. If used for regulatory review, the consequences of a signal should be developed in advance. The scope of the regulatory action will inform the choice of an appropriate signaling methodology and subsequent threshold placement. For example, if the regulatory body will sanction or, in the extreme case, close a transplant program after a signal, a very low false-positive rate would be appropriate. However, a low false-positive rate necessarily means a higher false-negative rate, or, in other words, a higher likelihood that true changes in outcomes would be missed or take longer to detect. If, on the other hand, a signal simply initiates an internal review to identify whether changes are necessary, the program may be willing to accept a higher false-positive rate to increase the chance of detecting problems early.

The United Kingdom initiates a detailed review process upon a signal and notes that, “this approach makes it possible to use a chart with relatively high sensitivity (probability of a signal when performance has changed), so that the possibility of missing genuine changes in failure rates is minimized.”6 Similarly, the French system proposes to use an “optimal” signal threshold determined by attempting to maximize chart sensitivity and specificity for programs of differing volumes. This threshold has a relatively higher false-positive rate (10% to 40%), which the authors consider acceptable: “Performance of the optimal threshold, with sensitivity and specificity always exceeding 60% for all centers, and the resulting frequency of signals makes this threshold appropriate for the monitoring of French transplant centers.” Here, the authors point out that resources, of the program and of the sponsoring agency, are also an important consideration. A methodology that results in too many signals coupled with a high false-positive rate is not a viable system. The proposed French system also will implement a peer review process upon initial signal, so a slightly higher false-positive rate was deemed acceptable to the report sponsor.

CUSUM charts have played an increasing role in monitoring transplant outcomes. Increasing data availability and standardized data collection practices have enabled various countries to implement strategies that allow for interprogram or intraprogram benchmarking and have allowed regulatory agencies near real-time performance-monitoring capabilities. The current study by Alexandrine et al8 highlights the types of decisions that report sponsors should consider in developing systems that meet the stated goals of the agency and country in which the charts will be used. As SRTR has begun to release CUSUM charts for internal monitoring of organ offer acceptance practices in the United States, the application of these types of statistical process control charting methodologies can be expanded to other areas of transplantation as data allow, for example, surgical failure rates, rehospitalizations, or other adverse event rates. These advances will ultimately serve to improve transplant program processes, with the goal of ultimately saving and healing more lives through the gift of transplant.

Back to Top | Article Outline

ACKNOWLEDGMENTS

We thank Chronic Disease Research Group colleague Nan Booth, MSW, MPH, ELS, for article editing.

Back to Top | Article Outline

REFERENCES

1. Merion RM, Goodrich NP, Johnson RJ, et al. Kidney transplant graft outcomes in 379 257 recipients on 3 continents. Am J Transplant. 2018; 18(8):1914–1923
2. Organ Procurement and Transplantation Network/Scientific Registry of Transplant Recipients. OPTN/SRTR 2016 Annual Data Report. Am J Transplant. 2018; 18(Suppl 1):1–503
3. Scientific Registry of Transplant Recipients. Program-specific reports. Available at http://www.srtr.org. Accessed November 29, 2018
4. Axelrod DA, Guidinger MK, Metzger RA, et al. Transplant center quality assessment using a continuously updatable, risk-adjusted technique (CUSUM). Am J Transplant. 2006; 6(2):313–323
5. Snyder JJ, Salkwoski N, Zaun D, et al. New quality monitoring tools provided by the Scientific Registry of Transplant Recipients: CUSUM. Am J Transplant. 2014; 14(3):515–523
6. Kasiske BL, McBride MA, Cornell DL, et al. Report of a consensus conference on transplant program quality and surveillance. Am J Transplant. 2012; 12(8):1988–1996
7. Collett D, Sibanda N, Pioli S, et al. The UK scheme for mandatory continuous monitoring of early transplant outcome in all kidney transplant centers. Transplantation. 2009; 88(8):970–975
8. Alexandrine B, Savoye E, Pessione F, et al. Continuous monitoring of transplant center performance: different options for different goals. Transplantation. 2019; 103:1935–1944
Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.