Prominent medical accidents such as the 1995 Florida wrong leg case (1) or the March 2003 heart-lung transplant mismatch (2) shape public perception of healthcare and influence public policy (3, 4). Apparently simple and easily understood, these events serve as benchmarks for public discussions of safety and have effects that resonate for years (5).
Spectacular accidents are routinely followed by changes—typically additional layers of defenses—intended to assure stakeholders that “this will never happen again.” After the 2003 event, the United Network for Organ Sharing changed the matching process to increase the number of donor-recipient compatibility tests required and the way that certain unmatched situations are handled (6).
New defense effectiveness is difficult to quantify, especially if the accident rate is already low. Adding defenses to already heavily defended systems can only produce small increases in process reliability and may introduce new forms of failure elsewhere (7). Especially when the new defenses are costly, effortful, or add workload, quantifying the impact of proposed changes is essential to rational policy development. Progress on patient safety requires proactive searches for new forms of failure (8). The Joint Commission for Accreditation of Healthcare Organizations (JCAHO) now requires prospective risk analysis methods as part of organizational patient safety plans and procedures (9). The Institute of Medicine (IOM) report Patient Safety: Advancing a New Standard of Care (10) recommends research on methods for assessment of the validity and efficiency of integrating retrospective techniques (e.g., postaccident analysis) with prospective techniques.
Probabilistic risk assessment (PRA) is particularly suited to evaluating the likelihood of specific, rare forms of failure in stable, well characterized, heavily defended systems (11). PRA is a quantitative method for system risk assessment. It uses mathematical models to determine the effect of specific process faults on various types of failure (12). PRA models are often used to provide numerical estimates of the frequency of particular rare outcomes, e.g. various types of accident.
In addition to these estimates, PRA is a way to systematically explore a complex system and to express the results of that exploration. The resulting models can be used to assess the impact of countermeasures proposed after accidents and as the base for prospective analysis of the effects of technological and organizational change on the potential for different forms of failure, as recommended by the Institute of Medicine (13). These uses are particularly helpful in informing discussions of risk, especially when the risk involves low probability events with high value outcomes (e.g., the accidental release of radioactive materials from a nuclear power plant).
We report here the results of a PRA conducted in order to estimate the likelihood of unintentional ABO-incompatible thoracic organ transplant. A detailed study of transplantation technical work (5) was used as the basis for a PRA of the processes that could lead to an unintentional ABO-incompatible implant. The three resulting models were used to estimate the likelihood of such an event in the time up to the 2003 event, between March 2003 and October 2003, and after October 2003. The models provide a quantitative basis for: 1) evaluating efforts to reduce the likelihood of future accidents; 2) guiding the response to future events; and 3) monitoring the impact of other changes on the risk of this particular form of failure. The PRA also demonstrates both the value and limitations of this method and provides a model for other uses of the method in healthcare.
With institutional review board approval and appropriate consent, we developed a probabilistic risk assessment for solid organ transplantation processes with particular attention to the ABO blood type matching of donor and recipient for thoracic organs. PRA and transplant experts worked together to identify and characterize the socio-technical process of organ matching within the U.S. transplantation system. The stages of transplantation were divided and individually analyzed, paying particular attention to the handling of compatibility information. Stages of transplantation from original listing of recipients and donors through to actual implant were observed and transplant personnel interviewed regarding elements of the work processes.
Fault trees for critical activities and event trees for the possible outcomes (14) were developed to reflect the work activities of donor-to-recipient matching. The critical activities related to the implantation of ABO-incompatible organs are the checking and rechecking of ABO compatibility between donor and potential recipients. Based on direct observation, debriefing of participants, and review of written documentation, the authors modeled ABO processing during procurement and implantation and constructed fault trees for these activities. Event trees were created to encompass the process and potential outcomes. The event trees include probability terms derived from empirical data and from the fault trees. The modeling was tailored to represent known forms of failure and to permit modeling of countermeasures and their effects. Model components were iteratively validated by experts from the transplant community including surgeons, nurses, transplant coordinators, technicians, and transplant organ procurement experts. Model calculations were performed using a PRA computer code frequently used for nuclear safety applications (15).
The PRA model event tree is shown in Figure 1. Transplantation matching is modeled as a series of five events, each with a binary outcome. Each event corresponds to an activity or set of activities in the U.S matching process in 2003. The event tree reads from left to right and represents the possible sequence of events leading to one of the three outcomes. The model includes events reflecting: 1) the acceptance of either a conventional match-list directed offer versus placement through an “open” offer; 2) local procurement and implantation versus transfer of the organ across OPO boundaries; 3) use for the recipient identified at the time of the match versus use for some other recipient; 4) determination that the organ is suitable for use versus declining to use the organ; and 5) confirmation testing of ABO compatibility versus no confirmation.
The likelihood of a single organ passing through a given pathway is simply the product of the probabilities of the branches in that pathway. The model is computed by calculating the product for each pathway and summing the results for the distinct outcomes. In this model, there are three distinct end states: 1) implantation of an ABO-compatible organ (compatible); 2) implantation of an ABO-incompatible organ (incompatible); and 3) any other use not involving implantation into a patient (unused). The likelihood of each outcome is the sum of the different ways in which that outcome can be produced. Thus the likelihood of a compatible implant is the sum of all the ways that compatible implantation can be produced, the likelihood of an incompatible implant is the sum of all the ways that incompatible implantation can be produced, and so on in the model.
The PRA model was populated with probability estimates derived from operational experience and fault trees. Probability estimates for selected events, such as use of an “open offer” to place an organ, were assigned based on review of experience within a large organ procurement organization and verified by comparison with national data on transplantation (16). Probabilities for fault trees involving human performance were derived from existing human reliability analysis data for similar tasks (17–19).
The processes used to check for ABO compatibility between donor and recipient were modeled as fault trees to provide estimates of faults in compatibility testing (11). Fault trees in the base model reflect processes in use prior to March 2003 (Fig. 2). After the March 2003 event, the United Network for Organ Sharing (UNOS) changed the ABO compatibility assurance processes to reduce the likelihood of an incompatible organ being implanted (20). Further changes were made in October 2004 (6). The base model fault trees were modified to reflect the conditions in place between March 2003 and October 2004 (Fig. 3), and after October 2004 (Fig. 4).
The three PRA models were used to estimate the expected frequency of unintentional ABO incompatible thoracic organ transplantation prior to March 2003, between March 2003 and October 2004, and from October 2004 onwards. Chi-square tables were used to estimate confidence intervals for the model predictions using the method of Kumamoto and Henley (14).
For comparison purposes, the observed frequency of unintentional ABO-incompatible thoracic organ transplant was calculated as follows. Data for numbers of organs transplanted were derived from UNOS data available through OPTN (16). Reports of thoracic organ transplant accidents were sought using proprietary research databases (21).
Finally, a detailed event sequence for the most recent event was reconstructed (22) using press reports, statements released by participating organizations, and publicly available sources was then analyzed and compared with the PRA model.
The PRA model results are shown in Table 1. The rate of unintentional implantation of ABO-incompatible thoracic organs predicted by the pre–March 2003 PRA model is 1.38×10−5/organ explanted, which is roughly one event for every 72,000 explants (1:72,000).
The rate from probabilistic risk assessment compares favorably with the actual experience with unintentional ABO-incompatible thoracic organ transplant. There were 28,625 thoracic organs recovered between 1995 (the year of the MDS accident in Oregon) and 2003 (the year of the JS event). This corresponds to an actual occurrence rate during that period of 1:28,625 or 3.49× 10−5/organ transplanted.
In addition to the March 2003 event, there are known to have been two instances of unintentional ABO-incompatible thoracic organ transplant in Oregon, one in 1991 and the other in 1995. OPTN public reports show that 46,993 thoracic organs were implanted during the period from 1988 through 2003. Together these yield a rate of 3:46,993 (6.38×10−5/organ transplanted).
The changes to the donor-recipient matching process undertaken by UNOS in the wake of the March 2003 event increased the number of checks at various stages of the testing and recording process. Each change reduced the likelihood of this type of accident approximately one order of magnitude. The PRA models indicate that the first intervention reduced the likelihood of unintentional ABO-incompatible thoracic organ implantation to 3.08×10−6/organ explanted (1:320,000 explants). The second further reduced the likelihood to 2.22×10−7/organ explanted (1:4.5 million explants).
The PRA results can be used to estimate the time to occurrence of another unintentional ABO-incompatible thoracic organ implantation. These values are shown in Table 2. Assuming that thoracic organ recovery rates remain at 2006 levels, the PRA model of current process predicts that an ABO incompatibility event will occur approximately once every 1,251 years (95% confidence interval 339 to 49,408 years).
The 2003 Event in PRA Context
Analysis of reports describing the March 2003 event identify this failure as a byproduct of an “open offer” of the donor heart-lung. Matching donor organs to recipients is coordinated via a computer-generated list of candidates. This “match list” is created by an organ procurement organization (OPO) using a UNOS computer program to select and display candidate recipients from a single, national database. The list is rank ordered, the rank of each potential recipient being determined by both technical (e.g., organ size) and social (e.g., regional preference) criteria. The donor-side organ procurement coordinator is obligated to offer donor organs to recipient-side coordinators in the order specified by the list. Failure to successfully negotiate assignment of organs to specific recipients may lead the donor-side coordinator to offer organs to specific transplant centers rather than individual patients, allowing the transplant center to choose the recipient (an “open offer”). This approach can be used to match donor and recipients efficiently, especially when the initial list of potential recipients is exhausted.
In 2003, open offers were used to “place” nearly one half of thoracic organs. Although the ABO compatibility between organ and recipient was checked, the process was not as robust as when the organ use was directed through the computer-generated match list. The countermeasures put in place after the March 2003 event changed the “open” offer process by adding cross checks for ABO compatibility as data entered into the UNOS system. The countermeasures put in place in October 2004 require that the procedure for ABO compatibility testing be the same for both open and closed offers.
The PRA results show that unintentional ABO- incompatible organ implantation was already highly unlikely at the time of the March 2003 event. At that time, the process of assuring ABO compatibility along the open offer pathway was slightly less reliable than the process used for the match list-directed pathway. These two pathways are used roughly equally often. The PRA model estimate of the rate of unintentional ABO-incompatible thoracic organ implantation was 1.38×10−5 per organ explanted. The estimate is within a factor of ∼3 of the observed accident. The PRA models indicate that the countermeasures now in place have further reduced the risk of such an event by about 60-fold. The risk of a transplant patient receiving an unintended ABO incompatible thoracic organ now (1 per 4.5 million organs explanted) is on the order of the risk of that same individual being killed by lightning in the next year (1 per 5.6 million) (23) and four orders of magnitude lower (that is, ∼1/10,000th) than other risks associated with thoracic organ transplantation.
PRA modeling is one of the few quantitative methods available for assessing the impact of countermeasures when the frequency of an event is already low. It is for this reason that it has been used for estimating the reliability of highly reliable processes such as nuclear power plant operations (24, 25) and space missions (26). As in these applications, the reliability of human performance contributes substantially to the overall reliability of the transplantation system. Estimates of the reliability of human performance used in this PRA were derived from the sources used for these estimates in other applications with similar process elements. Subtle distinctions between types of human performance failure, such as “slips” versus “mistakes” (27, 28) challenge analysts preparing PRA models (17). In the present model, transcription and data entry are estimated to have local failure rates of between 1:100 and 1:1000, comparable to failure rates for such activities in nuclear power operations (18).
The results obtained in this and other applications of PRA should be interpreted with care. PRA probability estimates may be artificially low if “common mode” (sometimes called “common cause”) failure pathways are present in the system (11). An example would be a poorly written ‵1′ or ‵9′ being read as ‵7′ in two independent reviews. There may be unmodeled process failures that reduce the reliability of the process. Another limitation is the assumption of independence of events. When people are required to check another’s work, several behaviors tend to act to reduce the effectiveness of the checking process and increase the probability of failure. If no failures have been discovered for a while, for example, production pressure may lead to reduction of the independence of these tests and a corresponding reduction in the reliability of that portion of the process. Provided that histocompatibility lab operations remain independent of other lab results, and if the checking by a third party is carried out with the typical reliability of human behavior (19), then the probability of failure derived from this PRA may be considered representative. The fact that this high reliability depends critically on the independence of these checks may provide added incentive to sustain that independence.
In addition to providing estimates of likelihoods, the PRA analysis draws attention to the reliability of the primary and alternate pathways for negotiating the “placement” of a donated organ. When a donor is identified, the organ procurement coordinator enters biological information about the donor into the UNOS national computer system and uses this system to generate a list of compatible recipients for each organ being donated. This “running the match list” produces a rank-ordered list of potential recipients. In 2003, the “match list” was usually provided (by facsimile) to the donor-side transplant coordinator who used the list to guide the process of finding a suitable recipient. The list was structured to support the donor-side coordinator in identifying the recipient. Then as now, the computer programs incorporated the rules for organ sharing. These rules are intended to assure the equitable distribution of organs across patients and across transplant centers. Using the list to identify the recipient guarantees that the rules of organ sharing are obeyed.
In 2003, the donor-side coordinators offered the organ to individual recipient-side transplant coordinators until the organ was accepted for a listed individual. Recipient-side coordinators, in cooperation with the transplant center surgeons, could either accept or reject the offer. Although the need for organs was always high, there are several reasons why an offer for a specific recipient might be rejected. The potential recipient may have been too ill for implantation, the organ may (in the opinion of the surgeon) not have been not well suited to the patient, the surgical team may not have been able to perform the transplantation procedure, and so forth. If the offer was accepted, the process of organ procurement and transplantation began. If the offer was rejected, the donor-side coordinator moved to the next listed potential recipient on the match list and the procedure was repeated until an offer was accepted. A single cycle of this process might have taken as little as 30 min or as long as 1.5 hr. The entire process could consume substantial time.
For about one-half of thoracic organ donations, this process did not result in a match. There are two main reasons for failure to match. In some cases, the match list contained only a few potential recipient names and the match list was exhausted without a match occurring. In other cases severe time pressure pushed the coordinator to generate a match quickly because of uncertainty about how long the organs would remain viable, such as when the donor was hemodynamically unstable. In such situations, the donor-side coordinator could make “open” offers in order to “place” the organ. In an open offer, the donor-side coordinator offered the organ to transplant centers for use for any patient registered at that center. The open offer allowed recipient-side transplant coordinators to select any compatible potential recipient to receive the organ.
The open offer pathway functioned as an escape valve for the organ transplantation process by relieving that process of the burden of assuring equity in the distribution of organs. Faced with the prospect of not implanting the donated organs at all, the open offer pathway helped the system achieve its primary goal of implanting every available organ in a compatible patient. Note that using the open offer pathway did not necessarily lead to the inequitable distribution of organs; it simply did not provide the strong assurance of equity of the match list pathway.
Significantly, the open offer reversed the roles of the two coordinators and the direction of information flow in the system. In the match list-directed process, the donor-side transplant coordinator proposed a specific recipient to the recipient-side coordinator who accepted the proposal. In contrast, during an open offer, the recipient-side coordinator proposed specific recipients to the donor-side coordinator who accepted the offer. The open offer pathway required additional communications work between coordinators, and sometimes between coordinators and the UNOS center, in order to complete the match.
At the time of the 2003 event, the open offer pathway was slightly less reliable than the match list–directed pathway in assuring ABO compatibility because there were fewer tests for compatibility in the open offer pathway. In a sense, the open offer’s increased vulnerability to this type of failure was the price paid for a system capable of placing organs quickly when the formal matching process was unworkable or threatened to be too slow. The PRA provides an estimate of this price: one in every 72,000 explantations could be expected to result in an unintentional ABO organ implantation.
The countermeasures put in place after March 2003 were intended to reduce the open offer pathway’s susceptibility to incompatible ABO failures. These countermeasures made the open offer process slightly more cumbersome. This, in turn, reduces the value of the open offer as a safety valve when donor organ viability is threatened. Extending the time required for placement may contribute to a higher incidence of posttransplantation complications such as primary nonfunction, infection, ischemic damage, or decreased graft-survival or even lead to an increase in the number of otherwise viable organs that go unused. Although undoubtedly small, the impact of such side effects may be the same order of magnitude as the reduction of risk derived from the countermeasures adopted in the wake of the 2003 accident.
The topic of open offers is controversial and stakeholders are at odds about what its existence implies. The purpose of PRA modeling is not to resolve such controversies but to illuminate them and to make the discussion of them more incisive and productive. PRA can help characterize the positions of stakeholders and the assumptions on which these positions rest.
The Role of PRA in Healthcare Risk Assessment
PRA allows quantitative evaluation of the reliability of some selected aspects of complex systems. The technique was developed because human endeavors that require deliberate encounters with catastrophic outcomes usually have a degree of complexity that makes quantitative assessment difficult. The complexity of these systems arises partly from layered defenses, such as those seen in transplantation organ matching. Success at reducing failure rates makes further reductions harder and more expensive. A low failure rate makes empirical determination of the actual rate and cost-benefit analysis of proposed changes difficult. With very low failure rates, even the minor side effects of well-intentioned changes may have paradoxical consequences for safety: the reduced rate of one type of failure may be offset by an even greater increase in other forms of failure. The process also provides parameters useful for other quantitative assessments, such as calculating the cost of preventing a single event.
PRA is one way to methodically explore the quantitative relationships that lead to success and failure of systems. By making the relationships and assumptions about event probabilities explicit, the use of PRA may encourage rational debate and inform public policy making. Healthcare applications of appropriately performed PRA are one way of exposing the costs and benefits of efforts to reduce risk and improve patient safety.
The authors gratefully acknowledge the contributions of Karyl Hildebrand, Joe Jachimiaka, Velta Lazda, Dean Lichtenfeld, Christopher Nemeth, Sandra Nunnally, Michael O’Connor, Evelyn Schultz, Chris Williams, and organ recovery coordinators and staff from the Gift of Hope Organ and Tissue Donor Network and the University of Chicago Medical Center.
1. Doctor Who Cut Off Wrong Leg Is Defended by Colleagues. New York Times
, New York: Sept 17, 1995: 28.
2. Dodd S. 17-year-old girl near death after botched transplant. Knight Ridder Tribune News Service.
Washington: Feb 18, 2003: 1.
3. Claims Doctors Cut Off Wrong Leg; Files Suit. Chicago Daily Tribune
, November 30, 1952: 14.
4. Congressional Record
, 1995 (Senate); 144: S5779.
5. Cook R, Woods D, Miller C. A Tale of Two Stories: Contrasting Views of Patient Safety. Chicago: National Patient Safety Foundation; 1998.
6. UNOS Policy 3.1. Available at http://www.unos.org/policiesandbylaws/policies.asp
. Accessed November 12, 2004.
7. Cook R, O’Connor M. Thinking about Accidents in Systems. In Manasse H, Thompson K, eds. Medication Safety: A Guide for Health Care Facilities. Bethesda, MD: American Society of Health-System Pharmacists, 2005: 73.
8. Woods D, Cook R. Nine steps to move forward from error
. Cognition Tech Work
, 2002; 4: 137.
9. O’Leary D. Patient Safety: Instilling Hospitals with a Culture of Continuous Improvement. Available at: http://www.jcaho.org/news+room/on+capitol+hill/061103_testimony.htm
. Accessed August 20, 2005.
10. Aspden P, Corrigan JM, Wolcott J, Erickson SM, eds. Patient Safety: Achieving a New Standard for Care. Washington DC: National Academies Press; 2004: 19.
11. Wreathall J, Nemeth C. Assessing risk: The role of probabilistic risk assessment
(PRA) in patient safety improvement. Qual Saf Health Care
2004; 13: 206.
12. Battles JB, Kanki BG. The use of socio-technical probabilistic risk assessment
at AHRQ and NASA. In Spitzer C, Schmocker U, Dang VN, eds. Probabilistic Safety Assessment and Management. London: Springer Verlag, 2004: 2212.
13. Kohn L, Corrigan JM, Donaldson MS, eds. To Err is Human: Building a Safer Health System. Washington DC: National Academy Press, 1999: 174.
14. Kumamoto H, Henley EJ. Probabilistic Risk Assessment
and Management for Engineers and Scientists, 2nd ed. New York: IEEE Press, 1996.
15. SAPHIRE: Systems Analysis Programs for Hands-on Integrated Reliability Evaluations (NUREG/CR-6116). Boise: Idaho National Engineering and Environmental Laboratory, 2001.
16. OPTN National Data Report. Available at: http://www.optn.org/latestData/viewDataReports.asp
. Accessed November 12, 2004.
17. Wreathall J. Issues in Selecting a Satisfactory HRA Method. In: Kauffer B, ed. OECD/NEA Workshop on Errors of Commission 2001. Rockville, MD: OECF, 2001.
18. Swain AD, Guttmann HE. Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications (NUREG/CR-1278, Rev. 1). Albuquerque, NM: Sandia National Laboratories, 1983.
19. Gertman DI, Blackman HS. Human Reliability & Safety Analysis Data Handbook. New York: John Wiley & Sons, 1994.
20. OPTN/UNOS Review of Transplant Error
Continues. Available at: http://www.optn.org/news/newsDetail.asp?id=251
. Accessed November 12, 2004.
21. FACTIVA, Dow Jones & Reuters News Services. Available at: http://www.factiva.com
. Accessed October 19, 2004.
22. Cook RI. Hobson’s choices: Matching and mismatching in transplantation work processes. In: Wailoo K, Livingston J, Guarnaccia P, eds. A Death Retold: Jesica Santillan, the Bungled Transplant, and Paradoxes of Medical Citizenship. Chapel Hill, NC: University of North Carolina Press, 2006: 46.
23. U.S. Census Bureau. Statistical Abstract of the United States: 2003 (123rd Ed). Washington, DC: U.S. Census Bureau; 2003.
24. Nuclear Regulatory Commission. WASH-1400, Reactor Safety Study – An Assessment of Accident
Risks in U.S. Commercial Nuclear Power Plants (NUREG-74/04). Washington, D.C.: Nuclear Regulatory Commission, 1975.
25. Nuclear Regulatory Commission. Severe Accident
Risks: An Assessment for Five U.S. Nuclear Power Plants (NUREG-1150). Washington, D.C.: Nuclear Regulatory Commission, 1989.
26. Frank MV. Probabilistic analysis of the inadvertent reentry of the cassini spacecraft’s radioisotope thermoelectric generators. Risk Analysis
2000; 20: 251.
27. Norman DA. Categorization of action slips. Psychol Rev
1981; 88: 1.
28. Reason J. Human Error
. New York: Cambridge University Press, 1990.