Recently, there have been calls for an increased focus on pretransplant outcomes and other patient-centered metrics in public reports of transplant center outcomes.1-5 Best practice guidelines for healthcare reporting reinforce the need to not only provide information but help users interpret information effectively in decision-making.6,7 For a transplant candidate (hereafter referred to as candidate), the feasible number of choices for transplant centers (hereafter referred to as centers) vary by region, organ type, and individual circumstances; however, many patients report having multiple choices available,8 chose centers far from their homes9,10 or list at multiple centers.11 Currently, public reports can be used to compare pretransplant and posttransplant outcomes, for example waitlist mortality, transplant rate,12 and posttransplant organ survival.13 The relevance of outcome metrics is organ-specific, and some center outcomes are associated with future patient mortality.14-16 There is a potential need for patients to consider trade-offs between pretransplant and posttransplant outcomes, or any future quality metric, when deciding on a center. Existing studies on patient priorities have used only abstracted center information such as text-based surveys8,17 and do not test decisions when viewing data. There is little understanding of how the content and presentation style of current and new information will influence decision-making.
A mixed methods approach combines qualitative (eg, focus groups) and quantitative (eg, randomized trial) methods and has been utilized in related fields to improve the usability of public healthcare reports,18-20 such as reports to aid in choosing a hospital.21 This approach can be an advantage over the use of a single method,22,23 for example, reporting if multiple methods produce consistent findings. These prior studies in related fields demonstrate significant effects of presentation style and content on decision-making, suggesting a need to evaluate reports specifically relevant to organ transplantation.
A better understanding of decision-making can inform how information is presented to patients and will be an important consideration as new quality metrics are evaluated. This study first used focus groups of transplant patients to develop and refine potential reports of center outcomes and understand patient perceptions when viewing information. A subsequent randomized survey of nonpatients was used to evaluate the effects of graphical presentations of information on decision-making. The 2 combined study phases provided a feasible approach to understand how information about centers can change patient perceptions and also statistical differences in decision-making.
MATERIALS AND METHODS
First, in-depth patient feedback informed iterative development of new graphics, content, and survey instruments. Second, evaluating multiple graphical presentations to identify potential impacts on decisions required a large sample that is not feasible when recruiting patients. Both study phases were approved by the human subjects research review board at each recruiting site. Additional detail about software and statistical methods is provided in Supplemental Materials, http://links.lww.com/TP/B755.
Qualitative Phase: Focus Groups
The qualitative phase included focus groups of local and national adults (age 18 or older) stratified into candidates and recipients. Local participants were transplant candidates at Hennepin Healthcare (HHS) and the University of Minnesota-Fairview (UMN). National participants were recipients who were healthy enough to travel. All groups were organ-specific and included kidney, liver, heart, or lung transplant patients. Non-English speaking candidates were excluded. Each participant gave written, informed consent, and participants received a $40 stipend.
For each organ, candidate focus groups were conducted first and located at HHS and the UMN clinics in the Upper Midwestern United States. Inclusion criteria for a convenience sample were English speaking, on the transplant waiting list, or proceeding with an evaluation. Recruiting was conducted by research coordinators during or after a regularly scheduled transplant appointment or by mail.
National transplant recipient focus groups were conducted in Chicago, IL. Participant expenses were paid, including airfare and lodging. Participants were recruited through email and social media via national patient groups (eg, National Kidney Foundation). Inclusion criteria for national groups were English speaking and a previous transplant (or retransplant). National participants were purposively recruited to ensure there was at least 1 participant for each organ type from most Organ Procurement and Transplantation Network regions.
Focus groups were moderated by the same individual (C.R.S.), trained in qualitative research, and attended by a social sciences researcher (M.J.B.) and transplant physician (A.K.I). The focus groups were described as an opportunity to learn about patient experiences when choosing a center and discuss feedback about new graphics and information displays. A discussion guide included mockups and questions (Table S1, SDC, http://links.lww.com/TP/B755) based on a phenomenological approach to understand how patients choose a center.24 The scope of the current analysis includes only discussions in which participants viewed mockups of public reports and described decisions based on the information provided. Discussions outside this scope, such as past experiences choosing a center, have been reported elsewhere.25 Mockups were developed iteratively in response to feedback; therefore, some concepts with consistent negative feedback were replaced with new iterations during the qualitative phase. All mockups shown during discussions of trade-offs included realistic pretransplant and posttransplant metrics.
Focus groups were 90–120 minutes. All participants completed a demographics and comorbidity questionnaire. Discussions were audio recorded and transcribed verbatim. All transcripts were read and coded by one analyst (C.R.S.), and 4 focus groups were independently coded by a second analyst (W.T.M.). Each researcher (C.R.S. and W.T.M.) reviewed each other’s coding schemes and discussed discrepancies in applying codes. Discrepancies that were not resolved were reviewed by A.K.I. and M.J.B. to reach consensus. Transcripts were open coded and axial coded, and themes emerged through an inductive, thematic analysis.24,26
Qualitative patient feedback about mockups informed the selection of several final design elements to test in a randomized survey. Each final design element was selected based on evidence of effectively helping patients interpret the content of reports (compared to other alternatives discussed in focus groups).
Quantitative Phase: Randomized Survey
Internet users from the United States were recruited online using Amazon Mechanical Turk (AMT) (www.mturk.com, Amazon.com, Inc. Seattle, WA) to evaluate potential concepts for new online reports. AMT is a platform to post short tasks to an online workforce. Any member of the AMT platform who met the following criteria was eligible to participate: US internet (IP) address, a 95% rate of accepted work on the AMT platform, and at least 50 completed tasks that did not include similar prior transplant surveys. Each participant consented to the survey electronically; written consent was waived because participants were anonymous. Participants received nominal payments (eg, $0.50) for the completion of the survey.
Surveys were provided in English. Survey participants answered 5 demographic questions and a series of questions related to organ transplantation, specifically to understand perceptions of risk (questions and response choices are shown in Table S2, SDC, http://links.lww.com/TP/B755). Survey questions were piloted in a previous randomized survey (before the development of a 5 tier transplant rate measure).27 Each participant viewed a randomly assigned image depicting a mockup of transplant center search results using graphical elements similar to those used on recent Scientific Registry of Transplant Recipients search tools (www.srtr.org). Mockups depicted hypothetical centers and data using distance and transplant volume as well as 5 tier assessments of waitlist mortality, transplant rate, and 1-year organ survival. The hypothetical center choices were designed to minimize a bias towards posttransplant outcomes. In particular, the center with the highest 1-year organ survival was also the greatest distance, and choices were sorted with the highest transplant rate at the top. The survey tested the hypothesis that adding context to these mockups would reduce the bias towards 1-year organ survival described in focus groups and seen in previous pilot survey tests.
Three graphical elements were evaluated for effects on decision-making, and combinations of multiple elements were used based on a factorial study design with 2 levels for each factor (described in more detail in Table S3, SDC, http://links.lww.com/TP/B755) with equal allocation in each arm. A control was included in which all new elements were omitted. The graphical elements included (1) a data table to provide numerical values corresponding to each tier (eg, a typical rate for 1-y organ survival at a tier 3 center was 91%); (2) an impact statement where an icon was shown next to the transplant rate heading and accompanied by a statement: “For transplant candidates, this measure has the largest impact on survival after listing. 1-year organ survival includes only candidates who received a transplant”; and (3) an uncertainty statement below metric headings: “Note: Future outcomes may be different than past outcomes. Groups (number of bars) do not represent the outcome for each individual patient.” Figures 1–3 include representative examples of the 6 mockups. All mockups included 3 identical hypothetical center options, where each of the 3 centers was shown as having the best rating in only 1 of the 3 included metrics (eg, Lake Hospital had the best rating for transplant rate).
Images were presented using a Qualtrics survey (www.qualtrics.com), which includes a function to randomize questions. After viewing a randomly assigned image, survey participants indicated which center they would choose based on the information shown. Participants also indicated which factor was most important in the decision (or “no single factor”). The target survey duration was 5–10 minutes. The survey for image mockups was available online between 27 February 2018 and 1 March 2018.
The primary outcome was the choice of center (see question #7, Table S2, SDC, http://links.lww.com/TP/B755). A multinomial regression evaluated the effect of the different presentations on the choice of center, because the outcome variable (ie, the selected hospital) had >2 unordered levels. Specifically, the analysis estimated the difference in relative risks (RRs) and marginal RR and 95% confidence intervals for selecting Alpine Hospital and Lake Hospital compared to Meadow Hospital (Meadow Hospital was depicted as the highest 1-y organ survival). The regression only included the main effects of the design elements because the likelihood ratio test for the interaction was not significant (P = 0.976) and adjusted for the perception of risk (response to question #6). See the Supplement for additional details on the methodology.
Qualitative Phase: Focus Groups
A total of 127 candidates and recipients participated in organ-specific focus groups: 62 kidney, 23 liver, 17 heart, and 20 lung (demographics shown in Table 1). Table 2 describes the number of local and national participants and time period for each organ type.
The development of mockups was informed by feedback on concepts with less explicit guidance about relevant risks and also recent research indicating organ-specific metrics were correlated to survival.14,15 For example, transplant rate has the largest impact on survival from waitlisting for kidney, liver, and heart; 1-year organ survival has the largest impact for lung. Therefore, direct statements, such as the “impact statement” (Figure 2), were used in later focus groups and well received. Mockups not selected for the randomized survey included patient narratives, general descriptions of the importance of both pretransplant and posttransplant outcomes (without explicitly highlighting a single outcome metric), and alternate headings and definitions of outcome metrics.
Three themes are described, including supporting quotations. Themes reflect general patient perceptions rather than responses to individual mockups: (1) outcome metrics have uncertainty relative to individual experiences; (2) patients, in particular candidates, describe a focus on posttransplant outcomes; and (3) individual circumstances factor into decisions. Table S4, SDC, http://links.lww.com/TP/B755 includes additional quotations illustrating the main themes.
Theme 1: Outcome Metrics Have Uncertainty Relative to Individual Experiences
Some participants described the pretransplant outcome of transplant rate as having limited utility due to perceived high levels of uncertainty, often resulting from random chance and high variability for the wait time to receive a deceased donor organ. Waitlist mortality was described as not consistently within the control of a center. Although not as frequently discussed, uncertainty in posttransplant outcomes was similarly described as reflecting deaths that were unrelated to transplant or outside the control of a center.
“Because you can have the most phenomenal medical center and there can still be issues. It’s going to be each individual experience.” [Local kidney candidate]
“I always qualify to the patients that an average is just an average... In our area, the average wait time for a liver transplant is 1 year. I waited two and a half years. A colleague of mine waited 8 days.” [Liver national recipient]
“So the statistic is kind of blurred if it’s including someone that died of other causes like a bike accident, a car accident.” [National heart recipient]
“Surviving on a waitlist, I mean, that’s just the roll of the dice.” [Lung national recipient].
Theme 2: Patients, in Particular Candidates, Describe a Focus on Posttransplant Outcomes
Participant points of view differed when making trade-off decisions; however, candidates appeared to more commonly prioritize a center with high posttransplant outcomes, and recipient decisions appeared more mixed. Table S4, SDC, http://links.lww.com/TP/B755 includes additional quotes and also supporting quotes with a focus on pretransplant outcomes.
“The survival afterwards is pretty much more important if it takes 2 years or 10.” [Local kidney candidate]
“I was thinking about survival. Mine was all survival.” [National liver recipient]
“The only reason I’m getting the transplant is to last longer after the transplant.” [Local liver candidate]
“But I’m looking at the survival rate... If I go through the surgery and everything else, where am I going to be in a year?” [National heart recipient]
Theme 3: Individual Circumstances Factor Into Decisions
Participants commonly described how information viewed or used in a decision would be considered within the context of other constraints or experiences. Examples included how medical urgency, number of available options, potential for a living donor, and personal values would potentially be considered along with center outcome data to make a trade-off decision.
“It’s based on situation. I had a live donor, and I wouldn’t even look at the [transplant rate] column. For me, it took away my decision.” [National kidney recipient]
“It just depends how sick you are.” [National liver recipient]
“I think that’s for the individual to decide, and I think this is what helps them decide.” [Local heart candidate]
“It depends on your value system.” [National heart recipient]
Quantitative Phase: Randomized Survey
Two thousand thirty-seven participants were randomized, and the analysis set included 975 responses (Table 3). Exclusions included a programming error (n = 630), missed attention question (n = 68), incomplete responses (n = 66), and data for a pilot concept (n = 293). Randomization and details of these exclusions are shown in Figure S1, SDC, http://links.lww.com/TP/B755.
Viewing different mockups impacted decision-making, specifically the proportion of responses selecting each center (Table 4). The control image (mockup #1, Figure 1) resulted in 50% choosing Meadow (highest 1-y organ survival) and 37% choosing Lake (highest transplant rate). The Impact statement (mockup #3, Figure 2) resulted in the largest increase in the proportion that selected Lake, relative to the control; 33% selected Meadow and 51% selected Lake. Individual design elements used in the factorial design impacted decision-making, shown as the relative probability of selecting Alpine Hospital and Lake Hospital compared to Meadow Hospital (Table 5). The impact statement graphical element resulted in a 50% higher probability of selecting Lake Hospital, depicted as the highest transplant rate, compared to Meadow Hospital, depicted as the highest 1-year organ survival (RR, 1.161.501.95). Alpine Hospital (best waitlist mortality) was selected least frequently when viewing the control; however, viewing the data table increased the probability of choosing Alpine Hospital from 0.31 (Control) to 0.55 (Data table). In summary, using an impact statement to indicate that the transplant rate had the greatest impact on survival resulted in the greatest proportion of participants selecting the center with the highest transplant rate. Similarly, respondents in this group were more likely to answer that the most important factor in a decision was Getting a Transplant Faster. The results for factors rated as most important are summarized in Table S7, SDC, http://links.lww.com/TP/B755.
The focus groups informed the development of concepts used in the randomized survey in a number of important ways. Local focus groups were conducted first, and these candidates often described a priority for posttransplant outcomes (see Theme 2 and Table S4, SDC, http://links.lww.com/TP/B755). The impact statement was included in survey options to quantity effects on a bias towards posttransplant outcomes.
Many participants described potential sources of uncertainty in the outcomes displayed (see Theme 1 and Table S4, SDC, http://links.lww.com/TP/B755). Guidelines for public healthcare reports suggest omitting confidence intervals on public reports, because users are unlikely to be familiar with these statistical concepts.6 A statement describing uncertainty in general terms was appropriate to help users interpret information (Figure 3) and included in the randomized survey to assess the impact on decisions.
The wide variability in waiting times was acknowledged by some participants (see Theme 1. These comments often reflected variability within a center for individuals, rather than geographic variation or variation across centers. A table of numerical values (Figure 3) allowed users to identify the potential range of outcomes reflected across the 5 tiers, for example the magnitude of differences in posttransplant outcomes compared to the magnitude of differences in transplant rates. However, presenting this data in patient-friendly displays was hindered by the complexity of underlying calculations and units for pretransplant measures, and additional research is warranted to best provide this context. This table was assessed in the randomized survey to determine if viewing the relative magnitude of differences would influence decisions. Current Scientific Registry of Transplant Recipients search tools (www.srtr.org) use less technical versions of units shown on data tables, for example “years of waiting” rather than “person-years.”
Although the wide variation in waiting times across regions is one reason a transplant rate metric would be relevant to patients, the wide variation in waiting times within a center may make the information harder to apply in a decision. Helping patients narrow this potentially wide range of waiting times based on individual characteristics may increase perceived relevance.28,29
The results of the randomized survey reinforce findings from related studies18-21,27 that content and presentation style can affect decisions. Although hypothetical choices were designed to reduce bias toward posttransplant outcomes, a preference for choosing a center with the best 1-year organ survival metrics (Table 4) was consistent with prior transplant surveys,30 and prior studies indicating patients may underestimate waitlist mortality risks.31,32 In this regard, the behavior of candidates and nonpatient survey participants was consistent. The qualitative responses confirmed that when faced with trade-off decisions, “fast and frugal” cognitive processes can be a default,19 and the short survey was intended to evaluate this type of decision.
The most significant change in decision behavior is attributed to the impact statement (Table 5), where this information resulted in a 50% higher probability of selecting Lake Hospital (highest transplant rate) compared to Meadow Hospital (highest 1-y organ survival). Waitlist mortality was consistently the lowest priority based on center decisions; however, the probability of this choice increased when viewing the data table. One potential explanation for this change is the presence of the word “Deaths” in the waitlist mortality units, which may draw attention to this measure.
The randomized survey results suggest that the impact statement would be effective in reducing a bias to choose centers based on posttransplant outcomes when multiple measures are provided. The use of this statement is not intended to communicate that a particular patient should choose based on the metric that is noted as most important to survival (or any other metric). Patients make decisions based on individual circumstances and values (see Theme 3 and Table S4, SDC, http://links.lww.com/TP/B755), and these values may not align with predicted risks. In other words, the graphical elements evaluated were not intended to convince users to choose a particular option because of predicted risks. Rather, if the information is available and a patient is aware of relevant risks, the reported data can support informed decision-making.
Additional mixed methods research is warranted to understand if new data or context to explain data would further support individual or shared decisions. New data may include a single survival measure combining pretransplant and posttransplant outcomes, patient-specific estimates of waitlist outcomes,33,34 center measures reflecting waitlist access,3 or inferred criteria for listing candidates with higher-risk characteristics.28 The number of potential future changes is high, and a combination of qualitative patient perspectives and decision-making data from nonpatients is a feasible approach to compare many alternatives, in particular for reporting tools targeting patients before a transplant evaluation.
A discrepancy was evident, where 4% of survey respondents had a perception that risks after transplant were greater than risks due to waiting (Table 3); however, a higher range of 31%–50% (Table 4) chose a center with the best 1-year survival. This discrepancy may be explained by the possibility that pretransplant metrics are unfamiliar and therefore prioritized less when viewed on a report. However, this issue warrants further study.
Additional themes related to making a choice about a center were discussed and have been previously described as common factors in decisions.25 These included the importance of perceived reputation, experience of staff or surgeons, proximity, convenience, provider referrals, etc.
The study included a number of important limitations. Focus groups included small samples, with the exception of kidney, which may not have been representative of patients nationally and were not sufficient to draw conclusions about differences in decision-making across organ types. Audio recordings did not allow transcriptions to identify individual speakers, limiting the ability to analyze multiple comments by the same individual. The randomized survey included participants who were not necessarily transplant patients; however, additional research is needed to verify that internet users from the general public perceive information similarly to early-stage patients seeking a referral. The AMT platform used for survey recruiting allows screening for workers of high rated quality and has previously been validated in academic research,35,36 including applications related to organ transplantation, for example, perceptions of left ventricular assist devices37 and liver organ donation.38 Participants were biased towards higher education, and survey participants were younger on average than transplant candidate populations (see Table S5, SDC, http://links.lww.com/TP/B755), which may limit generalizability. Focus group and survey protocols did not include assessments of health literacy and numeracy to analyze decisions across a literacy spectrum. Table S6, SDC, http://links.lww.com/TP/B755 includes a summary of a post hoc subgroup analysis. Groups stratified by age, sex, and education (survey questions #1–#3) did not correlate to changes in decisions. The survey did not assess how the respondents’ values may or may not align with data on predicted risks, although this was a theme of qualitative discussions.
In conclusion, qualitative feedback identified patient perceptions of uncertainty in outcome metrics, in particular pretransplant metrics, and suggested a need to explicitly provide guidance to interpret the most important metric for organ-specific patient mortality. Randomized survey results demonstrated that a note to indicate which metric has the largest impact on survival, for example, transplant rate, significantly increased the probability respondents choose hospitals with the best outcomes in this metric. The presentation of public reports influenced decision-making behavior. As new metrics are proposed and validated, understanding patient perspectives as well as decision-making behavior will better inform how to support patients.
1. Schold JD, Srinivas TR, Howard RJ, et al. The association of candidate mortality rates with kidney transplant outcomes and center performance evaluations. Transplantation. 2008; 85:1–6
2. Kasiske BL, Wey A, Salkowski N, et al. Seeking new answers to old questions about public reporting of transplant program performance in the United States. Am J Transplant. 2019; 19:317–323
3. Patzer RE. Quality metrics in transplantation—a new emphasis on transplant access. Am J Transplant. 2018; 18:1301–1302
4. Greenwald M. Shifting the conversation on outcomes reporting. Am J Transplant. 2018; 18:1303–1304
5. Brett KE, Ritchie LJ, Ertel E, et al. Quality metrics in solid organ transplantation: a systematic review. Transplantation. 2018; 102:e308–e330
6. Hibbard J, Sofaer S. Best practices in public reporting no. 1: how to effectively present health care performance data to consumers. 2017Agency for Healthcare Research and Quality. Updated August 2018. Accessed https://www.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/pubrptguide1/pubrptguide1.html
. Accessed July 15, 2019
7. Hibbard JH, Peters E. Supporting informed consumer health care decisions: data presentation approaches that facilitate the use of information in choice. Annu Rev Public Health. 2003; 24:413–433
8. Schaffhausen CR, Bruin MJ, Chu S, et al. The importance of transplant program measures: surveys of three National Patient Advocacy Groups. Clin Transplant. 2018; 32:e13426
9. Kasiske BL, Snyder JJ, Skeans MA, et al. The geography of kidney transplantation in the United States. Am J Transplant. 2008; 8:647–657
10. Axelrod DA, Dzebisashvili N, Schnitzler MA, et al. The interplay of socioeconomic status, distance to center, and interdonor service area travel on kidney transplant access and outcomes. Clin J Am Soc Nephrol. 2010; 5:2276–2288
11. Sanaei Ardekani M, Orlowski JM. Multiple listing in kidney transplantation. Am J Kidney Dis. 2010; 55:717–725
12. Wey A, Gustafson SK, Salkowski N, et al. Program-specific transplant rate ratios: association with allocation priority at listing and posttransplant outcomes. Am J Transplant. 2018; 18:1360–1369
13. Wey A, Salkowski N, Kasiske BL, et al. A five-tier system for improving the categorization of transplant program performance. Health Serv Res. 2018; 53:1979–1991
14. Wey A, Salkowski N, Kasiske BL, et al. Comparing Scientific Registry of Transplant Recipients posttransplant program-specific outcome ratings at listing with subsequent recipient outcomes after transplant. Am J Transplant. 2019; 19:391–398
15. Wey A, Gustafson SK, Salkowski N, et al. Association of pretransplant and posttransplant program ratings with candidate mortality after listing. Am J Transplant. 2019; 19:399–406
16. Schold JD, Harman JS, Chumbler NR, et al. The pivotal impact of center characteristics on survival of candidates listed for deceased donor kidney transplantation. Med Care. 2009; 47:146–153
17. Husain SA, Brennan C, Michelson A, et al. Patients prioritize waitlist over posttransplant outcomes when evaluating kidney transplant centers. Am J Transplant. 2018; 18:2781–2790
18. Hibbard JH, Greene J, Daniel D. What is quality anyway? Performance reports that clearly communicate to consumers the meaning of quality of care. Med Care Res Rev. 2010; 67:275–293
19. Damman OC, Hendriks M, Rademakers J, et al. How do healthcare consumers process and evaluate comparative healthcare information? A qualitative study using cognitive interviews. BMC Public Health. 2009; 9:423
20. Faber M, Bosch M, Wollersheim H, et al. Public reporting in health care: how do consumers use quality-of-care information? A systematic review. Med Care. 2009; 47:1–8
21. Peters E, Dieckmann N, Dixon A, et al. Less is more in presenting quality information to consumers. Med Care Res Rev. 2007; 64:169–190
22. Tong A, Chapman JR, Israni A, et al. Qualitative research in organ transplantation: recent contributions to clinical care and policy. Am J Transplant. 2013; 13:1390–1399
23. Creswell J, Klassen AC, Plano Clark V, et al; Best practices for mixed methods research in the health sciences. National Institutes of Health Office of Behavioral and Social Sciences Research. 2011. Available at https://obssr.od.nih.gov/training/online-training-resources/mixed-methods-research
. Accessed July 15, 2019
24. Patton MQ. Qualitative Research and Evaluation Methods: Integrating Theory and Practice. 2015, 4th ed. London, United Kingdom: SAGE Publications Ltd
25. Schaffhausen CR, Bruin MJ, McKinney WT, et al. How patients choose kidney transplant centers: a qualitative study of patient experiences. Clin Transplant. 2019; 33:e13523
26. Maxwell J. Qualitative Research Design: An Interactive Approach. 2005, 2nd ed. Thousand Oaks, CA: SAGE Publications
27. Schaffhausen C, Chu S, Bruin M, et al. Using visual icons to present program transplant rate information to the public: results of a randomized trial [Abstract]. Am J Transplant. 2017; 17Suppl 3
28. Schaffhausen C, Bruin M, Chu S, et al. Providing transplant candidates with patient-specific public reports about programs that transplant patients like them [Abstract]. Am J Transplant. 2017; 17Suppl 3
29. Schaffhausen C, Bruin M, Chu S, et al. Providing transplant candidates with guidance about differences in acceptance criteria across programs [Abstract]. Am J Transplant. 2017; 17Suppl 3
30. Schaffhausen C, Bruin M, Chu S, et al. The importance of transplant program measures: surveys of three National Patient Advocacy Groups [Abstract]. Am J Transplant. 2017; 17Suppl 3
31. Salter ML, Orandi B, McAdams-DeMarco MA, et al. Patient- and provider-reported information about transplantation and subsequent waitlisting. J Am Soc Nephrol. 2014; 25:2871–2877
32. Wachterman MW, Marcantonio ER, Davis RB, et al. Relationship between the prognostic expectations of seriously ill patients undergoing hemodialysis and their nephrologists. JAMA Intern Med. 2013; 173:1206–1214
33. Hart A, Schladt DP, Zeglin J, et al. Predicting outcomes on the liver transplant waiting list in the United States: accounting for large regional variation in organ availability and priority allocation points. Transplantation. 2016; 100:2153–2159
34. Hart A, Salkowski N, Matas A, et al. A calculator for kidney transplant waitlist outcomes under the new kidney allocation system [Abstract]. Am J Transplant. 2017; 17Suppl 3
35. Goodman JK, Cryder CE, Cheema A. Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples. J Behav Decis Mak. 2013; 26:213–224
36. Thomas KA, Clifford S. Validity and Mechanical Turk: an assessment of exclusion methods and interactive experiments. Comput Human Behav. 2017; 77:184–197
37. Paine AM, Allen LA, Thompson JS, et al. Anchoring in destination-therapy left ventricular assist device decision making: a Mechanical Turk survey. J Card Fail. 2016; 22:908–912
38. Stroh G, Rosell T, Dong F, et al. Early liver transplantation for patients with acute alcoholic hepatitis: public views and the effects on organ donation. Am J Transplant. 2015; 15:1598–1604