Secondary Logo

A PRO-cision Medicine Methods Toolkit to Address the Challenges of Personalizing Cancer Care Using Patient-Reported Outcomes

Introduction to the Supplement

Snyder, Claire, PhD*,†,‡; Brundage, Michael, MD, MSc§; Rivera, Yonaira M., MPH; Wu, Albert W., MD*,†

doi: 10.1097/MLR.0000000000001089

Patients are increasingly being asked to complete standardized, validated questionnaires with regard to their symptoms, functioning, and well-being [ie, patient-reported outcomes (PROs)] as part of routine care. These PROs can be used to inform patients’ care and management, which we refer to as “PRO-cision Medicine.” For PRO-cision Medicine to be most effective, clinicians and patients need to be able to understand what the PRO scores mean and how to act on the PRO results. The papers in this supplement to Medical Care describe various methods that have been used to address these issues. Specifically, the supplement includes 14 papers: 6 describe different methods for interpreting PROs and 8 describe how different PRO systems have addressed interpreting PRO scores and/or acting on PRO results. As such, this “Methods Toolkit” can inform clinicians and researchers aiming to implement routine PRO reporting into clinical practice by providing methodological fundamentals and real-world examples to promote personalized patient care.

*Johns Hopkins School of Medicine

Johns Hopkins Bloomberg School of Public Health

Sidney Kimmel Comprehensive Cancer Center at Johns Hopkins, Baltimore, MD

§Queen’s Cancer Research Institute, Kingston, ON, Canada

This paper is part of the PRO-cision Medicine Methods Toolkit paper series funded by Genentech. All authors received funding support from Genentech through their institutions. Y.M.R. is also supported by the National Institutes of Health, National Research Service Award T32 CA009314. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

The PRO-cision Medicine Methods Toolkit paper series was presented during a symposium at the 2018 Annual Conference of the International Society for Quality of Life Research, Dublin, Ireland.

The authors declare no conflict of interest.

Reprints: Claire Snyder, PhD, 624 N. Broadway, Room 649, Baltimore, MD 21205. E-mail:

Back to Top | Article Outline


When treating cancer patients, their functioning and well-being is of utmost importance. How cancer patients feel and function are best measured using patient-reported outcomes (PROs), specifically, patients’ direct reports on outcomes such as symptoms, functioning, and health-related quality of life.1,2 Traditionally, PROs have been measured in clinical trials and other research studies to compare the impact of different treatment options from the patient’s perspective.3–7 More recently, there has been interest in using PROs to monitor the progress of individual patients and inform their management.8–13 This approach of using PROs involves having a patient complete a standardized questionnaire, providing that patient’s results to his/her clinical team, and using the PRO data—along with other clinical information (laboratory values, imaging studies)—to manage that patient’s care.

The use of PROs in clinical practice has demonstrated benefits, including promoting patient-clinician communication,14–17 assisting with problem detection,11–13,17 influencing management,16 and improving outcomes, such as symptom control, health-related quality of life, and functioning.14,18–20 Recent studies have shown a survival benefit associated with the intervention.21,22 PRO-cision Medicine23 is the concept of using patients’ own reports of their functioning and well-being to personalize their care.

Although evidence supports the effectiveness of PRO-cision Medicine, there are barriers to broad implementation. To promote the use of PROs, a panel of experts at a recent meeting prioritized: (1) helping patients and clinicians interpret the PRO scores, and (2) helping patients and clinicians act on the PRO results.24 This supplement addresses those key issues.

Back to Top | Article Outline

Issue #1: Interpreting PRO Scores

The issues associated with interpreting PRO scores have been well documented. Problems with interpretation stem from the multitude of PRO instruments and lack of standardization in how these PRO instruments are scored and scaled. On some PRO instruments, higher scores represent better outcomes; on other PRO instruments, higher scores indicate worse outcomes; and on still other PRO instruments, higher scores indicate “more” of the outcome (such that higher scores are better for function domains but worse for symptoms). Beyond the directionality of scoring, PRO instrument scaling also differs, further complicating interpretation. For example, some instruments’ scores are normed to a general population average of 50, whereas others are linearly transformed to a 0–100 scale, and others are simply summed. Both patients and clinicians have reported that this variability is confusing, with quotes such as, “Of course I have no idea if this is a good score or a bad score,” “Until you address the scaling issues it isn’t very useful…,” “A score of say, 50, meant one thing on one graph and something different on another one, which I thought was strange,” and at the most basic level, “I don’t know what the numbers mean.”25

To address the issue of score interpretation, we previously conducted a 3-part research study to identify the formats for displaying PRO data that were most accurately interpreted and rated as clearest.26–28 Following the completion of this research project, our Stakeholder Advisory Board suggested that we had created an evidence base sufficient to inform development of best-practice recommendations for the display of PRO data to promote understanding and use. We thus undertook a modified-Delphi consensus process to develop stakeholder-driven, evidence-based standards for presenting PROs in practice.29

The recommendations that emerged from the Delphi consensus panel include the value of: (1) providing descriptive y-axis labels (eg, none, mild, moderate, severe); (2) indicating scores that are possibly concerning; and (3) providing scores for reference populations. However, the Delphi panel also noted that the information needed to implement these recommendations is not available for many PRO instruments. For example, while the descriptive labels along the y-axis help add meaning to the numeric scores, the score ranges that would be associated with each category are unknown for most PRO instruments. Similarly, many PRO instruments do not have established threshold values to indicate which scores may be concerning. There is little comparison data from reference populations, and questions remain with regard to the appropriate comparators.

Back to Top | Article Outline

Issue #2: Acting on PRO Results

In addition to the challenges associated with understanding what scores mean, there is also the question of how issues identified by PRO measures should be addressed. That is, once a score has been identified as possibly concerning, what should clinicians (and patients) do to respond? As noted above, PRO measures may assess a number of domains ranging from symptoms, functional status, well-being, and health-related quality of life. Although clinicians are trained to manage specific somatic symptoms, based in part on established evidence for effective interventions, issues such as social function (ability to participate in work and hobbies) receive less attention. Clinicians may be reluctant to use PRO assessments if they do not feel comfortable responding to the results. In addition, with the increasing focus on patient empowerment and self-management, patients may also want guidance to self-manage issues identified by PRO measures.

Evidence suggests that providing guidance on how to respond to PRO-identified issues can contribute to the successful use of PROs in clinical practice. One of the earliest randomized controlled trials evaluating a PRO intervention provided problem-specific resource and management suggestions and found significant improvements in identifying problems, better management of them, and improved patient outcomes in the intervention group.30 Later studies suggested that providing such recommendations for managing PRO issues is an important component of the intervention’s success.31 A number of groups have undertaken projects to develop suggestions for responding to PRO issues.32,33 However, each of these projects has used different methods, and guidance is needed for others seeking to implement PROs in routine patient care.

To address the issues with interpreting PRO scores and acting on PRO results, we invited a series of papers from experts with experience developing methods for interpreting and/or acting on PROs in clinical practice. This resulting supplement can serve as a “toolkit” of different methods to which researchers and clinicians can refer when implementing PRO use in routine care.

Back to Top | Article Outline


To develop the supplement, the principal investigators (PIs: C.S., M.B., A.W.) first convened a Steering Group with expertise in methods for interpreting and/or acting on PRO scores (see the Acknowledgments section). During a conference call and follow-up emails, the PIs and Steering Group identified additional experts who could also contribute papers to the supplement. Experts on the final list were invited to participate and contribute a paper to the series. An honorarium was provided to each Steering Group member and to each author team, with the author team determining the appropriate division of the funds.

To ensure coordination across papers in the supplement, the PIs held a kick-off conference call with the paper authors in December 2017. Subsequently, each author team submitted a draft abstract in January 2018, and then circulated its draft paper in May 2018. In June 2018, the PIs, Steering Group, and at least one representative from each author team met in person in Baltimore, MD. During this meeting, each paper was presented and discussed by the group as whole. Revised papers were reviewed by the PIs before being submitted to Medical Care for external peer review.

Notably, this supplement is designed to be a “methods toolkit,” with the individual papers describing different approaches to aid PRO score interpretation and/or develop guidance for acting on PRO results. Because of this focus on methods, many of the papers do not include data or results, although in some cases previous publications are referenced that have applied the methods. Although the supplement focuses on cancer, some papers also describe methods used in other conditions that would be applicable to cancer. Similarly, many of the methods applied in the cancer context could also be applied in other disease areas. Finally, the supplement focuses on PRO data in individual patient care, with the goal to describe methods that can help with interpretation/action at the individual patient level. Below, we provide an overview of the papers in the supplement.

Back to Top | Article Outline


The supplement includes 14 papers: 6 describe different methods for interpreting PROs, and 8 describe how different PRO systems have addressed issues related to interpreting PRO scores and/or acting on the PRO results.

The 6 papers describing methods to aid interpretation cover a range of approaches. The first paper, by Shi et al,34 describes quantitative methods for identifying cutpoints on questionnaires, while the second paper, by Cook et al,35 describes a qualitative approach, “bookmarking,” for setting cutpoints. Another innovative approach for PRO score interpretation, as described by Browne et al,36 is using modern psychometric methods, in this case the Rasch model, to benchmark the performance associated with different scores on a scale—and how this information could be used both to inform patient’s treatment decision-making, as well as to monitor their progress. As such, these 3 papers explain different methods, or “tools,” that can be used to determine what score ranges represent different levels, such as none, mild, moderate, or severe. The fourth paper, by Jensen et al,37 describes how reference values can aid interpretation of score meaning—and reviews methods for collecting and applying the data for these reference values. Another approach for aiding interpretation is discussion between patients and providers, what the Oliver et al38 paper refers to as “feedforward” of the patient’s perspective to his/her clinical team to inform that particular patient’s management. They also describe “feedback” population analytics based on data aggregated across patients to inform treatment decision-making, predictive modeling, and patient-centered care. Three case studies describe PRO systems at different levels of maturity in implementing feedforward and feedback. Finally, the paper by King et al39 provides a thoughtful review of different metrics commonly used for group-level PRO interpretation [eg, minimal important difference, definitions of responders (ie, improved, stable, worsened)], and the extent to which they can aid in the interpretation of individual patient PRO scores.

Eight papers describe different systems that have been developed to collect PROs for patient monitoring and management and the methods they have used to aid interpretation of PRO scores and/or guide action on the PRO results. Notably, the different systems vary in terms of their characteristics, including their purposes for data collection; the types of patients who are targeted for PRO completion; which PRO questionnaires are used; how, when, and where the PROs are collected; and how, when, and where the PROs are reported. The characteristics of these PRO systems, as well as the 3 case studies described by Oliver et al,38 are shown in Table 1. This diversity across the PRO systems’ designs, as well as the different approaches the systems have used to aid interpretation of the PRO scores and/or act on the PRO results, suggest a range of options for addressing these issues.



First, Blackford et al40 describe how the PatientViewpoint web system determines which scores to highlight as possibly concerning, either in absolute terms or a significant worsening, as well as how guidance was developed for acting on these possibly concerning scores. In addition, the paper describes methodologic research the authors pursued to use needs assessments and patients’ reports of their most bothersome issues to identify possibly concerning scores. Haverman et al41 describe the KLIK system developed for children with chronic disease and their parents in the Netherlands, which is now also being used in adults. They highlight how KLIK scores can be reported in various formats, including literal representation of the individual items, sum scores, and different graphic displays. From the United Kingdom, Absolom et al42 describe the electronic patient self-Reporting of Adverse events: Patient Information and aDvice (eRAPID) system, which focuses specifically on chemotherapy adverse events and provides immediate severity tailored feedback for self-management or advice to contact the provider. The paper describes the methods they used to enhance patients’ and clinicians’ engagement with the symptom reports to promote their use in clinical practice. The “Symptom Care at Home” PRO system in the Mooney et al43 paper is unique in that it uses daily phone symptom collection. The interactive voice response system provides automated self-management coaching for patients and alerts to the oncology team about poorly controlled symptoms. In contrast to the other PRO systems, which primarily collect multi-item PROs, Zahrieh et al44 discuss their Beacon system’s focus on single item data collection to highlight the patient’s single biggest concern. In an example of population-wide PRO data collection, Barbera et al45 describe how Ontario has implemented routine PRO reporting across the province over the past decade. They report the various pragmatic methods and evolving approaches used to make the PRO system more useful in practice. In an example from Australia, Girgis et al46 describe the Patient-Reported Outcomes for Personalised Treatment and Care eHealth system (PROMPT-Care), which is fully integrated in hospital electronic oncology information systems. The PROMPT-Care developers emphasized selection of PRO measures that are brief and clinically actionable, and providing care pathways that fit in the clinical workflow to address identified issues. Finally, Stover et al47 describe different methods for alerting clinicians about concerning symptom questionnaire responses from different research studies evaluating the use of PROs in clinical practice. The percentage of PRO reports that trigger an alert varied widely, depending on the guidelines used to determine which scores would generate an alert, as well as due to the different contexts in which the research studies were conducted.

Back to Top | Article Outline


The use of PROs in clinical practice has the potential to promote patient-centered, personalized care. However, until issues related to interpreting PRO scores and acting on PRO results are addressed, the impact of PROs in practice will be limited. Together, the 14 papers in this supplement provide a range of options or “tools” that clinicians and researchers can apply to the use of PROs in clinical practice. The methodologic papers suggest alternative approaches, which can be used alone or in combination to aid PRO score interpretation. In addition, the descriptions of the different methods used by the various PRO systems illustrate the diversity of approaches that can be used, both for interpreting PRO scores and for acting on the PRO results. As such, this supplement “Methods Toolkit” can inform clinicians and researchers aiming to implement routine PRO reporting by providing methodological fundamentals and real-world examples of how to interpret PRO scores and act on PRO results.

Patients value routine PRO data collection more highly when their clinicians actually use the data to inform their care.25 Implementing systems that arm patients and clinicians with the tools to understand the PRO scores, and then act on them, promotes the effective use of PROs in clinical practice. Seeing their data used in their care will encourage patients to complete the questionnaires. More complete data has the potential to not only improve patients’ own care, but also provides more complete datasets for secondary analyses in patient-centered and comparative effectiveness research.48 In all these ways, we can promote personalized patient care based on patients’ reports of their functioning and well-being—“PRO-cision Medicine.”

Back to Top | Article Outline


1. US Food and Drug Administration. Guidance for industry. Patient reported outcome measures: use in medical product development to support labeling claims. Federal Register. 2009;74:65132–65133.
2. Acquadro C, Berzon R, Dubois D, et al. Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001. Value Health. 2003;6:522–531.
3. Au H-J, Ringash J, Brundage M, et al. Added value of health-related quality of life measurement in cancer clinical trials: the experience of the NCIC CTG. Exp Rev Pharmacoecon Outcomes Res. 2010;10:119–128.
4. Till JE, Osoba D, Pater JL, et al. Research on health-related quality of life: dissemination into practical applications. Qual Life Res. 1994;3:279–283.
5. Lipscomb J, Gotay CC, Snyder C, eds. Outcomes Assessment in Cancer: Measures, Methods, and Applications. Cambridge: Cambridge University Press; 2005.
6. Brundage M, Bass B, Ringash J, et al. A knowledge translation challenge: clinical use of quality of life data from cancer clinical trials. Qual Life Res. 2011;20:979–985.
7. Bezjak A, Ng P, Skeel R, et al. Oncologists’ use of quality of life information: results of a survey of Eastern Cooperative Oncology Group physicians. Qual Life Res. 2001;10:1–13.
8. Greenhalgh J. The applications of PROs in clinical practice: what are they, do they work, and why? Qual Life Res. 2009;18:115–123.
9. Snyder CF, Aaronson NK. Use of patient-reported outcomes in clinical practice. Lancet. 2009;374:369–370.
10. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305–1314.
11. Greenhalgh J, Meadows K. The effectiveness of the use of patient-based measures of health in routine practice in improving the process and outcomes of patient care: a literature review. J Eval Clin Prac. 1999;5:401–416.
12. Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Prac. 2006;12:559–568.
13. Haywood K, Marshall S, Fitzpatrick R. Patient participation in the consultation process: a structured review of intervention strategies. Patient Educ Couns. 2006;63:12–23.
14. Velikova G, Booth L, Smith AB, et al. Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. J Clin Oncol. 2004;22:714–724.
15. Berry DL, Blumenstein BA, Halpenny B, et al. Enhancing patient-provider communication with the Electronic Self-Report Assessment for Cancer: a randomized trial. J Clin Oncol. 2011;29:1029–1035.
16. Santana M-J, Feeny D, Johnson JA, et al. Assessing the use of health-related quality of life measures in the routine clinical care of lung-transplant patients. Qual Life Res. 2010;19:371–379.
17. Detmar SB, Muller MJ, Schornagel JH, et al. Health-related quality-of-life assessments and patient-physician communications. A randomized clinical trial. JAMA. 2002;288:3027–3034.
18. Cleeland CS, Wang XS, Shi Q, et al. Automated symptom alerts reduce postoperative symptom severity after cancer surgery: a randomized controlled trial. J Clin Oncol. 2011;29:994–1000.
19. McLachlan SA, Allenby A, Matthews J, et al. Randomized trial of coordinated psychosocial interventions based on patient self-assessment versus standard care to improve the psychosocial functioning of patients with cancer. J Clin Oncol. 2001;19:4117–4125.
20. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with patient-reported outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol. 2016;34:557–565.
21. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing patient-reported outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318:197–198.
22. Denis F, Lethrosne C, Pourel N, et al. Randomized trial comparing a web-mediated follow-up with routine surveillance in lung cancer patients. J Natl Cancer Inst. 2017;109:djx029.
23. Jensen RE, Snyder CF. PRO-cision Medicine: personalizing patient care using patient-reported outcomes. J Clin Oncol. 2016;34:527–529.
24. Gensheimer SG, Wu AW, Snyder CF. Oh, the places we’ll go: Patient-reported outcomes and electronic health records. Patient. 2018;11:591–598.
25. Snyder CF, Blackford AL, Wolff AC, et al. Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice. Psychooncology. 2013;22:895–901.
26. Brundage MD, Smith KC, Little EA, et al. Communicating patient-reported outcome scores using graphic formats: results from a mixed methods evaluation. Qual Life Res. 2015;24:2457–2472.
27. Smith KC, Brundage MD, Tolbert E, et al. Engaging stakeholders to improve presentation of patient-reported outcomes data in clinical practice. Support Care Cancer. 2016;24:4149–4157.
28. Snyder CF, Smith KC, Bantug ET, et al. What do these scores mean? Presenting patient-reported outcomes data to patients and clinicians to improve interpretability. Cancer. 2017;123:1848–1859.
29. Snyder C, Smith K, Holzner B, et al. PRO Data Presentation Delphi Panel. Making a picture worth a thousand numbers: recommendations for graphically displaying patient-reported outcomes data. Qual Life Res. 2019;28:345–356.
30. Rubenstein LV, McCoy JM, Cope DW, et al. Improving patient quality of life with feedback to physicians about functional status. J Gen Intern Med. 1995;10:607–614.
31. Rosenbloom SK, Victorson DE, Hahn EA, et al. Assessment is not enough: a randomized controlled trial of the effects of HRQL assessment on quality of life and satisfaction in oncology clinical practice. Psychooncology. 2007;16:1069–1079.
32. Hughes EF, Wu AW, Carducci MA, et al. What can I do? Recommendations for responding to issues identified by patient-reported outcomes assessments used in clinical practice. J Support Oncol. 2012;10:143–148.
33. Ridgeway JL, Beebe TJ, Chute CG, et al. A brief patient-reported outcomes quality of life (PROQOL) instrument to improve patient care. PLOS Med. 2013;10:e1001548.
34. Shi Q, Mendoza TR, Cleeland CS. Interpreting patient-reported outcome scores for clinical research and practice: definition, determination, and application of cutpoints. Med Care. 2019;57(supp 1):S8–S12.
35. Cook KF, Cella D, Reeve BB. PRO-bookmarking to estimate clinical thresholds for patient-reported symptoms and function. Med Care. 2019;57(supp 1):S13–S17.
36. Browne JP, Cano S. A Rasch Measurement Theory approach to improve the interpretation of patient reported outcomes. Med Care. 2019;57(supp 1):S18–S23.
37. Jensen RE, Bjorner JB. Applying PRO reference values to communicate clinically relevant information at the point-of-care. Med Care. 2019;57(supp 1):S24–S30.
38. Oliver BJ, Nelson EC, Kerrigan CL. Turning feed-forward and feedback processes on patient-reported data into intelligent action and informed decision-making: case studies and principles. Med Care. 2019;57(supp 1):S31–S37.
39. King MT, Dueck AC, Revicki DA. Can methods developed for interpreting group-level patient-reported outcome data be applied to individual patient management? Med Care. 2019;57(supp 1):S38–S45.
40. Blackford AL, Wu AW, Snyder C. Interpreting and acting on PRO results in clinical practice: lessons learned from the PatientViewpoint system and beyond. Med Care. 2019;57(supp 1):S46–S51.
41. Haverman L, van Oers HA, van Muilekom MM, et al. Options for the interpretation of and recommendations for acting on different PROMs in daily clinical practice using KLIK. Med Care. 2019;57(supp 1):S52–S58.
42. Absolom K, Gibson A, Velikova G. Engaging patients and clinicians in online reporting of adverse effects during chemotherapy for cancer: the eRAPID system (electronic patient self-reporting of adverse events: Patient Information and aDvice). Med Care. 2019;57(supp 1):S59–S65.
43. Mooney K, Whisenant MS, Beck SL. Symptom care at home: a comprehensive and pragmatic PRO system approach to improve cancer symptom care. Med Care. 2019;57(supp 1):S66–S72.
44. Zahrieh D, Alexander A, Fruth B, et al. A case management system for integrating patient-reported outcome assessment in clinical practice: the Beacon tool. Med Care. 2019;57(supp 1):S73–S79.
45. Barbera L, Moody L. A decade in review: Cancer Care Ontario's approach to symptom assessment and management. Med Care. 2019;57(supp 1):S80–S84.
46. Girgis A, Durcinoska I, Arnold A, et al. Interpreting and acting on the PRO scores from the patient-reported outcomes for personalized treatment and care (PROMPT-Care) eHealth system. Med Care. 2019;57(supp 1):S85–S91.
47. Stover AM, Stricker CT, Hammelef K, et al. Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: approaches from 3 prospective studies. Med Care. 2019;57(supp 1):S92–S99.
48. Wu AW, Kharrazi H, Boulware LE, et al. Measure once, cut twice-Adding patient-reported outcome measures to the electronic health record for comparative effectiveness research. J Clin Epidemiol. 2013;66(suppl):S12–S20.

patient-reported outcomes; cancer; interpretation; methods; guidance

Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.