Secondary Logo

Using Stakeholder Engagement to Overcome Barriers to Implementing Patient-reported Outcomes (PROs) in Cancer Care Delivery

Approaches From 3 Prospective Studies

Stover, Angela M., PhD*,†; Tompkins Stricker, Carrie, PhD, RN; Hammelef, Karen, DNP, RN; Henson, Sydney, BS; Carr, Philip, BS; Jansen, Jennifer, MPH; Deal, Allison M., MS; Bennett, Antonia V., PhD*,†; Basch, Ethan M., MSc, MD*,†,§

doi: 10.1097/MLR.0000000000001103
Methods Used by PRO Systems

Introduction: Patient-reported outcome (PRO) measures used during cancer care delivery improve communication about symptoms between patients and clinicians and reduce service utilization for uncontrolled symptoms. However, uptake of PROs in routine cancer care has been slow. In this paper, we describe stakeholder engagement activities used to overcome barriers to implementing PROs. Implementation occurred in 2 study settings: PROs completed in the waiting room and reviewed during clinical visits to guide symptom management for multiple myeloma (visit-based PROs); and weekly PROs completed by cancer patients between chemotherapy visits to monitor symptoms at home (remote PROs).

Methods: PRO implementation steps across studies included: (1) clinician and patient input on key symptoms, PRO measures, and identifying which PRO responses are clinically concerning to better target nursing actions; (2) developing PRO-based clinical decision support (CDS) for responding to concerning PROs; (3) training clinicians and clinical research assistants to interpret PROs and use software; and (4) describing implementation impact (frequency of concerning PRO responses and nursing actions).

Discussion: Clinician and patient input was critical for identifying key symptoms, PRO measures, and clinically concerning response options. For the visit-based PRO observational study, all symptom scores appeared on a clinician dashboard, and those rated ≥1 by patients (on a 0–4 or 0–10 scale) had PRO-based CDS available for access. For the 2 remote PROs trials, stakeholders recommended that the 2 “worst” response options (eg, PRO responses of “often”/“always” or “severe”/“very severe”) would trigger an automated email alert to a nurse along with PRO-based CDS. In each study, PRO-based CDS was tailored based on clinician input. Across studies, the most common nursing response to concerning PROs was counseling patients on (or providing care plans for) self-management of symptoms. In the trials, the percentage of weekly remote PROs generating an alert to a nurse ranged from 13% at an academic center to 36% in community oncology practices.

Key Points: Across 3 prospective studies, PROs implemented into cancer care enabled tailored care based on issues identified on PROs. Stakeholder engagement was critical for successful implementation. This paper assists in addressing important PRO implementation challenges by describing a stakeholder-driven approach.

*Department of Health Policy and Management

Lineberger Comprehensive Cancer Center, University of North Carolina at Chapel Hill, Chapel Hill, NC

Carevive Systems Inc., Philadelphia, PA

§Department of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC

This paper is part of the PRO-cision Medicine Methods Toolkit paper series funded by Genentech. The PRO-cision Medicine Methods Toolkit paper series was presented during a symposium at the 2018 Annual Conference of the International Society for Quality of Life Research, Dublin, Ireland.

Research reported in this publication was partially funded through a Patient-Centered Outcomes Research Institute (PCORI) Award (IHS-1511-33392). This work is solely the responsibility of the authors and does not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee. Additional funding sources include K07 CA124851, Carevive Systems Inc., Celgene, Takeda Pharmaceuticals, and Millennium. The University of North Carolina’s Patient-Reported Outcomes Core is supported in part by the University Cancer Research Fund and the Lineberger Comprehensive Cancer Center core grant (P30-CA-016086).

Portions of this work were presented at the 2018 ASCO quality of care symposium and the 2018 International Society for Quality of Life research conference (Dublin, Ireland, October 25–27, 2018).

A.M.S: honorarium from Genentech. C.T.S.: Stockholder and Officer, Carevive Systems Inc., honorarium from Genentech for paper contribution, honorarium from role on PRO-Cision Medicine Methods Toolkit Steering Committee. K.H.: stockholder and employee, Carevive Systems Inc. The remaining authors declare no conflict of interest.

Reprints: Angela M. Stover, PhD, Department of Health Policy and Management, University of North Carolina at Chapel Hill, 1103-D McGavran-Greenberg Hall, CB # 7411, Chapel Hill, NC 27599. E-mail:

Back to Top | Article Outline


  • Across 3 prospective studies, patient-reported outcome (PROs) collected during cancer care delivery enabled tailored care based on issues identified on PROs.
  • PROs completed at clinic visits (visit-based) or at home between visits (remote PROs) had overlapping implementation barriers that were overcome with stakeholder engagement.
  • Across studies, there was a multistep implementation process:
    • Clinician and patient input on selecting key symptoms, PRO measures, and PRO responses indicating clinically concerning symptoms.
    • Developing PRO-based clinical decision support (CDS) with clinician input.
    • Training clinicians and research assistants to use PROs and software.
    • Describing frequency of concerning PRO responses and clinical actions taken.
  • This paper assists in addressing important challenges for implementing PROs into care delivery.
Back to Top | Article Outline


Symptoms experienced during cancer care contribute to patients’ distress, functional disability, and service utilization,1–4 yet half of symptoms go undetected and unaddressed.5,6 PRO measures used during cancer care improve communication about symptoms between patients and clinicians.7,8 Randomized trials have shown that symptom monitoring during care—via PROs—improves clinician awareness of symptoms, yielding significant benefits such as reduced emergency room (ER) visits, better satisfaction with care, and improved survival.2–4 However, uptake of PROs in cancer care has been slow.9,10

In this paper, we describe stakeholder engagement activities used to overcome barriers in 3 prospective studies. Implementation occurred in 2 study settings: PROs completed in the waiting room and reviewed during clinical visits to guide symptom management for multiple myeloma (visit-based PROs); and weekly PROs completed by cancer patients between chemotherapy visits to monitor symptoms at home (remote PROs).

Barriers to PRO implementation were anticipated at the practice, clinician, and patient levels.9,10 For example, PROs need to be brief and easily interpretable. It was also necessary to determine clinician and patient perceptions of which PRO responses are clinically concerning, in order to better target nursing actions. A related barrier was the lack of CDS for PROs. CDS (sometimes called “clinical pathways”) is guidance provided to clinicians about care that a typical patient should receive based on evidence-based practices.11,12 Conventionally, CDS has been developed based on clinician impressions of symptom grade.11 Few examples of PRO-based CDS are in the literature,13,14 and none of these were developed for US care delivery systems.

The final 2 anticipated barriers included PRO training needs for clinicians and staff, and estimating nursing workload for responding to concerning PRO responses. These barriers have hindered widespread adoption of PROs into routine cancer care.9,10 This paper addresses these limitations by describing implementation approaches, stakeholder engagement activities, and lessons learned.

Back to Top | Article Outline


Studies Illustrating Visit-based and Remote PROs

A single-arm intervention study is used to illustrate a visit-based PRO approach. The study, “Multiple Myeloma Patient Care Plans” (MM-PCP) enrolled 90 adults in active treatment at 3 cancer centers (Table 1).15,16 The main study outcome was provider adherence to evidence-based practices for symptom management. Electronic PROs were completed in the waiting room at 2 visits over a 12-week period using Carevive Systems Inc.’s cloud-based platform.17 Care plans and PRO-based CDS were automatically generated based on patients’ PRO responses, and accessible from a clinician dashboard.



Two randomized trials illustrate a “remote PROs” approach (Table 1). Both trials used a 2-arm design of usual care versus remote PROs completed at home with automated email alerts to nurses for concerning symptoms. Trial outcomes include survival, service utilization, and quality of life. The first trial, “Symptom Tracking and Reporting” (STAR), randomized 766 advanced cancer patients at one academic medical center.3,4 The second trial, “PROs to Enhance Cancer Treatment” [PRO-TECT (AFT-39)] is actively recruiting adults with advanced cancer at 50 community oncology practices and is cluster-randomized ( #: NCT03249090). As of August, 2018, 380 patients have enrolled out of a targeted 1000.

Back to Top | Article Outline

Multistep Implementation Process

Across visit-based and remote PROs studies, a common implementation process was used. PRO implementation steps included: (1) clinician and patient input on key symptoms, PRO measures, and which PRO responses are clinically concerning; (2) clinician input on PRO-based CDS; (3) training for clinicians and clinical research assistants (CRAs) to interpret and use PROs; and (4) describing implementation impact (frequency of concerning PRO responses and nursing actions). Examples from the 3 prospective studies described above are used to show implementation considerations in each step.

Back to Top | Article Outline

Stakeholder Input on Key Symptoms, PRO Measures, and Clinically Concerning PRO Response Options

In the MM-PCP study using visit-based PROs, clinical expert panels and industry experts identified 6 key symptoms for multiple myeloma during focus groups and web-based workshops (diarrhea, neuropathy, pain, fatigue, drowsiness, and sexual function). A combination of PRO measures was used to assess these symptoms: PRO version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE) diarrhea item18–20; Edmonton Symptom Assessment Scale—revised (ESAS-r),21 the Functional Assessment of Cancer Therapy—General (FACT-G) sexual function,22 and chemotherapy-induced peripheral neuropathy23 (Table 1). Response scales were either 0–4 or 0–10. These PROs were chosen because they were developed with patient and clinician input, have validity and reliability evidence in cancer samples, and are quick to complete.18–20,21–23 The clinical expert panels and industry experts recommended that a PRO response of ≥1 on any item should result in the symptom being shown on the clinician dashboard with linked PRO-based CDS available (Table 2).



The 2 trials engaged cancer patients and clinicians to choose key symptoms, PRO measures, and clinically concerning response options. Additional stakeholders in PRO-TECT included health services researchers, a scientific advisory board, and committees from the Alliance for Clinical Trials in Oncology (nursing, health outcomes, and patient advocate committees). Stakeholders chose common symptoms in both trials [gastrointestinal symptoms (diarrhea, constipation, loss of appetite, nausea, vomiting), pain, and physical function], which was consistent with a literature review24,25 (Table 1). STAR also assessed fatigue, interference with daily activities, and items specific to a cancer type (eg, dyspnea for lung cancer).3 Symptoms unique to PRO-TECT included insomnia, dyspnea, emotional distress, and falls (Table 1).

In STAR, items were written by its research team3,26 (early version of PRO-CTCAE). In PRO-TECT, PRO measures were selected with stakeholder input and literature reviews.27–31 Most symptoms were assessed with PRO-CTCAE items because they mirror clinician adverse event reporting (CTCAE). The PRO-CTCAE was also developed with patient and clinician input,18,19,32 demonstrates excellent psychometric properties,20 has validity and reliability evidence in cancer samples,19,20 and is quick to complete. Physical function was assessed with 1 item from the Scored Patient-Generated Subjective Global Assessment33 (PG-SGA, permission granted by developer). Items were written by the PRO-TECT research team to assess eating and drinking decreases and falls. These items underwent cognitive interviews34 with cancer patients from 6 cancer centers. Three rounds of interviews were needed to rewrite items so they were comprehensible and meaningful to patients.

Stakeholders in PRO-TECT raised concerns about alert thresholds that were too high and too low. Thresholds that were too high (eg, very severe symptoms) may result in missing important changes. Stakeholders were also concerned about “alert fatigue” where clinicians would receive too many alerts that were not clinically meaningful. In both trials, stakeholders recommended that alerts be triggered by the 2 “worst” response options (eg, PRO responses of “frequently”/“almost always” or “severe”/“very severe”), which was consistent with a literature review35–37 (Table 2). Stakeholders also felt that alerts were warranted when symptoms worsened by 2 points over the past 7 days (PRO response changed from “never” or “none” to “occasionally” or “moderate”).

Back to Top | Article Outline

Clinician Input on PRO-based CDS

PRO-based CDS was developed for MM-PCP and PRO-TECT (but STAR did not use it). PRO-based CDS was based on content from the major guideline producers in oncology, such as the National Comprehensive Cancer Network (NCCN), American Society of Clinical Oncology (ASCO), the Multidisciplinary Association of Supportive Care in Cancer (MASCC), and the Oncology Nursing Society (ONS).38–41 The CDS was also consistent with ASCO’s criteria for developing high-quality clinical pathways in oncology,42 and existing PRO-based CDS used in other countries.12,13

Clinician input on PRO-based CDS was obtained with the same stakeholder groups described above. In both studies, feedback showed that the initial PRO-based CDS was too long for use in practice. In MM-PCP, stakeholders also reviewed the presentation and formatting of the clinician dashboard. Stakeholders in PRO-TECT recommended reformatting the one-page CDS into sections describing how to assess the symptom over the phone, grading symptoms, and selecting appropriate action (eg, dose changes) (Fig. 1).



Back to Top | Article Outline

Clinician and CRA Training on PROs and Software

Webinar training (1 h) was provided to clinicians and CRAs in all studies on interpreting PRO responses and using external software systems. PRO software platforms were not integrated into local electronic health record (EHR) systems because the necessary intervention components are not available.

For MM-PCP, clinicians and CRAs were taught how to use a clinician-facing dashboard for PROs. The dashboard showed concerning PRO responses, and trained CRAs were resident in clinics to ensure that providers viewed the dashboard and accessed the automatically generated PRO-based CDS available. Clinicians also reviewed a patient-friendly version of CDS (called a “care plan”) with patients and revised it together to further tailor symptom management. Clinician actions (including nurses) were determined by reviewing symptom management strategies listed in the finalized care plan reviewed with the patient.

Training for clinicians and CRAs in STAR and PRO-TECT included examples of interpreting PRO responses, and how to use the PRO Core software system. Clinicians and CRAs were shown PRO items they might see in automated email alerts and which response options would trigger alerts. Clinicians were instructed how to use PRO-based CDS and asked to follow-up with patients within 72 hours. CRAs were taught how to enter new patients in the software and track progress of weekly PRO completion. CRAs were instructed to give patients reminder calls when needed and to follow-up with clinicians 72 hours after an alert to determine nursing responses taken. CRAs also printed symptom report graphs for clinicians when patients visited the clinic.

In PRO-TECT, CRAs were also instructed that automated alerts would be sent to them that included a patient’s study ID and problematic symptom(s). CRAs were asked to add identifying information to the email and forward it to the treating nurse(s). This step ensured that identifying information would only circulate through secure servers of the practice (rather than originating from an outside source—PRO Core).

Back to Top | Article Outline

Describing Implementation Impact (Frequency of Concerning Symptoms and Nursing Actions)

As described in step (2) above, stakeholders in each study determined which PRO responses were clinically concerning (Table 2). In MM-PCP, all patients [n=90 (100%)] reported at least one concerning symptom during any visit. In STAR and PRO-TECT, 278/441 patients (63%) and 319/380 patients (84%) reported at least 1 concerning symptom during the studies, respectively.

Symptoms commonly yielding concerning responses were tracked (Table 2). In MM-PCP, the denominator was the number of visits. PRO responses of ≥1 were most commonly reported for fatigue [119/161 visits (74%)], dyspnea [61/161 visits (38%)], diarrhea [52/161 visits (32%)], and neuropathy [40/161 visits (25%)].

In STAR and PRO-TECT, the denominator was the number of weekly PROs with an alert. Multiple concerning symptoms could appear on the same weekly alert, and thus the weekly rate of concerning symptoms is more representative of nursing workload. In STAR, 1070/8498 weekly PROs (13%) resulted in an alert. Of the 1070 weekly PROs with alerts, 662 included fatigue (62%), 342 pain (32%), and 171 appetite (16%). In the PRO-TECT trial in community practices, the weekly PRO alert rate was considerably higher: 1109/3103 weekly PROs (36%). Of the 1109 weekly PROs with alerts, 543 included pain (49%), 333 physical function (30%), and 189 diarrhea (17%).

We conducted a sensitivity analysis to compare weekly PRO alert rates when restricted to the 6 symptoms collected in both trials (pain, nausea, vomiting, diarrhea, constipation, and appetite). In STAR, the weekly PRO alert rate reduced to 609/8496 weekly PROs (7%) versus 776/3103 weekly PROs (25%) in PRO-TECT.

Finally, nursing responses to clinically concerning PROs were tracked. In MM-PCP, there were a total of 684 nursing responses recorded during 161 visits. Common clinician responses involved counseling patients on home care strategies [515/684 responses (75%)], referrals [96/684 responses (14%)], and changing medications [73/684 responses (14%)]. In a previous STAR publication,3 nursing interventions included telephone counseling about symptom management (77%), supportive medication initiation/change (12%), and referral to the ER/hospital (8%) (numerators and denominators were not published in the article).3 In PRO-TECT, there were a total of 616 responses recorded for potential nursing actions, where 392 (64%) indicated a clinical action had been taken. The most common responses to concerning PROs were counseling patients on self-management of symptoms [n=135/392 responses (34%)], planning to discuss at next visit after assessing grade over the phone [n=90/392 responses (23%)], and changing medications [n=46/392 responses (12%)].

In most cases where no action was taken (224/616 responses), the reason provided was that a clinician was already aware of the symptom(s) [206/224 (92%)]. For example, a patient’s PRO responses may have generated multiple weeks of alerts while the same symptom was being treated. The symptom was not “new,” and thus a new clinical action did not need recorded. Clinicians recommended adding a software feature where they could “pause” alerts for a symptom while it was being actively treated.

Back to Top | Article Outline


Three prospective studies show that PROs used during cancer care delivery enable tailored care based on issues identified on PROs. This paper assists in addressing important implementation challenges.

Back to Top | Article Outline

Strengths and Weaknesses of Implementation Approach

An important strength of our implementation approach is stakeholder engagement. Stakeholders determined important symptoms to assess and selected PROs. We recommend that stakeholders (including adults with the health condition) be involved early in implementation decision making.47

Stakeholder engagement was also critical for identifying clinically concerning PRO responses. Research teams made a priori selections about concerning PRO responses, but were unique to each study. Although there is not yet a clear consensus on what a meaningful threshold is for “concerning” PRO responses, the level at which a symptom becomes clinically significant and is important to patients is a good guideline.18,24,25,32,35–37 Evidence suggests that when symptoms are moderate or severe, they begin to meaningfully interfere with function and daily activities,48 which are likely to be significant to clinicians and patients. Stakeholder input in the current studies supported this idea, but more research is needed on selecting and validating thresholds for use in routine care. Research is also needed on how threshold selection impacts clinician workload, false-positive rates, and patient service utilization.

In the second implementation step, PRO-based CDS was refined based on clinician feedback in MM-PCP and PRO-TECT. PRO-based CDS is typically specific to 1 symptom, and thus versions are needed to cover all symptoms assessed, but it does have the advantages of brevity and specificity for use in routine care. MM-PCP and PRO-TECT also used data visualization (eg, symptom graphs over time) at point-of-care, which may increase clinician engagement. A limitation is that PRO-based CDS needs updated as new evidence becomes available and treatment guidelines evolve.

Guidelines specific to developing PRO-based CDS are needed that include recommended minimum standards for PRO measures27 and stakeholder engagement47 to increase content validity. The National Institutes of Health is taking a step toward this goal by funding Small Business Innovation Research contracts to develop algorithm-based CDS for management of common cancer symptoms (, of which Carevive is a recipient.

The third step in our implementation approach was training clinicians and CRAs on PROs and software. Research is needed to determine what additional training and resources will be needed by practices outside the research context. For example, case examples could highlight how to integrate patient preferences, PROs, and symptom management goals into shared decision making conversations.

Third-party PRO software was necessary across studies because most EHR systems lack critical features for using PROs at point-of-care (eg, alert algorithms). Some EHR vendors are working toward meaningful use of PROs at point-of-care but significant work remains to be done. In future studies, it will be important to examine characteristics of practices and EHR systems that are associated with successful PRO and CDS implementation to support replication.

The final step of our implementation approach was to evaluate implementation impact by describing the frequency of concerning PRO responses and clinical actions taken. In the trials, nursing workload for responding to remote PRO alerts was over 20% higher in community oncology practices than at an academic center. Contextual variables may help explain this difference, but caution must be exercised in interpreting results because patients are still enrolling in PRO-TECT. Community cancer centers may have fewer resources available for symptom management49 (eg, palliative care), and patients may be older, less educated, or have more comorbid health conditions, which are risk factors for poor chemotherapy outcomes.50,51 Finally, the difference may be due to different PRO wording. Future implementation studies should consider choosing PRO measures with patient and clinician input and high validity and reliability evidence.

Previous publications for the STAR trial show improved survival by 5 months, reduced ER visits and hospitalizations, and better quality of life among chemotherapy patients randomized to the PRO intervention.3,4 PRO-TECT is an ongoing trial and MM-PCP is currently analyzing data, and thus clinical processes and outcomes will be examined in future publications. PRO-TECT is also examining patient acceptability and usefulness, and clinician acceptability, satisfaction, and workload perceptions with semistructured interviews, which will be described in future publications.

Given the strengths and weaknesses of our implementation approach for visit-based and remote PROs, these methods are best applied with appropriate governance when a PRO program is being implemented for a study or in routine care.10 Visit-based PROs may be a suitable option when symptoms are stable (eg, survivorship visits or early-stage disease with few symptoms), or when a practice is deciding whether to implement PROs for all patients. Remote PROs may be better suited when symptoms are dynamic (eg, chemotherapy), and when practices are experienced with PROs or want to reduce avoidable service utilization (eg, ER visits).

Back to Top | Article Outline


The authors would like to thank several committees from the Alliance for Clinical Trials in Oncology for their expert input during development of the PRO-TECT (AFT-39) trial (Nursing, Patient Advocacy, and Health Outcomes Committees).

Back to Top | Article Outline


1. Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26:1846–1858.
2. Kotronoulas G, Kearney N, Maguire R, et al. What is the value of the routine use of patient-reported outcome measures toward improvement of patient outcomes, processes of care, and health service outcomes in cancer care? A systematic review of controlled trials. J Clin Oncol. 2014;32:1480–1501.
3. Basch E, Deal AM, Kris MG, et al. Symptom monitoring with Patient-Reported Outcomes during routine cancer treatment: a randomized controlled trial. J Clin Oncol. 2016;34:557–565.
4. Basch E, Deal AM, Dueck AC, et al. Overall survival results of a trial assessing Patient-Reported Outcomes for symptom monitoring during routine cancer treatment. JAMA. 2017;318:197–198.
5. Laugsand EA, Sprangers MAG, Bjordal K, et al. Health care providers underestimate symptom intensities of cancer patients: a multicenter European study. Health Qual Life Outcomes. 2010;8:104.
6. Atkinson TM, Li Y, Coffey CW, et al. Reliability of adverse symptom event reporting by clinicians. Qual Life Res. 2012;21:1159–1164.
7. Yang L, Manhas D, Howard A, et al. Patient-reported outcome use in oncology: a systematic review of the impact on patient-clinician communication. Support Care Cancer. 2018;26:41–60.
8. Chen J, Ou L, Hollis SJ. A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organisations in an oncologic setting. BMC Health Serv Res. 2013;13:211.
9. Antunes B, Harding R, Higginson IJ. Implementing patient-reported outcome measures in palliative care clinical practice: a systematic review of facilitators and barriers. Palliative Med. 2014;28:158–175.
10. Snyder CF, Aaronson NK, Choucair AK, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res. 2012;21:1305–1314.
11. Daly B, Zon RT, Page RD, et al. Oncology clinical pathways: charting the landscape of pathway providers. J Oncol Pract. 2018;14:e194–e200.
12. Osheroff JA, Teich JM, Middleton B, et al. A roadmap for national action on clinical decision support. J Am Med Inform Assoc. 2007;14:141–145.
13. Girgis A, Durcinoska I, Koh E-S, et al. Development of health pathways to standardize cancer care pathways informed by patient-reported outcomes and clinical practice guidelines. JCO Clin Cancer Inform. 2018;2:1–13.
14. Fallon M, Walker J, Colvin L, et al. Pain management in cancer center inpatients: cluster randomized trial to evaluate a systematic integrated approach—the Edinburgh Pain Assessment and Management Tool. J Clin Oncol. 2018;36:1284–1290.
15. Stricker CT, Faiman B, Rozario N, et al. Improving provider adherence to evidence-based supportive care practices for patients with multiple myeloma. Poster presentation at the Palliative Care in Oncology Symposium, San Francisco, CA, September 2016.
16. Brant J, Miller J, Dudley WN, et al. Supportive care plans: linking patient-reported outcomes to evidence-based supportive care across the cancer continuum. Podium presentation at the Oncology Nursing Society Annual Congress, San Antonio, TX, May 2016.
17. Carevive PRO platform (software system). Miami: Carevive Systems Inc. 2017. Available at: Accessed September 13, 2018.
18. Hay JL, Atkinson TM, Reeve BB, et al. Cognitive interviewing of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). Qual Life Res. 2014;23:257–269.
19. Basch E, Reeve BB, Mitchell SA, et al. Development of the National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). J Natl Cancer Inst. 2014;106:dju244.
20. Dueck AC, Mendoza TR, Mitchell SA, et al. Validity and reliability of the US National Cancer Institute’s Patient-Reported Outcomes Version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncol. 2015;1:1051–1059.
21. Hannon B, Dyck M, Pope A, et al. Modified Edmonton Symptom Assessment System Including Constipation and Sleep: validation in outpatients with cancer. J Pain Symptom Manage. 2015;49:945–952.
22. Cella DF, Tulsky DS, Gray G, et al. The Functional Assessment of Cancer Therapy (FACT) scale: development and validation of the general measure. J Clin Oncol. 1993;11:570–579.
23. Knoerl R, Gray E, Stricker C, et al. Electronic versus paper-pencil methods for assessing chemotherapy-induced peripheral neuropathy. Support Care Cancer. 2017;25:3437–3446.
24. Reilly CM, Bruner DW, Mitchell SA, et al. A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Support Care Cancer. 2013;21:1525–1550.
25. Reeve BB, Mitchell SA, Dueck AC, et al. Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials. J Natl Cancer Inst. 2014;106:dju129.
26. Basch E, Artz D, Dulko D, et al. Patient online self-reporting of toxicity symptoms during chemotherapy. J Clin Oncol. 2005;23:3552–3561.
27. Reeve BB, Wyrwich KW, Wu AW, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Qual Life Res. 2013;22:1889–1905.
28. Kirkova J, Davis MP, Walsh D, et al. Cancer Symptom Assessment Instruments: a systematic review. J Clin Oncol. 2006;24:1459–1473.
29. Catts S, Starkings R, Shlling V, et al. Patient reported outcome measures of the impact of cancer on patients’ everyday lives: a systematic review. J Cancer Survivor. 2017;11:211–232.
30. Seyidova-Khoshknabi D, Davis MP, Walsh D. Review article: a systematic review of cancer-related fatigue measurement questionnaires. Am J Hosp Palliat Care. 2011;28:119–129.
31. Ziegler L, Hill K, Neilly L, et al. Identifying psychological distress at key stages of the cancer illness trajectory: a systematic review of validated self-report measures. J Pain Symptom Manage. 2011;41:619–636.
32. Atkinson TM, Hay JL, Dueck AC, et al. What do “none,” “mild,” “moderate,” “severe” and “very severe” mean to patients with cancer? Content validity of PRO-CTCAE™ response scales. J Pain Symptom Manage. 2018;55:e3–e6.
33. Ottery F. Definition of standardized nutritional assessment and interventional pathways in oncology. Nutrition. 1996;12:S15–S19.
34. Willis GB. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage; 1999.
35. Giesinger JM, Kuijpers W, Young T, et al. Thresholds for clinical importance for four key domains of the EORTC QLQ-C30: physical functioning, emotional functioning, fatigue and pain. Health Qual Life Outcomes. 2016;14:87.
36. Farrar JT. Cut-points for the measurement of pain: the choice depends on what you want to study. Pain. 2010;149:163–164.
37. Cella D, Choi S, Garcia S, et al. Setting standards for severity of common symptoms in oncology using the PROMIS item banks and expert judgement. Qual Life Res. 2014;23:2651–2661.
38. National Comprehensive Cancer Network (NCCN). NCCN clinical practice guidelines in oncology. 2018. Available at: Accessed September 13, 2018.
39. American Society of Clinical Oncology (ASCO). Treatment guidelines, tools, and sources. 2018. Available at: Accessed September 13, 2018.
40. Oncology Nursing Society (ONS). Standards and guidelines. 2018. Available at: Accessed September 13, 2018.
41. Multinational Association for Supportive Care in Cancer (MASCC). Practice resources and guidelines. 2018. Available at: Accessed September 13, 2018.
42. Zon RT, Edge SB, Page RD, et al. American Society of Clinical Oncology criteria for high-quality clinical pathways in oncology. J Oncol Pract. 2017;13:207–210.
43. Hickey M, Newton S, Eds. Telephone triage for oncology nurses. Pittsburgh, PA: Oncology Nursing Society, 2012.
    44. National Comprehensive Cancer Network (NCCN). Guidelines 2017: Palliative Care. Available at: Accessed February, 2019.
      45. National Cancer Institute (NCI). Gastrointestinal complications - practical, detailed, quick - Health Professional 2016. Available at: Accessed February, 2019.
        46. Oncology Nursing Society (ONS). ONS putting evidence into practice: constipation 2017. Available at: Accessed February, 2019.
          47. Goodman MS, Sanders Thompson VL. The science of stakeholder engagement in research: classification, implementation, and evaluation. Transl Behav Med. 2017;7:486–491.
          48. Atkinson TM, Andreotti CF, Roberts KE, et al. The level of association between functional performance status measures and patient-reported outcomes in cancer patients: a systematic review. Support Care Cancer. 2015;23:3645–3652.
          49. American Society of Clinical Oncology. The State of Cancer Care in America, 2017: a report by the American Society of Clinical Oncology. J Oncol Pract. 2017;13:e353–e394.
          50. Muss HB, Berry DA, Cirrincione C, et al. Toxicity of older and younger patients treated with adjuvant chemotherapy for node-positive breast cancer: the Cancer and Leukemia Group B Experience. J Clin Oncol. 2007;25:3699–3704.
          51. Busch EL, Martin C, DeWalt D, et al. Functional health literacy, chemotherapy decisions, and outcomes among a colorectal cancer cohort. Cancer Control. 2015;22:95–101.

          patient-reported outcomes; cancer care delivery; clinical decision support

          Copyright © 2019 Wolters Kluwer Health, Inc. All rights reserved.