Comparative effectiveness research (CER) “inform[s] health-care decisions by providing evidence on the effectiveness, benefits, and harms of different treatment options.”1 Patient-centered outcomes research (PCOR) stresses the importance of research “informed by the perspectives, interests, and values of patients” throughout the research process.2 Both CER and PCOR capitalize on primary data collection and secondary data sources to provide timely and relevant information to decision makers. Although PCOR and CER comprise largely overlapping sets of activities—both aim to provide practical evidence to support real-world decision making—PCOR particularly emphasizes the patient perspective. Given the central role of the patient perspective in PCOR, methods to incorporate patient-reported outcomes (PROs) are critical to its successful conduct. However, PROs have not routinely been collected in a way that facilitates their use in PCOR. Electronic medical records (EMRs), disease registries, and administrative data have only rarely collected, or been linked to, PROs. Recent technological developments that facilitate the electronic collection of PROs and the linkage of PRO data with other clinical data offer new opportunities for putting the patient perspective in PCOR.
In this paper, we describe the importance of and methods for using PROs for PCOR by (1) defining PROs; (2) identifying how PROs can be used in PCOR; (3) outlining the challenges and key unanswered questions for routine use of PROs in PCOR; and (4) discussing policy and research interventions to accelerate the integration of PROs with clinical data.
PROs are defined by the Food and Drug Administration (FDA) and National Quality Forum (NQF) as “… a report that comes directly from the patient (ie, study subject) about the status of a patient’s health condition without amendment or interpretation of the patient’s response by a clinician or anyone else.”3–4 Considering the meaning of each word in “PRO” is instructive. First, although the “P” in PRO stands for “patient,” it could refer to any person reporting on his/her own health condition or treatment. For example, the definition above includes “study subjects” and could also apply to healthy individuals. Use of the term “patient” rather than “person” conveys the health-related nature of PRO. Second, “reported” is a critical aspect of the PRO definition. As noted above, the standard is for PROs reported directly by the patient, “without amendment or interpretation by …anyone else.”3–4 However, when patients cannot report for themselves (eg, due to disability), proxy respondents can be used in an attempt to obtain the patient’s perspective. PROs are distinct from other patient outcomes such as physiological measures (eg, hemoglobin A1c), clinician-reported measures (eg, global impressions), and caregiver-reported measures.5 Finally, “outcome” in PRO is interpreted broadly to reflect a variety of information reported directly by the patient, including health-related quality of life, functional status, symptoms, and treatment adherence.5 The definition of PRO put forward by AcademyHealth, “health data that are provided by the patient through a formal and tested system of reporting,” does not explicitly emphasize the importance of reporting directly by a patient. Given the central role of direct reporting by the patient in most PRO definitions and the broad application of the FDA and NQF definition, this paper adopts the FDA and NQF definition of PROs.
IMPORTANCE OF ELECTRONIC DATA METHODS
The diverse use of PROs can be classified in 3 applications: population surveillance, individual patient-clinician interactions, and research studies.6 When combined with other clinical data, the PRO data resulting from these uses can be used for PCOR. Figure 1 displays how PRO data from these different applications can be linked with other clinical data sources (eg, EMRs, disease registries) to create PCOR datasets that can inform patient choice, provider recommendations, and policy-maker decisions and serve as key prognostic covariates. However, linking PRO data with other clinical data can be challenging7 and electronic data methods play a critical role in facilitating this linkage of PRO data with other clinical information. Below, we describe how electronic data methods can be used to integrate PROs in PCOR and then provide examples for each of the applications.
Electronic Data Methods to Integrate PROs in PCOR
Across all levels of PRO application, collection of PRO data is difficult—albeit not impossible—without electronic data methods. Electronic data capture improves feasibility of collection, decreases burden, and enables sophisticated survey administration.8 Take, for example, paper-and-pen PRO data collection. The paper form has to be given in-person or mailed, the patient must complete the questionnaire, and then return it (in-person or by mail). The data have to be entered into a database. Standards need to be established for addressing improper questionnaire completion (eg, >1 response to the same question), and there is no practical way to recover missing data. Finally, the questionnaires must be scored before conducting the analyses of interest. This process is expensive, labor intensive, error-inducing, and involves a considerable time lag between questionnaire administration and result reporting.
In contrast, with electronic data collection, particularly through the Internet, the process is much simpler.8–9 Patients can access questionnaires anyplace with Internet access, allowing flexibility to complete questionnaires both inside and outside of clinic visits. An estimated 77% of the US population had Internet access in 2010.10 As patients submit their completed questionnaires, the data are automatically entered into a database. Respondents can receive real-time, automatic alerts that provide guidance for improperly completed questions (eg, not allowing >1 answer, reminders regarding missing responses). The survey scores can be calculated and reported immediately. Automatic skip patterns can be built-in, or more sophisticated techniques such as computer-adaptive testing (CAT) can be implemented. CAT uses the respondent’s prior answers to inform the next item administered from a large pool of items on the same topic, thereby providing reliable assessments more efficiently than fixed-item questionnaires.11 For example, on a physical function questionnaire, if a patient responds that s/he can walk several blocks “none of the time,” the next question administered would not ask if s/he can run 1 mile. It would select a question more suitable for someone with lower physical functioning, such as performing self-care activities. Thus, electronic PRO collection improves data quality and collection efficiency across a wide range of patients, facilitating the use of PROs in PCOR.
At the population level, clinical information is collected using health surveys, disease registries, and administrative claims. Collection of PROs at the population level produces various opportunities to incorporate PROs in PCOR. For example, the UK’s National Health Service collects PROs before and after certain elective surgical procedures: hip replacement, knee replacement, varicose vein surgery, and hernia repair.12,13 Given the large scale of this effort, the United Kingdom now has PRO data from hundreds of thousands of surgical patients. These PRO data can be used to compare care quality across providers or by patients to choose a provider. In addition, the PRO data can help patients understand the likely impact of the procedure on patient-centered outcomes. For these applications, important issues such as risk adjustment need to be addressed.
Disease registries also offer opportunities to capture population-based clinical information, but the data in these registries generally come from medical record abstractions, making inclusion of standardized PROs challenging because PRO data are not routinely available in medical records. Similarly, administrative claims data offer a valuable resource for PCOR but currently lack a mechanism for including PRO data.
With increasing use of electronic data capture, new possibilities are emerging for collecting PROs for large populations. For example, the Commission on Cancer has developed an online platform for hospital-based cancer registries to report on their cancer cases.14 The Commission is also pilot-testing a Rapid Quality Reporting System and plans to expand it to include PROs. The linkage of the PRO data with the registry data will enable the use of PROs in a range of PCOR studies. Another example of using PROs at the population level is the Washington Heights/Inwood Informatics Infrastructure for Community-Centered Comparative Effectiveness Research (WICER) initiative.15 WICER is sampling the population of 5 New York ZIP codes annually for 3 years and collecting data on PROs, demographics, vital statistics, and neighborhood information. Survey data are linked with medical records in a data warehouse, thereby providing a rich data resource.
Individual Patient-Clinician Interactions
On the individual level, PROs can be used clinically to screen for conditions (eg, depression), monitor patient progress over time, and in decision aids.16–18 With the increasing use of EMRs, there are more opportunities to link PROs with the EMR. Eighty-one percent of hospitals and 41% of office-based physicians intended to take advantage of federal incentive payments for adoption and meaningful use of certified EMR technology in 2011–2012.19 Thus, PRO data collected to inform individual patient’s care can be aggregated across patients and analyzed in conjunction with the other clinical information available in the EMR, creating a practical opportunity for PCOR.
Online webtools designed to collect PROs and link them with the EMR can promote the use of PROs for patient care and for PCOR. For example, PatientViewpoint (http:\\www.PatientViewpoint.org) allows clinicians to assign PROs for patients to complete, just as they would with any laboratory test.20–22 Patients receive an email asking them to complete the assigned PROs, which they can do anywhere with Internet access. PatientViewpoint tracks scores over time in reports available through the Website and in the patient’s EMR. Although the primary purpose of this data collection is to improve individual patient care, the PRO data can be aggregated across patients, combined with clinical information from the cancer registry or medical records, and used to compare treatments or evaluate the quality of care.
PROs are most commonly used as outcome measures in clinical trials and observational studies.23–32 For example, the Surgical Care and Outcomes Assessment Program (SCOAP) Comparative Effectiveness Research Translation Network (CERTN) is a prospective cohort study of intermittent claudication.33 The investigators have developed a Survey Center, which allows collection of data using multiple modes, including web-based, to collect PRO and other data outside of the index hospitalization so that prehospitalization and posthospitalization information can be collected.
However, PRO data are generally used to answer a given study’s specific questions and are not readily accessible for secondary analyses. The ability to link these data with other clinical information across studies would improve their usefulness for PCOR. To that end, a draft effectiveness guidance document from the Center for Medical Technology Policy recommends that PROs be included in all late-phase and CER studies in oncology, and that these studies measure a core set of 12 symptoms to facilitate cross-study comparison.34
Illustrating the critical role of electronic data capture in the collection of PRO data for research studies, the Patient-Reported Outcomes Measurement Information System (PROMIS) has developed an online data collection platform in conjunction with the PROMIS questionnaire content. PROMIS is “a system of highly reliable, precise measures of patient-reported health status for physical, mental, and social well-being.”35,36 The PROMIS Assessment Center is a free, online research tool that allows investigators to administer both PROMIS and other PRO questionnaires to patients electronically. PROMIS measures were developed based on modern psychometric concepts, and Assessment Center was designed to make these features accessible to researchers. It allows for customized domain-specific questionnaires and multiple administration methods (both static and computer-adaptive). Domain scores are comparable regardless of length or administration method. This system is already in wide-spread use with over 1000 registered researchers.37
An Application for Further Exploration: Use of PROs as Prognostic Variables
In addition to using PROs in the 3 applications described above, there has been recent discussion of using PROs as important stratification variables or covariates. Because many PCOR studies use observational designs, without patient randomization, confounding is an important methodologic issue. PROs may be useful as balancing variables given that there is substantial evidence that PROs are important prognostic variables38 and improve the accuracy of survival predictions.39 Although there are methodologic concerns with this application of PROs in PCOR, given the unique prognostic contribution of PROs, this potential application is worthy of further exploration.
KEY CHALLENGES AND UNANSWERED QUESTIONS
A number of practical issues need to be confronted to integrate PROs in PCOR (Table 1). Challenges include the proprietary nature of PRO measures, governance related to setting standards, selection of tools for clinical use, confidentiality issues, and coordination among organizations in all of the above.
Although many PROs are available without charge in the public domain and others are available on the basis of “copyleft” (ie, copyrighted so that they can be made freely available for public use),40 other PROs are copyrighted and may require payment for use, as is common for other kinds of intellectual property. Currently, EMR vendors, clinicians, and health care organizations are generally unaccustomed to this model for PROs. In the future, electronic methods should make it feasible to track use of PROs, and to charge for them as appropriate. For example, a clinician may order a PRO test, and the patient or his/her insurer will be billed for this service, as with laboratory tests.
For health care organizations, there are few precedents for the governance of PROs used on an institutional basis. In general, individual investigators or research groups select PRO tools for their studies. Important questions to consider as PROs are integrated in clinical systems include: Who has the authority to establish standards regarding the selection of PRO tools for clinical use? Who should establish score interpretation guidelines? In practice, these decisions may fall to technical members of the health information technology team, but it seems clear that practitioners should lead efforts to establish clinical standards.
In general, PRO information that exists in EMRs is available to any health care worker with a system log-on in the same way that a blood count or radiograph is available. Clinically, this is generally restricted to members of the patient’s care team, on a need-to-know basis. However, some confidentiality considerations are specific to PROs and may trump common practice for other data elements. For example, a patient who completes a satisfaction survey may not want his/her provider to see the results, for fear of retribution if the responses are critical. Perhaps these data should only be available to individual providers on an aggregated basis. Just as psychiatric care is privileged due to the high risk for discrimination associated with psychiatric diagnoses, some patients may want to restrict access to their self-reported information on sexual functioning. The balance between confidentiality and clinical utility of specific PROs must be considered carefully and systematically. We believe that organizations and other standard-setting bodies should establish procedures for considering the adoption of specific PRO for use within EMRs, and then apply these procedures to evaluate each candidate PRO.
Once these governance and data policy decisions have been made, to whom should they apply? It seems logical for large health systems or organizations sharing specific electronic data tools to use the same standards. Common standards could also be applied across different organizations using, for example, a common EMR (eg, EPIC). Most small organizations do not have the capacity to handle the technical work and decision making independently. Accountable care organizations and regional health information exchanges may play an important role in centralizing decision making and establishing standards for broad application.
There are a number of unanswered questions related to using PROs in PCOR, including how to select the PRO measure and interpret the results. Thousands of PROs have been developed and published. Which of these are appropriate for clinical use? The Patient-Reported Outcome and Quality of Life Instruments Database, a curated, categorized library of PRO measures provides a reference source for available PROs.41 Also, in some cases, particular PROs have long been in use (eg, the International Prostate Symptom Score).42 Clinicians in these specialties have become accustomed to using these PROs as clinical tools, but their tacit knowledge regarding these tools has not generally been codified for a wider audience. This is a necessary step for broader dissemination.
For most PROs, broad use would be aided by establishing criteria for psychometric properties and test characteristics for clinical use. For example, how many points on a given scale represent a meaningful or clinically important difference,43 and what cutoff score represents a level of dysfunction that requires a clinician’s attention.44 In evaluating treatment effectiveness, missing PRO data are often informative, for example, sicker patients are less likely to complete their PRO questionnaires. Thus, it must be determined how missing data are to be handled in analysis. Organizations should consider this question prospectively and establish standard/default procedures in consultation with experts in PROs and missing data. These procedures would then be applied to specific PROs, with due consideration to established evidence about the performance of the PRO related to missing data.
An additional issue is the potential to link PRO data to other data sources. In some cases, patients with records in administrative data sources such as insurance billing or pharmacy benefits managers can be identified and invited to respond to survey questions. This requires patient identifiers, but the operation can be performed through a third party in a secure, HIPAA compliant manner.7 Although issues remain related to patient consenting procedures and implementation, this represents a novel mechanism to produce enhanced sources of information.
POLICY (AND RESEARCH) INTERVENTIONS TO PROMOTE PROs IN PCOR
To accelerate the adoption of PROs in PCOR, interventions are needed that can be aimed at stakeholders across the health care system, including patients, family members, patient organizations, clinicians, clinical managers, health care organizations, manufacturers of medications and devices, payers, employers, communities, states, and researchers themselves.
Various policy interventions could be applied to stimulate the adoption of PROs for PCOR. Research policy could require the collection of PROs in specific prospective studies, registries, and cohort studies. Electronic data collection methods make this requirement less onerous and less costly than it may have been previously. FDA regulations could require the collection of these data in phase III and postmarketing surveillance studies. The Patient-Centered Outcomes Research Institute (PCORI) and National Institutes of Health could encourage the creation of data cross-walks (sometimes referred to as backdoors) between large national studies to permit comparison of results across studies. A critical advantage of PROMIS is the ability to compare scores from any PROMIS measure normed to the general US population, regardless of whether a fixed-item form or CAT is used.36
Coverage/payment policy could create financial incentives to increase PRO assessments. For example, Medicare could reimburse physicians or hospitals for PRO data collection when these data are shown to add value. Another fruitful area is including PROs in cases of “coverage with evidence development” (eg, when Medicare agrees to pay for a promising but still unproven treatment subject to collection of registry data).
In other cases, organizations such as the Joint Commission and NQF might require PRO data collection as a condition for accreditation or compliance, similar to the Joint Commission’s requirement that data on pain be adopted as a fifth vital sign.45 Health systems or hospitals could also bundle PROs together as part of the default or recommended panel of tests for a given condition. This is often done for clinical tests that might otherwise be ordered incorrectly, such as thyroid function or iron studies. For example, the HAQ (a PRO for rheumatoid arthritis) might be part of a default set of measures collected for rheumatoid arthritis patients.
As PROs are increasingly included in clinical trials, methods are needed to support literature synthesis and meta-analysis of PRO data. A fledgling effort is already underway by a wing of the Cochrane Collaborations dealing with PROs.
In summary, PROs have the potential to improve the quality and patient-centeredness of medical care in a variety of ways. They can be used at the individual patient level to improve interactions between patients and clinicians. They can also be used in research studies to identify benefits and harms of interventions. Finally, they have a role to play in policy making and population surveillance, including contributing to guideline development, informing coverage and reimbursement decisions, evaluating care quality, and identifying the impacts of policy options. The PRO data collected for these purposes can in turn be used for PCOR. However, this integration of PROs in PCOR requires linking PRO data with clinical information from a range of data sources. Electronic data methods play a pivotal role in executing this linkage.
This is an optimistic vision, and there is a great deal of research to be done before PROs will be fully embraced by all stakeholders. This will take time, as for all new technologies, as well as evidence for utility and value added. Looking ahead, research and policy initiatives are needed to facilitate the routine use of PROs in policy, research, and practice, and to enable the linkage of PRO data with other clinical data to enable PCOR.46 In doing so, it will be important to address the ethical, methodologic, and governance considerations that currently prevent optimal application of PROs in PCOR. Funders such as PCORI and the Agency for Healthcare Research and Quality should support research investigating solutions to those challenges. Addressing these barriers will be critical to putting the patient perspective in PCOR.
2. .Available at: http://www.pcori.org/
. Accessed December 10, 2012.
3. .Patient reported outcome measures: use in medical product development to support labeling claims.Federal Register.2009;74:65132–65133.
5. Acquadro C, Berzon R, Dubois D, et al..Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group Meeting at the Food and Drug Administration, February 16, 2001.Value Health.2003;6:522–531.
6. Lipscomb J, Donaldson MS, Hiatt RA.Cancer outcomes research and the arenas of application.J Natl Cancer Inst Monogr.2004;33:1–7.
7. Wu AW, Snyder C, Clancy CM, et al..Adding the patient perspective to comparative effectiveness research.Health Aff (Millwood).2010;29:1863–1871.
8. Jones JB, Snyder CF, Wu AW.Issues in the design of Internet-based systems for collecting patient-reported outcomes.Qual Life Res.2007;16:1407–1417.
9. Rose M, Bezjak A.Logistics of collecting patient-reported outcomes (PROs) in clinical practice: an overview and practical examples.Qual Life Res.2009;18:125–136.
10. .Available at: http://www.internetworldstats.com/am/us.htm
. Accessed December 10, 2012.
11. Hambleton RKLipscomb J, Gotay CC, Snyder C.Applications of item response theory to improve health outcomes assessment: developing item banks, linking instruments, and computer-adaptive testing.Outcomes Assessment in Cancer: Measures, Methods and Applications.2005.Cambridge:Cambridge University Press;445–464.
13. Devlin NJ, Appleby J.Getting the most out of PROMS: putting health outcomes at the heart of NHS decision-making.The King’s Fund.2010
14. Edge SB.American College of Surgeons Commission on Cancer: Defining and Managing Quality. Presented at George Washington Cancer Institute’s Cancer Health Policy Scholars Forum, November 9, 2011.
15. Boden-Albala B.Building a Community-Centered Translational Research Infrastructure with Electronic Data: The WICER Study. Presented to AcademyHealth.
16. Snyder CF, Aaronson NK.Use of patient-reported outcomes in clinical practice [Comment].Lancet.2009;374:369–370.
17. Wu AW, St Peter R, Cagney C.Health status assessment: completing the clinical database.J Gen Intern Med.1997;12:254–255.
18. Greenhalgh J.The applications of PROs in clinical practice: what are they, do they work, and why?Qual Life Res.2009;18:115–123.
19. .Surveys show significant proportions of hospitals and doctors already plan to adopt electronic health records and qualify for federal incentive payments. Available at: http://www.hhs.gov/news/press/2011pres/01/20110113a.html
. Accessed December 11, 2012.
20. Snyder CF, Blackford AL, Wolff AC, et al..Feasibility and value of PatientViewpoint: a web system for patient-reported outcomes assessment in clinical practice.Psychooncology.2013;22:895–901.
21. Snyder CF, Jensen R, Courtin SO, et al..PatientViewpoint: a website for patient-reported outcomes assessment.Qual Life Res.2009;18:793–800.
22. Web-based System Helps Doctors, Patients Communicate. Available at: http://www.youtube.com/watch?v=S-r4ykaUhfU
. Accessed December 11, 2012.
23. Brundage M, Osoba D, Bezjak A, et al..Lessons learned in the assessment of health-related quality of life: selected examples from the National Cancer Institute of Canada Clinical Trials Group.J Clin Oncol.2007;25:5078–5081.
24. Patrick DL, Ferketich SL, Frame PS, et al..National Institutes of Health State-of-the-Science Conference Statement: Symptom Management in Cancer: Pain, Depression, and Fatigue, July 15-17, 2002.J Natl Cancer Inst.2003;95:1110–1117.
25. Nayfield SG, Ganz PA, Moinpour CM, et al..Report from a National Cancer Institute (USA) workshop on quality of life assessment in cancer clinical trials.Qual Life Res.1992;1:203–210.
26. Moinpour CM, Feigl P, Metch B, et al..Quality of life end points in cancer clinical trials: review and recommendations.J Natl Cancer Inst.1989;81:485–495.
27. Wu AW.Quality of life assessment comes of age in the era of highly active antiretroviral therapy.AIDS.2000;14:1449–1451.
28. Liang MH, Cullen KE, Larson MG.Measuring function and health status in rheumatic disease clinical trials.Clin Rheum Dis.1983;9:531–539.
29. Wiklund I, Lindvall K, Swedberg K.Assessment of quality of life in clinical trials.Acta Med Scand.1986;220:1–3.
30. Guyatt GH, Bombardier C, Tugwell PX.Measuring disease-specific quality of life in clinical trials.CMAJ.1986;134:889–895.
31. Spilker B.Quality of Life and Pharmacoeconomics in Clinical Trials.1995.Philadelphia:Lippincott Williams & Wilkins.
32. Bing EG, Hays RD, Jacobson LP, et al..Health-related quality of life among people with HIV disease: results from the Multicenter AIDS Cohort Study.Qual Life Res.2009;9:55–63.
33. Flum DR, Devine B.SCOAP CERTN: Incorporating Patient Reported Outcomes into Prospective Cohort Study of Intermittent Claudication. Presentation to AcademyHealth EDM Forum Methods Stakeholder Symposium. October 28, 2011.
34. .Effectiveness Guidance Document: Recommendations for Incorporating Patient-Reported Outcomes into the Design of Clinical Trials in Adult Oncology. Version 10.0. Release Date: December 12, 2011.
35. .Patient Reported Outcomes Measurement Information System. Available at: http://www.nihpromis.org
. Accessed December 11, 2012.
36. Cella D, Riley W, Stone A, et al..Initial item banks and first wave testing of the Patient-Reported Outcomes Measurement Information System (PROMIS) network: 2005–2008.J Clin Epidemiol.2010;63:1179–1194.
37. Gershon R, Rothrock NE, Hanrahan RT, et al..The development of a clinical outcomes survey research application: assessment Center.Qual Life Res.2010;19:677–685.
38. Gotay CC, Kawamoto CT, Bottomley A, et al..The prognostic significance of patient-reported outcomes in cancer clinical trials.J Clin Oncol.2008;26:1355–1363.
39. Quinten C, Maringwa J, Gotay CC, et al..Patient self-report of symptoms and clinician ratings as predictors of overall cancer survival.J Natl Cancer Inst.2011;103:1851–1858.
40. Newman JC, Feldman R.Copyright and open access at the bedside.N Engl J Med.2011;365:2447–2449.
41. Emery MP, Perrier LL, Acquadro C.Patient-reported outcome and quality of life instruments database (PROQOLID): frequently asked questions.Health Qual Life Outcomes.2005;3:12.
42. Barry MJ, Fowler FJ Jr, O’Leary MP, et al..The American Urological Association symptom index for benign prostatic hyperplasia. The Measurement Committee of the American Urological Association.J Urol.1992;148:1549–1557.
43. Guyatt GH, Osoba D, Wu AW, et al..Methods to explain the clinical significance of health status measures.Mayo Clin Proc.2002;77:371–383.
44. Snyder CF, Blackford AL, Brahmer JR, et al..Needs assessments can identify scores on HRQOL questionnaires that represent problems for patients: an illustration with the Supportive Care Needs Survey and the QLQ-C30.Qual Life Res.2010;19:837–845.
45. Dahl JL, Saeger L, Stein W, et al..The new JCAHO pain assessment standards: implications for the medical director.J Am Med Dir Assoc.2000;1supplS24–S31.
46. Ahmed S, Berzon RA, Revicki DA, et al..The use of patient-reported outcomes (PRO) within comparative effectiveness research: implications for clinical practice and health care policy.Med Care.2012;50:1060–1070.