The Institute of Medicine (IOM) Roundtable on Evidence-Based Medicine's Learning Healthcare System workshop (July 2006) launched a new healthcare paradigm. The initiative derived momentum from concerns regarding patient safety,1 quality of care,2 and healthcare delivery and outcomes.3 Among the causes of cited inadequacies, the IOM noted a growing problem in the “structural inability of evidence to keep pace with the need for better information to guide clinical decision-making,”4 The tightly controlled nature of clinical trials and a labyrinth of regulatory requirements compound the growing gap between evidence generation and translation.
In the emerging healthcare paradigm, intelligent use of the full range of available evidence ensures optimal quality, safety, and effectiveness of treatments. A new concept—the learning healthcare system—has formed around 3 fundamental purposes: (1) to generate and apply the best evidence relevant to each patient; (2) to propel scientific discovery “as a natural outgrowth of patient care;” and (3) to support quality assessment and improvement, spark innovation, enhance patient safety, and maximize healthcare value.4,5 Comparative effectiveness research (CER)—“real-world” studies comparing treatment choices, with broad eligibility criteria and long-term outcomes such as clinical benefit, quality of life (QOL), and disability4—addresses these 3 purposes. Differing fundamentally from the efficacy trials of the last 2 decades, CER is made possible by the establishment of a rapid learning healthcare infrastructure. In a recent report on national priorities for CER, the IOM clarified this need for critical infrastructure: “A large public-private CER enterprise will require a supporting infrastructure to efficiently move the science forward. In addition to the capacity to support high-efficiency, pragmatic randomized trials, the program will require large-scale clinical and administrative data networks that enable observational studies of patient care while protecting patient privacy and data security. New methods for linking patient-level data from multiple health care organizations will promote inclusion of populations frequently omitted from clinical trials.”6
Advances in information technology have both shaped the rapid learning healthcare vision and made it attainable.7 New information technology will also support the development of decision support systems that will help clinicians temper professional judgment with clinically relevant evidence, to prevent undesirable outcomes or unintended harm to patients.4 A powerful integrated data system comprising multiple types of linked information underlies and drives rapid learning healthcare; data sources include the basic sciences such as genomics, proteomics, metabolomics; clinical research and outcomes datasets; clinical care; administrative data; and patient-reported outcomes (PRO) data.
PROs are measures of patients’ physical symptoms, side effects, QOL, psychosocial experiences, and satisfaction with care. An umbrella term, a PRO is now generally considered to be any end point derived from patient reports, whether the data be collected at the clinic, in a diary, or by other means; methods of PRO data collection include single-item outcome measures, event logs, symptom reports, and formal assessment instruments.8 PROs have increasingly gained acceptance as important and valid measures of patients’ symptoms, experiences, and QOL.9–11 They are a vital component of CER, as we seek to understand the impact of available treatments on patients’ experiences and well-being. New technologies including tablet computers, handheld electronic devices, interactive voice response systems, and diaries, offer multiple mechanisms for posing questions directly to patients. Although various technologies are now being used to collect and study PROs, to our knowledge no other medical center has developed a PRO-based system for implementing rapid learning healthcare at the clinic level.
To ensure that patients’ symptoms remain central outcomes of inquiry and that learning seeks to improve measures important to patients, it is critical that PROs be one of the fundamental building blocks of rapid learning healthcare. Routine collection of patient-reported data supports a system of data linkage that starts at the patient level—beginning with the patient's chief complaint, incorporating additional information to develop a full view of the patient's status, and integrating additional data linked at the individual level. This specificity to the patient is the essence of personalized medicine and, indeed, of good clinical practice.
This article provides a description of 1 academic medical center's effort to develop a rapid learning healthcare model in the cancer clinic. We present: (a) results of 2 demonstration projects, each of which tested a vital component of the overarching rapid learning healthcare model; and, (b) results of 2 use cases, in which we used aspects of the new infrastructure to support learning and translation at the clinic level.
Because this was an infrastructure development project, the methods presented below do not adhere to the standard format for presentation of research results. Instead, we describe as Methods our development and testing of key components of the envisioned rapid learning cancer clinic, and as Results 2, preliminary use cases of these components, serving as harbingers of the potential for improvement of care under a rapid learning healthcare paradigm. To maintain the focus of this article on the development of a new model (rather than on the studies themselves) and for space considerations, we do not provide detail on the separate studies; interested readers are encouraged to read published results.
To prepare for rapid learning healthcare, in which ongoing CER is the modus operandi, we advocate a staged approach beginning with operationalization of a rapid learning cancer clinic model, using electronic PROs (ePROs) as basic building blocks and the ePRO dataset as the core database, and subsequently linking additional datasets to the ePRO dataset (Fig. 1). We have initiated this process in the Duke cancer clinics, with efforts organized into 6 task areas: (1) establish feasibility of the ePRO system; (2) validate data collected by the ePRO system; (3) link data stored in the ePRO system with data residing in other institutional datasets; (4) operationalize the system in the cancer clinic; (5) attain buy in from diverse stakeholders including oncologists, nurses, clinic staff, researchers, and administrators; and (6) demonstrate rapid learning cancer care through use cases. These steps will lead to preliminary construction of a replicable, well-functioning, rapid learning cancer clinic model. The sections below describe the accomplishment of tasks 1, 2 (presented as Methods), and 6 (presented as Results).
Task 1: Establish Feasibility Study of the ePRO System
We first conducted a feasibility study to determine whether patients with cancer in the academic setting find a data collection technology—wireless tablet computers (“e/Tablets”)—logistically acceptable and satisfactory for communicating symptoms. The study enrolled patients (n = 65) with metastatic breast cancer with prognosis of ≥6 months. e/Tablets were programmed with several well-recognized assessment instruments and the Patient Care Monitor [PCM, a comprehensive 86-item review of systems, rating patients’ experiences on a 0–10 numerical rating scale (NRS)] and a satisfaction survey. At 4 clinic visits within 6 months, participants used e/Tablets before their visit to complete the surveys.
Participants’ mean age was 54 years [standard deviation (SD) 12]; 77% were white; 47% had less than a college degree. Regarding the e/Tablets, 94% reported that they were easy to read; 98%, easy to use to answer survey questions; 99%, easy to navigate; and 90%, a comfortable weight. Initially, 75% indicated satisfaction with the PCM for reporting symptoms; this proportion increased over time. By the last visit, 88% indicated willingness to recommend PCM to other patients, and 74% felt that the e/Tablet helped them remember symptoms to report to their doctors. Patients enjoyed the cancer-related educational content which they could access after completing surveys. We concluded that e/Tablets are a feasible and acceptable method of PRO data collection in an academic oncology clinic12; results were confirmed in 2 additional cancer populations—gastrointestinal (GI) and lung cancer.
Task 2: Validate an ePRO Data Collection Approach Across 3 Cancer Types
We next performed a validation study to determine whether data collected using the ePRO system were equivalent to data collected using standard paper-based assessment instruments. Sequential studies in 3 distinct clinics enrolled patients with breast cancer (n = 65), GI cancer (n = 113), and lung cancer (n = 97). At 4 visits within a 6-month period, participants used e/Tablets to complete multiple assessment surveys including the Functional Assessment of Cancer Therapy-General scale (FACT-G), MD Anderson Symptom Inventory (MDASI), and PCM. Additionally, at each visit, the participant completed a paper version of one of the electronic assessment instruments (eg, FACT-G using e/Tablet and FACT-G on paper); paper instruments were completed before the electronic assessments. Cronbach alpha coefficients were calculated to verify internal consistency. Differences between electronic and paper scores were compared using raw scores and paired Student t tests; results were verified with the Sign test.
All Cronbach alpha coefficients were acceptable (>0.66). Responses collected by paper and electronic methods were statistically similar for both MDASI subscales (Severity and Interference) and for the FACT-G Physical and Emotional Well-being subscales. Paper and electronic scores on the FACT-G Social Well-being subscale differed statistically within breast and lung cancer cohorts. We concluded from this study that ePRO data on most subscales were sufficiently similar to consider ePRO data as being “research quality.”13,14
Implementation in 3 sites confirmed the logistical feasibility of this method of data collection. Use of e/Tablets did not disrupt the clinic flow. No internet or data security problems arose. The color-coded PCM reports, highlighting moderate and severe symptoms, were well received by clinicians.
Having demonstrated 2 critical aspects of the rapid learning cancer clinic model—feasibility of the data collection methodology and validity of data collected using the electronic data infrastructure—we piloted the use of the ePRO system to support, in a preliminary fashion, rapid learning cancer care. In the 2 studies described below, we illustrate the use of the ePRO infrastructure to test the potential for rapid learning healthcare in an academic cancer clinic. Herein, we focus on model development and demonstration rather than on research results; interested readers are encouraged to read the studies’ results in the published literature.
Task 6: Demonstrate Rapid Learning Cancer Care Through Use Cases
A: Sexual Distress Among Patients With GI and Breast Cancer
Identification of a Problem
Sexual distress is recognized in various cancer populations including breast,15 cervical,16 and rectal,17 but oncologists typically sidestep the issue because of lack of time, training, or confidence to handle the matter appropriately. To determine how commonly oncology patients report sexual problems and how these concerns relate to other PROs, we analyzed data collected from the GI and breast cancer cohorts described above. Measures included sexual problems on a 0 to 10 NRS (using the PCM), QOL (FACT-G), MDASI, and National Comprehensive Cancer Network (NCCN) Distress Scale.
Patients with GI cancer (n = 113) were mean age 58 years (SD 11), 68% men, 80% white. Mild (NRS 1–3) sexual problems were reported by 20%, moderate (NRS 4–6) by 17%, and severe (NRS > 6) by 19% of the cohort. As severity of sexual problems increased, mean FACT-G Functional Well-Being decreased from 20.4 (SD 5.0) for patients reporting no sexual problems to 13.0 (SD 6.3) for those reporting severe problems at baseline. Severity of sexual problems correlated highly with all FACT-G subscales: Physical (−0.35, P < 0.001), Social (−0.32, P = 0.001), Functional (−0.50, P < 0.0001), Emotional (−0.27, P = 0.004), and Total (−0.48, P < 0.0001). Similar patterns emerged for MDASI Severity, MDASI Interference, and NCCN Distress. Among patients with breast cancer (cohort described above), 24% reported mild sexual problems, 11% moderate, and 18% severe. In patients with breast cancer, correlations of sexual problems with other PROs were similar to those in patients with GI cancer. Overall, this study found that reductions in sexual enjoyment, interest, and performance were common, stable, and correlated with decreases in a broad range of PROs including QOL and distress.18
Characterization of the Problem
We next compared sexual problems in patients with GI cancer vs. patients breast cancer, and examined associations between sexual problems and other PROs using linear mixed model analyses. Mean sexual problems were similar in GI cancer patients and breast cancer patients at baseline (P = 0.49) and over time (P = 0.63). For patients with GI cancer, gender was not significantly associated with sexual problems (P = 0.17). In both samples, sexual problems were significantly associated with FACT-G (P < 0.002) and MDASI Interference (P < 0.02). For patients with GI cancer, there were also significant effects of sexual problems on MDASI Severity (P = 0.001) and NCCN Distress (P = 0.01). The relationship between sexual problems and outcomes did not differ by cancer type (P > 0.56) or over time (P > 0.15). Reports of sexual problems were similar and remained stable in GI and breast cancer samples. Sexual problems were significantly associated with poorer QOL and disease interference irrespective of length of time since diagnosis.19
Development of an Intervention to Address the Problem
To address identified sexual distress of cancer patients, we convened physicians, nurses, and psychologists to develop a response plan. The nurses focused on developing a patient education package to be delivered by e/Tablets. The psychologists focused on new coping interventions; a postdoctoral psychology fellow designed a cognitive-behavioral intervention using a model of coping flexibly with illness-related sexual concerns,20 and secured funding from the American Cancer Society. Her study is testing the efficacy of a skills-based telephone intervention addressing the sexual/intimacy concerns of patients with colorectal cancer and their partners.
Evaluation of the Intervention's Impact on the Identified Problem
The study uses the e/Tablet-based ePRO system to collect, warehouse, and analyze sensitive patient-reported data. The ePRO system assists in participant screening and triage to the research intervention; for example, when a patient with GI cancer reports sexual distress of >7 on a 0–10 NRS, the patient is immediately asked whether he/she would like to learn more about the study, and, if so, the study coordinator is contacted. The study opened approximately 3 years after collection of the first patient-reported sexual distress data at Duke and within 1 year of the analysis which highlighted the need for an intervention.
B: Evaluation of a Psychosocial Care Program for Patients With Cancer
Identification of a Problem
Psychologic distress—including anxiety, depression, and mood disturbance—affects >35% of patients with cancer.21 An IOM report, media coverage, and published research have drawn national attention to the fact that many patients with cancer need assistance adjusting to life with a potentially fatal illness.22 However, the evidence base contains a paucity of studies examining the effectiveness of interventions targeting psychosocial needs of patients with cancer.
Evaluation of an Intervention's Feasibility
Pathfinders, a program for patients with cancer, seeks to relieve psychologic distress through a strength-based model that uses cognitive restructuring, coping skills, complementary/alternative and mind/body techniques, and lifestyle counseling. In a pilot study, we gauged the feasibility of incorporating Pathfinders into cancer care in the academic setting, and gathered preliminary data on efficacy.23 This prospective, single-arm, study enrolled metastatic breast cancer patients who met with a trained social worker (ie, a Pathfinder) at least monthly for 6 months. Participants (n = 50) had a mean age of 51 years (SD 12); 24% were nonwhite; 74% were married; and 50% did not complete college. Attrition from death at 6 months was 18%. Of participants completing the 3-month assessment (n = 42), 29 (69%) were asked the Pathfinders helpfulness question on a paper questionnaire; 27 of 29 (93%) indicated that Pathfinders was helpful, and 2 provided no response.
Evaluation of the Intervention's Impact
At baseline, month 3, and month 6, participants used e/Tablets to complete surveys including the PCM and Functional Assessment of Chronic Illness Therapy—Fatigue (FACIT-F) subscale. Scores on the PCM Distress subscale improved from baseline to 3 months with a mean change of −3.42 (n = 36; P = 0.008) and from baseline to 6 months of −4.11 (n = 28; P = 0.002). PCM Despair subscale scores also improved: mean change −4.53 (P = 0.006) and −6.93 (P = 0.016), respectively. PCM QOL and FACIT-F subscale scores improved from baseline to 3 months; the change at 6 months, with a smaller sample, was not statistically significant. Mean change in QOL from baseline to 3 and 6 months was 2.88 (n = 30; P = 0.006) and 2.66 (n = 25; P = 0.079), respectively. We concluded that, in an uncontrolled pilot study, Pathfinders had significant positive effect on key psychosocial outcomes, notably distress and despair, for patients with cancer despite advanced disease and worsening symptoms.24–26
New Directions Based on Results of the Pilot Efficacy Study
Pathfinders pilot study results indicated potential efficacy in ameliorating distress and a perception among patients that the intervention helped them. Data from participating patients have been securely warehoused, to enable longitudinal analyses of outcomes. Next steps are to (1) open a registry study to include additional cancer populations; (2) explore predictors of response and clinically efficient measures of distress; (3) study the program’s impact on cost and health service utilization; and (4) conduct a comparative effectiveness study that evaluates Pathfinders’ impact relative to that of other options (eg, patient navigation). Data gathered from each initiative, as well as new methodology, will support further development of the central ePRO system to strengthen the dataset, enable additional analyses, and inform clinical decisions. Furthermore, lessons learned in the Pathfinders pilot study, together with and PCM distress and despair data from a general clinic patient sample, have contributed to the development of a psychosocial care triage algorithm, so that the intensity of psychosocial care provided is better matched to the level of distress/despair reported. The overall level of psychosocial distress in the clinic is being monitored as each of these interventions is introduced.
CONCLUSIONS AND FUTURE DIRECTIONS
This model development and demonstration project applied, at the local/clinic level, rapid learning healthcare principles: real-time data collection (task 1); rapid analysis of data and feedback to inform subsequent clinical care (task 6); use of powerful large scale, coordinated datasets as an evidence base to support clinical decisions and personalized care (in microcosm under task 6); and continuous monitoring and improvement in outcomes to support patient safety, and quality of care. Our purpose in this project was to show how rapid learning healthcare might be implemented and structured, function in clinical practice, and support CER.
We assumed that patient centeredness should be a central attribute of the new healthcare paradigm and that CER should evaluate interventions’ impact on patients’ symptoms and experiences. Therefore, we designed a rapid learning healthcare prototype using ePROs as the first building blocks. A previously developed and tested (ie, through tasks 1 through 5) PRO data collection infrastructure enabled us to explore and demonstrate a preliminary version of a rapid learning cancer clinic model (ie, through task 6). We started with concepts fundamental to patient centeredness, namely, use of PROs to describe the patient experience, and of interventions to improve that experience. These data represent describe an important aspect of effectiveness.
Scalability, an important feature of a nationally relevant rapid learning cancer clinic model, requires the linkage of heterogeneous and dispersed datasets. To determine whether diverse data can be integrated at the clinic level, we practiced linking administrative and clinical data with ePRO data and found it feasible. By locally conquering the logistics of data linkage, we aimed to ensure that the system can grow to accommodate other clinic- and health system-level issues (eg, evaluating comparative effectiveness of treatments, monitoring quality of care, and translating basic science findings into clinical practice). Results of local innovation will fuel rapid learning cancer care at the national and societal levels (Fig. 2), making CER possible across institutions and health systems.
With this model's utility for CER readily apparent, it was important to demonstrate that data can be collected at different sites and seamlessly integrated into a single database. We accomplished this important step in our validation studies, which were conducted at 2 academic medical centers and 1 community hospital. We showed that ePROs can provide detailed data on the individual patient's experience, to inform the clinical visit, while also building aggregate datasets to evaluate different treatment strategies and their outcomes.
In the rapid learning cancer clinic model, learning becomes increasingly possible as the database grows. System uses—including clinician queries, retrospective studies, and various sorts of data mining—will be joined by prospective studies as the size of the database increases. Currently, the ePRO system is being modified to add triggers, so that when specific thresholds are reached patients are presented with new options. For example, if a patient nominates troublesome insomnia (eg, >5/10), then she might be offered an educational session on healthy sleep habits or information on a relevant clinical trial. Similar triggers are envisioned for the spectrum of data, ranging from biospecimens to satisfaction and delivering content to clinicians and patients.
Institutionally, certain steps are necessary to prepare for rapid learning healthcare and its core research activity, CER. Data collection processes in targeted clinics will need to be linked to the ambulatory electronic health record and the electronic patient portal. Data collection and reporting will need to have a common ontology, which includes psychosocial and QOL elements and logically supports incorporation of patient safety triggers. A variety of virtual service options could be considered as alternatives to more personnel-intensive approaches; for example, the infrastructure could be used to test the relative effectiveness of virtual counseling versus face-to-face counseling for patients with cancer who indicate psychosocial concerns. Options for patients with low health literacy, or for those lacking comfort with technology interfaces, will need to be incorporated into the system. Discrete data elements should be defined to transcend transitions between oncology, primary care, general internal medicine, and other disciplines. Importantly, research at each step of implementation will be essential to confirm the validity of data; further assess the feasibility, acceptability, and value of the new processes; practice use of the new infrastructure and methodology; and engrain the inquiry-discovery-translation-evaluation cycle within the culture of the clinic.
The rapid learning cancer clinic embodies a new vision of cancer care. In this vision, data collected at the individual patient level will inform care for that person, contribute to local evidence development and implementation projects, and be available for large-scale evidence synthesis, CER, and evidence implementation on the health system and societal levels (Fig. 1). The system will “learn,” in that multiple stakeholders will regularly use data contained in the system to analyze and improve outcomes, safety, and quality and other measures relevant to patient, provider, and institution (Fig. 3). The rapid learning cancer clinic will be linked with other elements of the healthcare delivery system, including primary care, a patient portal to electronic information, safety, quality monitoring, and cost and efficiency initiatives. Ultimately, this model may contribute to healthcare reform that achieves optimal outcomes for the individual both effectively and efficiently.
1. Institute of Medicine. To Err is Human: Building a Safer Health System.
Washington, DC: The National Academies Press; 2000.
2. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century.
Washington, DC: The National Academies Press; 2001.
3. Wennberg JE. Unwarranted variations in healthcare delivery: implications for academic medical centres. [see comment]. BMJ
4. Institute of Medicine. The Learning Healthcare System: Workshop Summary (IOM Roundtable on Evidence-Based Medicine)
. Washington, DC: The National Academies Press; 2007.
5. Etheredge LM. A rapid-learning health system. Health Aff (Millwood)
6. Institute of Medicine. Initial National Priorities for Comparative Effectiveness Research.
Washington, DC: The National Academies Press; 2009.
7. Slutsky JR. Moving closer to a rapid-learning health care system. Health Aff (Millwood)
8. Willke RJ, Burke LB, Erickson P. Measuring treatment impact: a review of patient-reported outcomes
and other efficacy endpoints in approved product labels. Control Clin Trials
9. Coates A, Porzsolt F, Osoba D. Quality of life in oncology practice: prognostic value of EORTC QLQ-C30 scores in patients with advanced malignancy. Eur J Cancer
10. Dancey J, Zee B, Osoba D, et al. Quality of life scores: an independent prognostic variable in a general population of cancer patients receiving chemotherapy. The National Cancer Institute of Canada Clinical Trials Group. Qual Life Res
11. Cella D, Webster K. Linking outcomes management to quality-of-life measurement. Oncology (Williston Park)
12. Abernethy AP, Herndon JE II, Wheeler JL, et al. Feasibility and acceptability to patients of a longitudinal system for evaluating cancer-related symptoms and quality of life: pilot study of an e/Tablet data-collection system in academic oncology. J Pain Symp Manage
13. Abernethy AP, Herndon JE II, Wheeler JL, et al. Improving health care efficiency and quality using tablet personal computers to collect research-quality, patient-reported data. Health Serv Res
14. Abernethy AP, Zafar SY, Coeytaux R, et al. Electronic patient-reported data capture as the foundation of a learning health care system. J Clin Oncol.
15. Barni S, Mondin R. Sexual dysfunction in treated breast cancer patients. Ann Oncol
16. Schover LR, Fife M, Gershenson DM. Sexual dysfunction and treatment for early stage cervical cancer. Cancer
17. Hendren SK, O'Connor BI, Liu M, et al. Prevalence of male and female sexual dysfunction is high following surgery for rectal cancer. Ann Surg
18. Reese JB, Shelby R, Keefe FJ, et al. Sexual problems and psychosocial outcomes: a comparison of GI and breast cancer patients. Paper presented at the Society of Behavioral Medicine 30th Annual Meeting
, Montreal, Canada; 2009.
19. Reese JB, Keefe FJ, Herndon JEI, et al. Reduced sexual enjoyment, interest, and performance are a concern of GI and breast cancer patients. Support Care Cancer
20. Barsky JL, Friedman MA. Rosen RC. Sexual dysfunction and chronic illness: the role of flexibility in coping. J Sex Marital Ther
21. Zabora J, BrintzenhofeSzoc K, Curbow B, et al. The prevalence of psychological distress by cancer site. Psycho-Oncology
22. Institute of Medicine. Cancer Care for the Whole Patient: Meeting Psychosocial Health Needs.
Washington, DC: The National Academies Press; 2008.
23. Abernethy AP, Herndon JE, Coan A, et al. Phase 2 pilot study of Pathfinders: a psychosocial intervention for cancer patients. Support Care Cancer
. in press.
24. Abernethy AP, Staley TS, Shaw HS, et al. Pathfinders, an integrative psychosocial care program, improves quality of life, psychological measures, and fatigue in metastatic breast cancer patients. Presented at the 5th International Conference of the Society for Integrative Oncology.
Atlanta, GA: Best of the SIO Meeting session; 2008: Abstract 5.
25. Abernethy AP, Staley T, Shaw H, et al. Pathfinders: A psychosocial complement to cancer treatment. Presented at the American Society of Clinical Oncology Breast Cancer Symposium
. Washington, DC; 2008: Abstract 251.
26. Abernethy AP, Staley T, Herndon JEI, et al: A pilot study of an integrative psychosocial care program for cancer patients. Support Care Cancer