Lessons from the Electronic Data Methods Forum: Collaboration at the Frontier of Comparative Effectiveness Research, Patient-Centered Outcomes Research, and Quality Improvement

Holve, Erin PhD, MPH, MPP*; Calonge, Ned MD, MPH

Medical Care:
doi: 10.1097/MLR.0b013e31829c518f

Background: The Electronic Data Methods (EDM) Forum, with support from the Agency for Healthcare Research and Quality, exists to advance knowledge and practice on the use of electronic clinical data (ECD) for comparative effectiveness research, patient-centered outcomes research, and quality improvement (QI). The EDM Forum facilitates collaboration between the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies, Scalable Distributed Research Network, and Enhanced registry projects funded by Agency for Healthcare Research and Quality.

Objectives: This overview describes a second set of papers commissioned by the EDM Forum, published in this supplement to Medical Care. The papers that are included discuss challenges and innovations from the research and QI community using ECD.

Conclusions: The papers in this supplement provide lessons learned based on experiences building transparent, scalable, reusable networks for research and QI. Through these papers, and a new open access e-journal, eGEMs, the EDM Forum is working to advance the science of health research and QI using ECD to improve patient outcomes.

Author Information

*AcademyHealth, Washington, DC

The Colorado Trust, Denver, CO

E.H. and N.C.: contributed to the conception and design of the article and provided a significant amount of time to the drafting and revision process to produce the final manuscript.

Supported by the Agency for Healthcare Research and Quality, “AHRQ Grant U13 HS19564-01 (Electronic Data Methods Forum for Comparative Effectiveness Research).”

The authors declare no conflict of interest.

Reprints: Erin Holve, PhD, MPH, MPP, AcademyHealth, 1150 17th Street, NW, Suite 600, Washington, DC 20036. E-mail: erin.holve@academyhealth.org.

Article Outline

Since 2010, AcademyHealth has directed the Electronic Data Methods (EDM) Forum with support from the Agency for HealthCare Research and Quality (AHRQ). The EDM Forum's purpose is to advance the knowledge and practice of the use of electronic clinical data (ECD) for comparative effectiveness research (CER), patient-centered outcomes research (PCOR), and quality improvement (QI). Facilitating exchange and collaboration between the eleven Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies (PROSPECT), Scalable Distributed Research Network (DRN), and the Enhanced Registry for QI and CER projects funded by AHRQ as part of the American Recovery and Reinvestment Act is a central task of the Forum. Over the past few years, the EDM Forum has undertaken a range of activities, including workshops, symposia, issue briefs, commissioned papers, and collaborative methods projects. These efforts are guided by the EDM Forum Steering Committee, a 19-member expert panel working with the EDM Forum staff to convene and foster collaboration between the research, provider, patient, policy, and payer communities, and other stakeholders with an interest in using ECD for research and QI.

An important early effort was commissioning papers to enhance understanding of key technical and nontechnical issues facing the PROSPECT, DRN, and Enhanced Registry projects, as well as discussing innovative approaches to using ECD. Topics for the commissioned work focus on the 4 domains of the EDM Forum: analytic methods; clinical informatics; governance of multi-institutional networks using ECD for research and QI; and ways in which new infrastructure contributes to building a “learning health system.” All of the papers and collaborative work have a pragmatic focus, highlighting generalizable lessons learned to guide future efforts to learn from—and drive improvement in—patient care and outcomes based on ECD. This current special issue in Medical Care is the latest contribution to these efforts.

The first set of papers commissioned by the EDM Forum included perspectives from 41 collaborators at 30 institutions participating in the PROSPECT, DRN, and Enhanced Registry projects. These papers, published in Medical Care in June 2012, addressed issues ranging from conceptual models for evaluating data variability and quality in multisite research studies1 to a description of ECD informatics platforms for CER2 to approaches to achieve Institutional Review Board approval for multisite research networks in CER, PCOR, and QI.3

To complement these first set of topics selected by the EDM Forum’s Steering Committee, a call for commissioned papers was released to further examine current challenges and opportunities for conducting CER, PCOR, and QI with ECD. This call was opened to all investigators in the community working on relevant topics for the EDM Forum with the intent of broadening participation in the Forum. In addition to the core domains of methods, informatics, and governance, papers on using ECD in learning health systems were invited. The EDM Forum Steering Committee subcommittees (organized by domain) reviewed the submissions and assessed the extent to which the proposed papers had potential to advance understanding of key issues when using ECD for research and QI and provide useful, generalizable lessons for stakeholders. The reviewers ultimately selected 14 of the 42 submissions received. An additional set of papers were developed based on work by collaborators in the Clinical and Translation Science Award (CTSA) program, one of which was also supported by the EDM Forum. Many of these papers respond to the challenges identified in the first set of published papers.4 In total, the second set of papers includes work from 55 collaborators at 34 institutions.

This current Medical Care supplement is organized by the four domains of the Forum, and highlights a set of useful and four important lessons for building infrastructure to generate evidence and improve patient outcomes. In total, there are 12 commissioned papers in the special issue, with three papers in each of the domains.

Of the papers in the methods domain, the first, by Toh et al5 addresses the critical issue for many multisite CER studies of assessing the strengths and limitations of various confounding adjustment approaches for observational CER studies, as well as major issues to consider when selecting among strategies to adjust for confounding. In the second methods paper, Raebel et al6 consider current inconsistencies in defining medication adherence—based on 315 published studies, and propose a unifying set of definitions for these concepts as well as a conceptual framework to operationalize these definitions consistently across future studies. In the third paper on methods, Brown et al7 review existing ad-hoc approaches for checking data quality and recommend steps for identifying and addressing data quality concerns in multisite research networks, building on the work of Kahn and colleagues.1

For the informatics papers, Hersh and colleagues8 from the CTSA CER Informatics Taskforce review a broad set of issues to consider when using operational electronic health record (EHR) data for clinical research. Embi et al9 provided a set of three case examples to highlight some of the knowledge management and informatics challenges common to the reuse of ECD for research. Ogunyemi et al10 address issues with representing and mapping data for CER using EHR and medical claims data through the lens of common or reference data models. The authors compare the strengths and weaknesses of existing data modeling standards—such as those from Observational Medical Outcomes Partnership, the Biomedical Research Integrated Domain Group, and the Clinical Data Interchange Standards Consortium—as well as assessing the impact of having a common data model on data collection, mediation, and exchange.

The governance papers discuss issues related to patient privacy and data security. Whicher et al11 propose a new ethical framework for thinking about patient consent in prospective QI and research studies that emphasizes the balance of risk and burden to participants and makes a case for minimizing Institutional Review Board and privacy board review of studies conducted in “learning systems.” Jiang et al12 conduct a review of data sharing techniques for CER, finding that some new privacy-protection techniques can be applicable for CER. Kim et al13 discuss an analysis of privacy and security laws and state health information exchange guidelines from California, Illinois, Massachusetts, and the Federal Veteran’s Administration, and reviews the resulting policy framework that was developed for the SCAlable National Network for Effectiveness Research project to propose a use-case approach to achieve data sharing.

The learning health system papers largely address issues and strategies to maximize the use of patient-level data across systems for research, QI, and patient care. Snyder et al14 consider the unique contributions of patient-reported outcomes (PRO) to research and QI and discuss the critical role of EDM for facilitating the use of PRO data in PCOR, the key challenges and unanswered questions that need to be addressed to achieve routine use of PROs in PCOR and policy and research interventions that can accelerate the integration of PROs with clinical data. Bayley et al15 present the origins and nature of the challenges encountered in conducting CER studies, and they suggest strategies to deal with these issues based on the experiences of four leading health systems using EHR data for research. Finally, Embi and Payne16 propose a paradigm shift from “evidence-based medicine” to the concept of “evidence-generating medicine” (EGM), which redefines research and practice as a bidirectional relationship the authors assert could accelerate much-needed systems-level change in health care.

The current set of commissioned papers included in this supplement demonstrates significant progress towards our desired goal of engaging stakeholders beyond the PROSPECT, DRN, and Enhanced Registry projects, and sharing early lessons learned from infrastructure efforts at the frontier of CER, PCOR, and QI. As evidence of the EDM Forum’s ongoing connections and projects with other relevant entities, collaborators include individuals from the Health Maintenance Organization Research Network, CTSA CER Informatics Taskforce, the High Value Health Care Collaborative, AHRQ’s Developing Evidence to Inform Decisions about Effectiveness Network, and others working with the Observational Medical Outcomes Partnership, the Office of the National Coordinator Query Health Initiative, and current grantees of the Patient-centered Outcomes Research Institute (PCORI). In addition, continued outreach and collaboration has garnered participation from >7,000 individuals who have registered for webinars and stakeholder symposia, or used the EDM Forum's websites, including http://www.edm-forum.org.

The newest member of this family of resources is the EDM Forum’s new open access e-journal, eGEMs (Generating Evidence and Methods to improve patient outcomes). With the support of an expert, multidisciplinary panel of senior editors,17 eGEMs launched in late January 2013 to provide a peer-reviewed publishing venue for papers focused on “learning how to learn from ECD.”18 To date, eGEMs’ published papers have included commentaries, methods papers, and case studies. For example, in “Security approaches in using tablet computers for primary data collection in clinical research,”19 Wilcox et al20 build on the authors’ 2012 Medical Care paper addressing the advantages and disadvantages of mobile devices (ie, iPads) for community-level data collection of personal health information (PHI). In eGEMs, Wilcox et al19 discuss the selection and implementation of a the Health Insurance Portability and Accountability Act (HIPAA)-compliant, reasonably cost-effective strategy for collecting PHI on a tablet computer as part of a community-wide health study. The paper also lays out a framework for thinking about security issues in this context and describes the options considered by the investigators. Thus far, the large number of downloads for the Wilcox paper and others indicate sincere interest from the community in publishing best practices and generalizable lessons learned from innovators in the field.

The EDM Forum has continued to focus on its goal of facilitating collaboration between multiple disciplines and stakeholder perspectives. Collaborative science can be a long road to travel, and requires vision. This work would not be possible without the leadership of Carolyn Clancy, MD, Director AHRQ, Jean R. Slutsky, Director, Center for Outcomes and Evidence, AHRQ, and especially, Gurvaneet Randhawa, MD, MPH, who serves as the project officer of the EDM Forum project and the portfolio of PROSPECT, DRN, and Enhanced Registry projects. Dr Randhawa’s role guiding the development of the American Recovery and Reinvestment Act-CER infrastructure portfolio, consistent support for this work, and philosophy emphasizing the need for transparency and sharing lessons learned at every step of these projects has been crucial to fostering collaboration and rapid learning to benefit the broader community. We also extend thanks to Lisa Simpson, MB, BCh, MPH, FAAP, President and CEO of AcademyHealth, and the EDM Forum staff. We acknowledge and thank the members of the EDM Forum Steering Committee, especially the subcommittee chairs: Lisa Schilling, MD, MSPH; John F. Steiner, MD, MPH; and Clete Kushida, MD, PhD, RPSGT.

The set of papers in the current supplement are an important subset of the lessons learned to building more transparent, scalable, reusable networks for health research and QI. There is much learning ahead, and lots of unanswered questions about the future of this nascent infrastructure in light of current fiscal pressures.21 Sustainability is a key consideration, and an area in which the EDM Forum is working with the projects to accelerate discussion of structures, services, and successful business models that highlight the value of network-based research.22 One of the principal lessons that emerges from the current and prior EDM Forum supplements in Medical Care is that support for infrastructure has enabled a new and creative approache to conducting “big science” in health and health care. By sharing these early lessons learned, we hope the national dialog facilitated by the EDM Forum is advancing the scientific evolution of multisite health research using ECD in the service of improving patient outcomes.

Back to Top | Article Outline


1. Kahn MG, Raebel MA, Glanz JM, et al..A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research.Med Care.2012;50suppl 1S21–S29.
2. Sittig DF, Hazlehurst BL, Brown J, et al..A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogenous clinical data.Med Care.2012;50suppl 1S49–S59.
3. Marsolo K.Approaches to facilitate institutional review board approval of multicenter research studies.Med Care.2012;50suppl 1S77–S81.
4. Holve E, Segal C, Hamilton Lopez M.Opportunities and challenges for comparative effectiveness research (CER) with electronic clinical data: a perspective from the EDM forum.Med Care.2012;50suppl 1S11–S18.
5. Toh SD, Gagne JJ, Rassen JA, et al..Confounding adjustment in comparative effectiveness research conducted within distributed research networks.Med Care.2013;51suppl 3S4–S10.
6. Raebel MA, Schmittdiel J, Karter AJ, et al..Standardizing terminology and definitions of medication adherence and persistence in research employing electronic databases.Med Care.2013;51suppl 3S11–S21.
7. Brown JS, Kahn M, Toh D.Data quality assessment for comparative effectiveness research in distributed data networks.Med Care.2013;51suppl 3S22–S29.
8. Hersh WR, Weiner MG, Embi PJ, et al..Caveats for the use of operational electronic health record data in comparative effectiveness research.Med Care.2013;51suppl 3S30–S37.
9. Embi PJ, Hebert C, Gordillo G, et al..Knowledge management and informatics considerations for comparative effectiveness research: a case-driven exploration.Med Care.2013;51suppl 3S38–S44.
10. Ogunyemi OI, Meeker D, Kim H-E, et al..Identifying appropriate reference data models for Comparative Effectiveness Research (CER) studies based on data from clinical information systems.Med Care.2013;51suppl 3S45–S52.
11. Faden R, Kass N, Whicher. D, et al..Ethics and informed consent for comparative effectiveness research with prospective electronic clinical data.Med Care.2013;51suppl 3S53–S57.
12. Jiang X, Sarwate AD, Ohno-Machado L.Privacy technology to support data sharing for comparative effectiveness research: a systematic review.Med Care.2013;51suppl 3S58–S65.
13. Kim KK, McGraw D, Mamo L, et al..Development of a privacy and security policy framework for a multistate comparative effectiveness research network.Med Care.2013;51suppl 3S66–S72.
14. Snyder CF, Jensen RE, Segal JB, et al..Patient-reported Outcomes (PROs): putting the patient perspective in patient-centered outcomes research.Med Care.2013;51suppl 3S73–S79.
15. Bayley KB, Belnap T, Savitz L, et al..Challenges in using electronic health record data for CER: Experience of 4 learning organizations and solutions applied.Med Care.2013;51suppl 3S80–S86.
16. Embi PJ, Payne PRO.Evidence generating medicine: redefining the research-practice relationship to complete the evidence cycle.Med Care.2013;51suppl 3S87–S91.
17. .Editorial Board. Available at: http://repository.academyhealth.org/egems/editorialboard.html. Accessed April 1, 2013.
18. Simpson L.Learning how to learn: how AcademyHealth is supporting evidence generation in a transforming world. eGEMs (Generating Evidence and Methods to improve patient outcomes). 2013;1(1). Available at: http://repository.academyhealth.org/egems/vol1/iss1/1. Accessed April 15, 2013.
19. Wilcox AB, Gallagher KD, Bakken SR.Security approaches in using tablet computers for primary data collection in clinical research. eGEMs 2013;1(1). Available at: http://repository.academyhealth.org/egems/vol1/iss1/7. Accessed April 15, 2013.
20. Wilcox AB, Gallagher KD, Boden-Albala B, et al..Research data collection methods: from paper to tablet computers.Med Care.2012;50suppl 1S68–S73.
21. McDonough JE.Budget Sequestration and the US Health Sector.N Engl J Med.2013Available at: http://www.nejm.org/doi/full/10.1056/NEJMp1303266. Accessed April 1, 2013.
22. Holve E.Ensuring support for research and quality improvement (QI) networks: four pillars of sustainability—an emerging framework. eGEMs. 2013;1(1). Available at: http://repository.academyhealth.org/egems/vol1/iss1/5. Accessed April 1, 2013.

analytic methods; clinical informatics; governance; learning health system; comparative effectiveness research; patient centered outcomes research; quality improvement; collaborative science

© 2013 by Lippincott Williams & Wilkins.