Project Extension for Community Healthcare Outcomes (ECHO) is a telementoring and capacity-building intervention developed at the University of New Mexico in response to disparities in access to and the quality of hepatitis C care across the state.1 Project ECHO uses a “Hub” and “Spoke” model to promote knowledge exchange between health care specialists typically located at academic centers (“the hub”) and primary care providers (PCPs) at the front line of community health care (the “spokes”). Through regularly scheduled virtual educational clinics, Project ECHO creates a supportive community network where PCPs can connect with specialists and with other PCPs practicing in similar settings via multipoint video technology to discuss best practices in care and complex cases managed within their practice. This multifaceted knowledge and capacity-building intervention includes two fundamental components: a didactic lecture delivered by a member of the hub team (based on curriculum developed from guidelines, best practices, and/or a needs assessment); and recommendations for case management (telementoring) offered by the community in response to anonymized clinical cases presented by spoke sites.
When implementing a complex intervention such as Project ECHO, it is important to understand organizational readiness relating to factors involved in the intervention. Significant evidence in implementation science suggests that assessing organizational readiness before implementing any major intervention within an organization is crucial to minimize risk that the intervention will not be accepted or fail, reducing wasted time and financial investment.2–5 Identifying whether organizational readiness exists is an important practice that will allow an organization to understand whether they are best equipped to implement a new project or to understand whether and why it is likely to be rejected.2
This article will outline how ECHO Ontario Mental Health (ECHO-ONMH) adapted and integrated Damschroder's (2009) Consolidated Framework for Implementation Research framework to create a readiness tool for the implementation of ECHO and a set of implementation guidelines. While determining what knowledge translation and capacity-building model would best support primary care providers looking to build capacity to treat clients with mental health and addiction concerns, our team was introduced to the ECHO model, which had been replicated globally with positive outcomes. Our team understood that a significant investment of time, money, and effort was required to set up an ECHO project, so we endeavored to assess whether we had sufficient organizational and leadership traits to succeed in the implementation of ECHO. We received advice from peers who had launched a similar project and sought replication support from the ECHO Institute in New Mexico; however, we believe that we would have benefited from a tool to assess organizational readiness for an ECHO project. To address this resource gap, we turned to implementation science, more specifically, to validated frameworks such as the CFIR6 and proposed taxonomy of implementation outcomes by Proctor et al.7,8
In the seminal Project ECHO study by Arora et al, rural primary care physicians supported by Project ECHO were able to achieve comparable treatment outcomes for hepatitis C compared with outcomes in a specialist-run hepatitis C clinic located in an academic center (nonassociated with ECHO), demonstrating ECHO's ability to support nonspecialists to use best practices and treatment algorithms to improve health outcomes.1 In addition, our recent systematic review of the evidence for Project ECHO assessed a total of 39 studies, which addressed 17 different medical conditions and found that these studies support the ECHO model's effectiveness in improving the knowledge (n = 4) and self-efficacy of participating PCPs (n = 8), as well as changes in provider behavior (n = 1), improved patient outcomes (n = 7), and cost-effectiveness (n = 2) across a number of disease conditions and medical specialties.9 In addition, in a recent assessment of our ECHO Ontario Mental Health project outcomes, our team found that there was a significant improvement in pre-post knowledge of over 12% for participating primary care providers (P < .001, d = 1.13).10
The ECHO model uses both situational and social cognitive learning theories and enables participating PCPs to identify learning gaps (cognitive dissonance) and reflect critically on their learning process.11–14 Project ECHO allows for problem-centered learning to occur in the clinical context where new knowledge will be applied; promotes interprofessional collaboration among participants; models best-practice care; supports learners to feel that they are benefiting and improving self-efficacy; and allows participants to receive positive feedback and reinforcement from clinical opinion leaders.13–15
The Project ECHO model has rapidly expanded in recent years, with over 140 international ECHO projects covering chronic diseases such as hepatitis C, HIV/AIDS prevention, addictions and psychiatry, chronic pain management,16 asthma,17 and dementia18 among others. In addition, large ECHO projects have been implemented across the United States of America through the Veterans Health Administration that spans numerous conditions and states.19,20 Enthusiasm for and expansion of the ECHO model has been further supported by the introduction and approval of the ECHO Act in 2016 in the USA, which will allow for further evaluation of this model as a capacity-building tool.21 According to the ECHO Institute, 13 of 142 ECHO projects are no longer active (about 9%). Although this number is currently low, the costs associated with implementing an intervention such as ECHO make it important to reduce the potential of funding an ECHO that is not equipped to succeed. Although the data assessing sustainability of the ECHO model are still new and relatively unknown, it is a critical metric to be able to understand the long-term impact of this educational intervention.
As this model expands, it is increasingly important to ensure that emerging programs maintain fidelity to the model to allow across-program evaluation and to ensure a strong return on investment. The aim of the model is to offer flexibility while ensuring that new projects are in alignment with the following key principles:
- The project must use technology to leverage scarce resources.
- The project must use case-based learning to master complexity-learning loops.
- The project shares best practices to improve knowledge (to increase desired outcomes).
- The project uses a web-based database (iECHO) to monitor outcomes.
ECHO projects are required to sign an agreement with the ECHO Institute in New Mexico before obtaining training, assuring that they will implement their project following the four aforementioned key principles. Currently, according to the ECHO Institute, there are a total of 223 potential ECHO sites that have either signed contracts but not joined a replication training at the ECHO Institute in New Mexico (N = 83) or trained but have not launched yet (N = 140). For the latter group, the average time in this prelaunch state is 230.3 days. Our hope is that these tools can help sites implement quickly and effectively.
Damschroder et al6 created the meta-theoretic CFIR framework, which has provided implementation scientists with a standard set of constructs such as intervention characteristics, internal and external settings, characteristics of individuals, and processes. We selected this framework because it is a comprehensive set of constructs that integrates 19 different well-known implementation models, including Promoting Action on Research Implementation in Health Services (the PARiHS Framework), Technology Implementation Process Model, Replicating Effective Programs framework, and a Practical, Robust Implementation and Sustainability Model (PRISM) among others.6 The CFIR framework has been shown to be effective for guiding successful implementation across numerous health domains, such as weight management,22 health records systems,23 internet patient-provider communication,24 and cancer screening.25 Although the CFIR list of constructs is comprehensive, the framework does not specify interactions between the constructs6; some constructs may be linked and need to be considered concurrently. Likewise, the implementation outcome measures put forth by Proctor et al7,8 especially acceptability, adoption, appropriateness, cost, feasibility, penetration, sustainability, and fidelity (which are further defined in Table 1 below) are also regarded as integral to ensuring implementation success.7,8 Similarly, we selected this framework, as it provided us with a comprehensive list of implementation outcomes by aggregating common taxonomy proposed through important implementation outcome evaluation frameworks (eg, Roger's Theory of Diffusion of Innovation26 and the RE-AIM Framework).8,27 Our goal for using these two frameworks was to ensure that we have considered and addressed all important factors, based on well-validated constructs, which are essential to preparing new ECHOs to assess their organizational readiness and guide an effective implementation.
Using theoretical concepts from the CFIR6 and implementation outcomes by Proctor et al7,8 combined with practical experience from the launch of ECHO-ONMH, this study aimed to: (1) build on already validated implementation tools, such as the CFIR, by adapting and consolidating frameworks to create a set of assessment questions to help organizations assess their readiness and capacity to support an ECHO project; (2) provide those who have determined ECHO is the correct model for their organization with a checklist to support a successful implementation process that maintains fidelity to the ECHO model based on our experience and implementation science. These tools will support an approach to implementing the ECHO model globally in a standardized manner and will offer further opportunity for validation of the questions (and CFIR constructs) and harmonized implementation outcomes in the future.
Checklist development was divided into two phases: phase one that focused on the development of key considerations before adopting the ECHO model and phase two that focused on practical, high-level processes to guide implementation.
Preimplementation Organizational Readiness Questions
During phase one, a comprehensive list of Damschroder's CFIR Constructs (intervention characteristics, internal and external settings, characteristics of individuals, and processes) and their associated descriptions were reviewed and discussed by the ECHO-ONMH team, each with intimate familiarity with the ECHO implementation process, including the initial project conceptualization and proposal process. ECHO-related tasks were then mapped onto each CFIR construct and subconstructs, creating a third column of ECHO-specific implementation constructs, linking the ECHO practical steps with CFIR theoretical considerations. Subsequently, questions corresponding to each of the ECHO-centric implementation construct were developed that could be used to assess whether important implementation factors and steps were feasible within the organization and externally. These questions were geared toward individuals/organizations who were thinking about adopting the ECHO model. Questions were organized into a table with three columns: (1) the Damschroder construct and corresponding definition; (2) the ECHO adaptation of the construct; and (3) a key question related to ECHO that addressed the initial Damschroder implementation construct. These key questions, based on each construct and ECHO-ONMH implementation experience, were discussed by a team of four to confirm that the developed question effectively addressed key factors within the construct. The questions were reviewed for clarity by all members of the implementation team, and a subset of the questions was piloted for clarity and applicability by potential end-users at workshops for the Council of Psychiatric Continuing Education (ie, directors of continuing education in Psychiatry from Canadian universities) and the American Psychiatric Association. Finally, to link all considerations to support the eventual evaluation of ECHO project implementation, questions were organized according to four of Proctor's implementation outcomes (appropriateness, acceptability, feasibility, and cost). Only four of the eight implementation outcomes proposed by Proctor et al7,8 were used in this analysis because the remaining outcomes were better linked with postimplementation assessment (adoption, penetration, sustainability, and fidelity). At the conclusion of this process, overlapping questions were discarded (Fig. 1). Figure 1 outlines how we have linked Proctor's implementation outcomes to the CFIR constructs, with sample questions in each domain.
Phase two sought to develop a detailed implementation process (the fifth CFIR domain), which included tasks and procedures to implement an ECHO. ECHO-ONMH project plans, which were developed by the ECHO manager and team, were extensively reviewed with all tasks listed. The ECHO Institute's repository for implementation and project plan tools was also reviewed for any additional implementation steps that were not included in ECHO-ONMH's plan. The resulting set of implementation tasks and procedures were linked with the four subcategories of the CFIR process construct, especially: (1) planning, (2) engaging, (3) executing, and (4) reflecting and evaluating. Practical implementation steps were discussed by the implementation team and categorized. A list was developed that proposed high-level implementation steps to support the implementation of the ECHO project based on practical experience linked with theoretical implementation approaches (Fig. 2).
Table 1 outlines a list of 20 key questions for consideration by individuals and/or organizations interested in developing an ECHO program. The 20 questions reflect a robust set of considerations meant to help guide the decision to adopt and plan an ECHO project. These questions are not meant to be sequential; rather, each section can occur concurrently. As can be seen in the table, most questions developed fall within the “Appropriateness” and “Acceptability” categories, this reflects the focus on evaluating whether the ECHO model is the best tool to meet end-user need before commencing implementation. This tool is not meant to provide a ranking for the level of preparedness or likelihood to succeed; rather, the goal of these questions is to provide additional clarity around some of the complexities of implementing Project ECHO, which can equip new projects with a broad understanding of both internal and external factors that might affect the successful implementation of ECHO.
Table 2 provides a checklist for implementation processes and steps required to implement an ECHO. Some of the articulated procedural steps overlap two sections but after discussion and consensus from the ECHO implementation team were placed in one section to reduce duplication. Although a customized, detailed project plan with a timeline is important for all newly developing ECHOs, these are the key aspects necessary to launch an ECHO project and to support an ongoing quality-improvement process. The four sections have steps that will occur concurrently. The steps include the following: (1) Planning, for example, identifying end-user needs, hiring and training hub and operations team, securing funding, and developing a project and communication plan. (2) Engaging internal and external stakeholders (through meetings or focus groups) and identifying champions for the model. (3) Executing to implement ECHO through four key areas, including operations; curriculum development; end-user recruitment/registration, and communications. (4) Reflecting and evaluating by ensuring that the program has a clear evaluation framework that enables a robust assessment of how well the intervention meets its stated aims, such as improving knowledge, or achieving health outcomes. Certain continuing medical education evaluation models, for example Moore's Framework for Continuing Medical Education28, are effective frameworks to assess not only how ECHO supports participant change in knowledge and self-efficacy but also how this knowledge change translates into performance-, patient-, and population-level improvements. In addition, using the Implementation Outcome framework by Proctor et al7,8 , key implementation outcomes should also be assessed to understand whether a project was successfully implemented as expected.
As suggested by Proctor et al,7,8 it is important to implement an intervention effectively and complete intermediate assessments of implementation to make assertions about the effect of the intervention on provider learning and patient health outcomes. The growing evidence for the ECHO model, in combination with the recent introduction of the ECHO Act,21 has led to a significant growth in ECHO projects globally and lends increasing importance to the understanding of key implementation success and organizational readiness factors. As may be expected, given the implementation of the ECHO model across various disciplines, each of the preimplementation considerations put forth in the table above may not be applicable for certain individuals/organizations; however, it provides an overview of important factors for an ECHO project. Of additional note, many of the questions and processes highlighted in the tables may occur simultaneously and as articulated by Damschroder et al6 does not describe the relationship between each construct and/or outcome. Although this set of questions does not generate a score about readiness to start an ECHO, it can be used to facilitate a discussion and assess gaps that should be deliberated when considering this project model. Although it is not yet clear whether there is a threshold that would identify optimal readiness to implement based on these questions, new projects should aim to address as many of these questions as possible during implementation.
A limitation is that these questions have been adapted and tested within the context of one ECHO project. The next steps for our team are to test these questions with approximately 10 ECHO projects in varying countries among different disease areas and complete a full validation through factor analysis to confirm construct validity. Although the tools developed in this framework will provide an important first step to supporting organizational readiness assessments for ECHO, some analysis that correlates which of these constructs are consistently present in successful ECHOs and which are lacking in ECHOs that do not successfully implement might help build a more predictive model for implementation. This could help determine which constructs are key drivers of implementation success and which are just beneficial. In addition, identifying benchmarks to support optimal implementation (ie, a certain threshold of participants, fidelity to the project, etc) would help projects determine whether implementation has been successful.
These tools, developed based on implementation science theories, are a preliminary step to supporting an evidence-informed approach to the implementation of the ECHO model globally. They will offer further opportunities for implementation research and validation of constructs and harmonized implementation outcomes in the field. Effective implementation of ECHO, an educational model that has key facets that require high fidelity, is crucial to attribute whether observed outcomes (positive or negative) are an effect of the intervention as delivered with high fidelity. In addition, these tools have utility beyond the ECHO model to help organizations determine readiness for new continuing medical education models delivered within primary care. The ECHO model is increasing in reach and scale, with growing evidence for its potential to improve knowledge and capacity in primary care. Although the ECHO Institute in New Mexico provides replication support to newly developing ECHO projects, these tools are complimentary and can support the rapid expansion of Project ECHO throughout the world. This set of foundational questions, rooted in validated implementation science frameworks, can help ensure organizational fit and readiness to start an ECHO project and reduce wasted cost to funders by avoiding financing projects that are not prepared to implement an ECHO successfully.
Lessons for Practice
Lessons learned include the importance of assuring organizational readiness to implement a continuing medical education model, such as Project ECHO, to ensure fidelity. Effective implementation is paramount to confirm that practice change and health outcomes can appropriately be attributed to the educational intervention model. Using evidence-based theoretical frameworks such as the CFIR to standardize an approach to assessing organizational readiness and implementation of the ECHO model will help improve the ability to assess this intervention globally. These lessons can also support implementation of other similar interventions in the rapidly growing field of virtual continuing medical education and capacity building.
The authors thank and acknowledge the ECHO Institute team at the University of New Mexico and the ECHO Ontario Chronic Pain team for their tremendous support and contributions throughout the implementation of ECHO Ontario Mental Health at CAMH and the University of Toronto. The views expressed in the publication are the views of the authors and do not necessarily reflect those of the Ontario Ministry of Health and Long-Term Care. This work is affiliated with the Medical Psychiatry Alliance, a collaborative health partnership of the University of Toronto, the Centre for Addiction and Mental Health, the Hospital for Sick Children, Trillium Health Partners, the Ontario Ministry of Health and Long-Term Care, and an anonymous donor.
1. Arora S, Thornton K, Murata G, et al. Outcomes of treatment for hepatitis C virus infection by primary care
providers. New Engl J Med. 2011;364:2199–2207.
2. Jennett P, Maryann Y, Monica P, et al. Organizational readiness
: implications for success and failure. J Telemed Telecare. 2003;9(suppl 2):S27–S30.
3. Shea CM, Jacobs SR, Esserman DA, et al. Organizational readiness
for implementing change: a psychometric assessment of a new measure. Implement Sci. 2014;9:7.
4. Weiner BJ, Lewis MA, Linnan LA. Using organization theory to understand the determinants of effective implementation of worksite health promotion programs. Health Educ Res. 2009;24:292–305.
5. Weiner BJ. A theory of organizational readiness
for change. Implement Sci. 2009;4:67.
6. Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.
7. Proctor E, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36:24–34.
8. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76.
9. Zhou C, Crawford A, Serhal E, et al. The impact of project ECHO
on participant and patient outcomes: a systematic review. Acad Med. 2016;91:1439–1461.
10. Sockalingam S, Arena A, Serhal E, et al. Building provincial mental health capacity in primary care
: an evaluation of a project ECHO
mental health program. Acad Psychiatry. 2017;7:1–7.
11. Kaufman DM. ABC of learning and teaching in medicine: applying educational theory in practice. BMJ. 2003;326:213.
12. Kolb AY, Kolb DA. Learning styles and learning spaces: enhancing experiential learning in higher education. Acad Manag Learn Educ. 2005;4:193–212.
13. Arora S, Kalishman S, Thornton K, et al. Expanding access to hepatitis C virus treatment-extension for community healthcare outcomes (ECHO) project: disruptive innovation in specialty care. Hepatology. 2010;52:1124–1133.
14. Socolovsky C, Masi C, Hamlish T, et al. Evaluating the role of key learning theories in ECHO: a telehealth educational program for primary care
providers. Prog Community Health Partnersh. 2013;7:357–358.
15. Flodgren G, Parmelli E, Doumit G, Gattellari M, O'Brien MA, Grimshaw J, Eccles MP. Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2007;1.
16. Scott JD, Unruh KT, Catlin MC, et al. Project ECHO
: a model for complex, chronic care in the Pacific Northwest region of the United States. J Telemed Telecare. 2012;18:481–484.
17. Harkins M, Raissy H, Moseley K, et al. Project ECHO
: improving asthma care in New Mexico with telehealth technology. Chest J. 2011;140:861A.
18. Knoefel J, Herman C. Dementia care training for primary care
providers: project ECHO
™(P6. 182). Neurology. 2015;84(14 suppl):P6–P182.
19. Frank JW, Carey EP, Fagan KM, et al. Evaluation of a telementoring intervention for pain management in the Veterans Health Administration. Pain Med. 2015;16:1090–1100.
20. Knapp H, Pangarkar S. Utilizing the ECHO model in the veterans health affairs system: guidelines for setup, operations and preliminary findings. Future Internet. 2015;7:184–195.
22. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research
(CFIR). Implement Sci. 2013;8:51.
23. Richardson JE, Abramson EL, Pfoh ER, et al; HITEC Investigators. Bridging informatics and implementation science: evaluating a framework to assess electronic health record implementations in community settings. AMIA Annu Symp Proc. 2012;2012;770. American Medical Informatics Association.
24. Varsi C, Ekstedt M, Gammon D, et al. Using the consolidated framework for implementation research
to identify barriers and facilitators for the implementation of an internet-based patient-provider communication service in five settings: a qualitative study. J Med Internet Res. 2015;17:e262.
25. Liang S, Kegler MC, Cotter M, et al. Integrating evidence-based practices for increasing cancer screenings in safety net health systems: a multiple case study using the consolidated framework for implementation research
. Implement Sci. 2016;11:109.
26. Rogers EM. Diffusion of Innovations. 4th ed. New York, NY: The Free Press; 1995.
27. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–1327.
28. Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29:1–15.