Applying the Project ECHO Model to Support Implementation and Sustainment of Cognitive Behavioral Therapy for Psychosis

Introduction: Project Extension for Community Healthcare Outcomes (ECHO) is a teleconsultation model that leverages technology to sustain specialized interventions in underresourced settings. We present the application of the ECHO model to longitudinal training and consultation for community behavioral health providers learning to deliver cognitive behavioral therapy for psychosis, an evidence-based psychotherapy for individuals with psychotic disorders that has poorly penetrated the US mental health system. Methods: We analyzed within-group change over practitioners' 6-month ECHO participation cycle using the Expanded Outcomes Framework. We evaluated outcomes associated with participation, satisfaction, knowledge acquisition, performance, patient symptom severity, and functional impairment. Results: In the first 3 years, the cognitive behavioral therapy for psychosis ECHO Clinics supported 150 providers from 12 community agencies. Forty percent did not complete the 6-month ECHO calendar, most commonly due to separation from their agency. Participants reported high degrees of satisfaction. Declarative and procedural knowledge increased over the 6-month period. Of the 24 providers who received a fidelity review, 87.5% met or exceeded the competency benchmark within the 6-month period. Clinical outcomes reflected reductions in hallucinations, negative symptoms, depression, mania, and functional impairment, but no reductions were detected in delusions, disorganized speech, or abnormal psychomotor behavior. Discussion: ECHO Clinics offer a mode of providing continuous access to expert instruction, peer-to-peer consultation, and case-based learning that other workforce training models lack. Our evaluation suggests that the ECHO model supports continuous professional development for practitioners, most of whom had indicated inadequate preparation for their role. We observed improved learner and select patient outcomes.

always a participant" and "All teach, all learn," the CBTp ECHO Clinics were adopted as a means of fostering a professional community, mitigating therapeutic drift through ongoing professional development, enhancing CBTp funds of knowledge and competencies, and improving outcomes of CBTp consumers.Despite a robust body of literature demonstrating satisfaction, engagement, enhanced knowledge, and therapeutic benefit of ECHO participation, 9,10 we were interested in ascertaining whether such benefits extend to practitioners learning a psychological treatment for psychotic disorders.Consistent with previous evaluations of ECHO clinics for other complex treatments and conditions, 11,12 we employed the Expanded Outcomes Framework, a wellestablished framework for evaluating professional development activities developed by Moore et al (Figure 1). 13

METHODS CBT ECHO Clinic Description
The CBT ECHO Clinic protocol was based on the ECHO replication model 14 and previous empirical literature proposing effective elements of longitudinal consultation for CBT interventions, including recommended minimum dose 15 and opportunities for didactic and case-based learning as well as behavioral rehearsal. 16,17The didactic schedule was designed to build on concepts introduced during the workshop training and to align with core elements of CBTp previously identified through expert consensus. 18The CBTp ECHO Clinics adhere to the ECHO huband-spoke model.The hub team is composed of a clinic coordinator, IT support, and a panel of CBTp trainers with a cumulative 25 years of experience administering the intervention.The CBTp ECHO Clinics are open only to behavioral health providers who have attended a synchronous multiday intensive workshop, during which they receive didactic instruction, experiential activities, and role plays demonstrating the application of CBT to psychotic and mood symptoms.Following participation in the live workshop, participants are expected to attend a minimum of nine of the first 12 ECHO Clinics offered within the first 6 months post workshop (75% attendance requirement).The first 12 ECHO sessions represent the core CBTp ECHO curriculum; subsequent didactics are intended to supplement the core curriculum (Table 1).Thereafter, trainees have an open invitation to attend ECHO Clinics at their own discretion.Contact hours are offered in accordance with state and professional organizational guidelines for teleconsultation, but no other incentives are offered.

CBT ECHO Clinic Protocol
The traditional ECHO session protocol includes introductions of the spokes (prioritizing video callers to reinforce communitybuilding behaviors), introductions of the hub panelists, announcements from spokes and/or hub, a live didactic or role play, and a case consultation.In accordance with the "all teach all learn" credo, questions and recommendations are first elicited from the spoke participants and then are offered by the hub panelists.The clinic coordinator notes clinical recommendations for later dissemination to clinic participants along with resources that were recommended during the clinic.Clinics range from 60 to 90 minutes depending on the complexity of the case presentation or CBTp protocol.

Clinic Evaluation
The methods and data that were used to evaluate the CBTp ECHO Clinics in accordance with Moore's evaluation framework are described below.This work was reviewed by the Washington State Institutional Review Board and granted exempt determination.

Level 1-Participation
Programmatic data were collected February 2017 and August 2021 using iECHO, a web-based data management system operated by the University of New Mexico SuperHub.Data include administrative data, participant attendance, session components, program documents, and awarded CME/CEU/CE credits.

Level 2-Satisfaction
We randomly administered a 3-item end-of-session poll using Poll Everywhere to 50% of all clinics to evaluate satisfaction, confidence, and perceived impact on clinical practice.Participants rated their satisfied with the clinics and their degree of benefit on a scale of 1 (Not at all) to 5 (Very great extent) and provided a dichotomous (yes/no) response to "Will this consult affect your work with one or more CBTp clients?"The impact on practice item was also included at 3-and 6-month time points through REDCap (Research Electronic Data Capture 19,20 ).
Level 3A-Learning: Declarative Knowledge To assess fund of knowledge of CBTp, 1 multiple-choice "Test Your Knowledge" question was administered following each of the initial 12 core didactics.Scores were coded dichotomously (pass/fail).

Level 3B-Procedural Knowledge
Participants were asked to rate their CBTp skill level across 20 CBTp techniques on a 5-point Likert scale (1-minimal; 5advanced).Example items include "Teaching clients how to reduce distress by evaluating inaccurate or unhelpful thoughts" and "Working collaboratively with clients to develop a list of concerns that cause distress and/or interfere with life goals."Ratings were totaled for a score range of 20 to 100, with higher scores indicating increased skill in providing CBTp.This selfassessment was administered at preworkshop, postworkshop, and 6-month follow-up.

Levels 4 and 5-Competence and Performance
In accordance with national recommendations for establishing CBTp competencies, 21,22 participants were asked to submit session audio to enable the hub team to assess the degree to which CBTp was being administered to clients with fidelity to the treatment model.Sessions were rated with the Revised Cognitive Therapy for Psychosis Adherence Scale (R-CTPAS), 23 a psychometrically validated fidelity scale designed to assess adherence and competency across therapeutic activities considered to be core components of CBTp.The scale includes items related to general features of the session (eg, agenda setting, home practice review, and home practice assignment) as well as specific features of a session that correspond to phases of treatment (eg, psychoeducation and engagement, assessment, intervention, skill generalization).Participants were given an overall rating on a scale of 1 to 5; ratings of three or higher are considered to meet competence in CBTp.

Level 6-Patient Health
To track patient outcomes, CBT ECHO participants administered the DSM-5 Clinician Rated Dimensions of Psychosis Symptom Severity Scale (CRDPSS) 24 and the Social and Occupational Functioning Assessment Scale (SOFAS) to their clients. 25Both measures are recommended by the DSM-5 task force and as ways of monitoring progress in the CBTp Stepped Care model. 26The CRDPSS is a dimensional (0-not present, to 4-present and severe) scale designed to assess 8 symptom domains associated with psychotic disorders, such as hallucinations, delusions, disorganized speech, abnormal psychomotor behavior, negative symptoms, depression, mania, and cognitive functioning.We omitted the cognitive functioning item as trainees were not credentialed to obtain valid and reliable measures of cognitive functioning.The SOFAS is a clinician-rated measure of functioning scored on a scale of 1 (grossly impaired) to 100 (excellent functioning), with benchmarks provided at each multiple of 10.Deidentified patient assessment data were entered into a data dashboard that was mutually accessible to the ECHO participant and the research team.Complete data were defined as 2 or more administrations of a measure, including 1 at baseline and at least 1 at follow-up.
For those who completed a given measure at multiple time points, the final time point was defined as "follow-up" and used in analysis.

RESULTS
During the observation period, 187 behavioral health providers participated in a CBTp workshop.Thirty-seven participants (19.8%) were ineligible to participate in a CBTp ECHO Clinic by virtue of their role as an agency administrator or manager.One hundred fifty eligible clinicians attended at least 1 ECHO clinic (80.2%; see Figure 2).ECHO participants were employed by 12 agencies across 10 counties in the state.Demographics are reported in Table 2. Most (56.2%) reported inadequate or no training in CBT in their programs of study.
The training hub hosted 217 CBTp ECHO Clinics between February 2017 and October 2021, facilitating 60 unique didactics and 146 case presentations.Clinic attendance ranged from 2 to 35 trainees (M = 12.9, SD = 4.6).Inclusive of providers who dropped out of the training program prematurely or who paused their participation, trainees attended between 1 and 70 clinics (M = 12.5, SD = 11.1),generating a cumulative 3428 continuing education hours.Sixty percent of trainees (n = 90) met the initial 6-month attendance requirement.Of those who did not meet attendance requirements, the most common reason was separation from their agency.

Level 2-Satisfaction
Trainees reported high degrees of satisfaction immediately following clinics, with a mean rating of 4.2 out of 5 (SD = 0.4).Approximately 550 responses were collected to the end-ofsession forced choice question "Will this consult affect your work with one or more CBTp clients?" with 97% of responses endorsing this statement.This postclinic question was administered after nearly half of all ECHO clinics (n = 95), with a response rate of approximately 45%.Trainees reported having benefitted from ECHO clinics at both 3-and 6-month points, with a significant increase in self-report ratings from 3-month follow-up (M = 3.52, SD = 1.00) to 6-month follow-up (M = 3.74, SD = 1.04); t(57) = 22.09, P = .041.

Level 3A-Declarative Knowledge
We collected 58 responses to the postclinic "Test Your Knowledge" questions following the 12 core didactics (37% response rate).The correct response rate was 73.2% (range = 33.3% to 100%).Trainees scored the lowest on an initial didactic on the topic of case conceptualization and scored the highest on later didactics related to working with clients with cognitive impairments and wellness planning.The temporal trend toward higher pass rates suggests that trainees' understanding of CBTp principles and techniques improved over the course of the ECHO series.

Level 3B-Procedural Knowledge
To assess whether ratings changed over time, we conducted a pairwise comparison.Mauchly test of sphericity indicated a violation of the assumption of sphericity (X 2 (2) = 7.886, P = .019).We therefore applied Greenhouse and Geisser (1958) correction.We detected a significant main effect of time (F(1.57,40.93) = 29.89,P < .001;h p 2 = .54),such that participants of both clinics perceived their skills to have improved significantly from pretraining to posttraining and remained significantly   Complete CRDPSS and SOFAS data were available for 24 and 16 clients, respectively (Table 3).Severity of clients' hallucinations, negative symptoms, depression, and mania were reduced from baseline to follow-up.Disorganized speech and abnormal psychomotor behaviors were scored as minimally severe at both baseline and posttreatment, with no significant changes observed between time points.We detected a statistically significant improvement in functioning on the SOFAS.

DISCUSSION
ECHO Clinics offer a mode of providing continuous access to expert instruction, peer-to-peer consultation, and case-based learning that other workforce training models lack.As ECHO programs proliferate internationally and are recommended by governmental bodies, 27 program evaluation and research on ECHO programs are needed to establish best practices for diverse ECHO participants and focal problems.We describe trainee outcomes for the first teleconsultation clinic that applied the ECHO model to a cognitive behavioral treatment and to psychotic spectrum disorder treatment.We report here on the inaugural CBT ECHO Clinic model and participant and patient outcomes using the Expanded Outcomes Framework to facilitate replication and refinement of this model of professional development.Principal advantages of the ECHO model are the ability to reach mental health care practitioners who do not readily have access to professional development in evidence-based interventions and sustain their access over time.We were able to offer longitudinal support, conferring a total of 3428 continuing education hours, to more than 150 behavioral health providers across 12 community behavioral health agencies, most of whom had indicated inadequate preparation for their role.This is notable for 2 reasons.First, it affirms the importance of workforce development for practitioners working with individuals with psychosis in community settings.Second, it provides evidence that our clinics were successful in targeting those practitioners who may be most in need of professional development.
We used several data collection methods throughout the consultation period to facilitate program evaluation, including active monitoring of attendance, live participant polling, and routine survey administration.Multimodal data collection facilitated quality improvement efforts.Specifically, we were able to tailor and develop new didactics in response to trainee demand.9][30][31] That said, only 60% of trainees met the 75% attendance requirement.Although this posed a challenge to data collection, it was not unexpected.Trainees were employed in high-volume, publicly funded settings, which commonly experience high caseloads and staff turnover. 32mong those trainees who met the minimum attendance requirement, we observed significant gains in measures of procedural CBTp knowledge.Although clinical outcome data were limited, we found that many of the clinical targets for which CBTp is indicated-namely, hallucinations, mood symptoms, and negative symptoms 33 -did improve over the course of the 6-month training program.Conversely, clinical data did not support statistically significant improvements in delusions, for which CBTp is typically effective. 34mitations and Future Directions Many of the challenges we observed among our sample are common to EBT implementation efforts and are well-addressed in recently published CBTp implementation guides. 22,35As the data reported here represent a program evaluation, the absence of a control group limits our capacity to draw inferences related to how our trainee and client outcomes compare with other consultation approaches.That said, the observed improvements among most focal psychotic and mood symptoms suggests that clients were benefitting from the services they were receiving.Consistent with the previous literature evaluating EBT implementation in community health settings, 36 many clinicians in our sample faced significant hardships related to client recruitment and retention, audio recording sessions, and protected time for both continuing education activities and delivering psychotherapy.As a result, missing data are a notable limitation of the current evaluation.We encountered attrition among our sample of behavioral health care practitioners in the first 6 months post workshop.Although our attrition rate is consistent with national data on workforce turnover in community behavioral health settings, 37 we cannot discount the threat of attrition bias to the validity of our findings.
Our evaluation demonstrates that the ECHO model offers a viable model of longitudinal support to practitioners newly learning an evidence-based psychological treatment for Serious Mental Illness.However, sustaining EBTs in routine care settings requires more than access to continuing professional development and clinical supports.Both internal factors external factors are implicated in the long-term sustainment of EBTs. 38We find support for use of Project ECHO as one component of a CBTp sustainment plan.We find particular value in the fact that ECHO Clinics can enable provider-toprovider and expert-to-provider learning across health care settings.We recommend a randomized clinical trial to systematically evaluate trainee and client outcomes associated with ECHO-facilitated longitudinal consultation compared with a consultation as usual approach.n Constraints inherent to community mental health work (ie, high staff turnover rates; large, diagnostically heterogeneous caseloads; and limited technological resources) also affect the degree to which trainees can fully participate in longitudinal professional development activities.
n Further research is needed to investigate the effects of ECHO versus traditional consultation methods on provider training process, retention, trainee outcomes, and clinical outcomes for clients served.

FIGURE 1 .
FIGURE 1. Expanded outcomes framework.Details the seven levels of evaluation of clinical professional development in the Expanded Outcomes Framework created by Moore and colleagues.13

FIGURE 2 .
FIGURE 2. Consort diagram of CBTp Consort diagram indicating trainees' reasons for discontinuing participation and the number of trainees who discontinued for each identified reason.CBTp, cognitive behavioral therapy for psychosis

Level 6 -
Clinical Outcomes Trainees were expected to administer CBTp to at least 1 client on their caseload during the 6-month training period.Of those trainees who completed both the 3-month and 6-month posttraining survey (n = 61), 53 trainees (86.9%) had initiated CBTp with at least 1 client.Number of clients treated by the 6month follow-up ranged from 1 to 50 (M = 5.21, SD = 8.48).

n
The teleconsultation model of Project ECHO is a viable means of delivering relevant and satisfactory statewide professional development in psychotherapy to community behavioral health practitioners.
*CRDPSS is the Clinician Rated Dimensions of Psychosis Symptom Severity Scale.Lower scores reflect less symptomatology.†SOFAS the Social and Occupational Functional Assessment Scale.Higher scores reflect higher functioning.CI, confidence interval.*P < .05;**P < .001.higher