Secondary Logo

Journal Logo

Implementation of Off-Site Integrated Care for Children

A Scoping Review

Spencer, Andrea E. MD*; Platt, Rheanna E. MD, MPH*; Bettencourt, Amie F. PhD; Serhal, Eva MBA; Burkey, Matthew D. MD, PhD, MPH; Sikov, Jennifer BA; Vidal, Carol MD, MPH; Stratton, Julia MD; Polk, Sarah MD, ScM; Jain, Sonal DO; Wissow, Lawrence MD, MPH

doi: 10.1097/HRP.0000000000000239
Reviews
Free
SDC

Background As an alternative to co-located integrated care, off-site integration (partnerships between primary care and non-embedded specialty mental health providers) can address the growing need for pediatric mental health services. Our goal is to review the existing literature on implementing off-site pediatric integrated care.

Methods We systematically searched the literature for peer-reviewed publications on off-site pediatric integrated care interventions. We included studies that involved systematic data collection and analysis, both qualitative and quantitative, of implementation outcomes (acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability).

Results We found 39 original articles from 24 off-site programs with a variety of study designs, most with secondary implementation outcomes. Models of off-site integration varied primarily along two dimensions: direct vs. indirect, and in-person vs. remote. Overall, off-site models were acceptable to providers, particularly when the following were present: strong interdisciplinary communication, timely availability and reliability of services, additional support beyond one-time consultation, and standardized care algorithms. Adoption and penetration were facilitated by enhanced program visibility, including on-site champions. Certain clinical populations (e.g., school-age, less complicated ADHD) seemed more amenable to off-site integrated models than others (e.g., preschool-age, conduct disorders). Lack of funding and inadequate reimbursement limited sustainability in all models.

Conclusions Off-site interventions are feasible, acceptable, and often adopted widely with adequate planning, administrative support, and interprofessional communication. Studies that focus primarily on implementation and that consider the perspectives of specialty providers and patients are needed.

From the Department of Psychiatry, Boston University School of Medicine, Boston Medical Center, Boston, MA (Dr. Spencer and Ms. Sikov); Department of Psychiatry and Behavioral Sciences (Drs. Platt, Bettencourt, and Vidal) and Pediatrics (Dr. Polk), Johns Hopkins University School of Medicine, Baltimore, MD; Centre for Addiction and Mental Health, Toronto, Ontario (Ms. Serhal); Department of Psychiatry, University of British Columbia (Dr. Burkey); Pacific Spirit and Raven Song Community Health Centres, Vancouver Coastal Health Authority, Vancouver, British Columbia (Dr. Stratton); New York Institute of Technology College of Osteopathic Medicine (Dr. Jain); Department of Psychiatry and Behavioral Sciences, University of Washington (Dr. Wissow).

* Drs. Platt and Spencer contributed equally and have agreed to share first authorship.

Original manuscript received 14 June 2019, accepted for publication subject to revision 26 August 2019; revised manuscript received 13 September 2019.

Correspondence: Andrea Spencer, MD, Department of Psychiatry, Boston Medical Center, Doctor’s Office Building, 720 Harrison Ave., Boston, MA 02118. Email: Andrea.Spencer@bmc.org

Supplemental digital contents are available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (www.harvardreviewofpsychiatry.org).

Back to Top | Article Outline

INTRODUCTION

Mental health problems are among the most common causes of morbidity and mortality in youth. Up to an estimated 20% of children have psychiatric disorders.1,2 The gap between the need for pediatric mental health services and their availability is nevertheless large and growing.3 Service costs, insurance restrictions, logistical barriers, stigma associated with seeking services, and the shortage and unequal distribution of qualified mental health providers (MHPs) all limit the availability of psychiatric care for children.4–6 Consequently, there is an urgent need both to expand the capacity of the existing workforce and to support the provision of mental health services by other types of providers (e.g., primary care providers).

The integration of mental health services with primary care has received increasing attention in recent years as a promising strategy for decreasing barriers to care and serving a larger population of patients.7 The primary care setting is often convenient for families, and offers a less stigmatizing setting in which to discuss sensitive concerns.8 Primary care providers (PCPs) often have limited training, however, in assessing and treating mental health problems, as well as little time and low reimbursements for such care.9,10 Co-locating MHPs within the primary care setting is one method used to achieve integration but may not be feasible in some settings because of workforce shortages,11 space constraints, and financial and administrative barriers.12

One way to address barriers to co-locating MHPs is through off-site integration, which we define as partnerships between primary care practices and non-embedded specialty MHPs whose involvement does not primarily take place on-site at the primary care practice. These arrangements involve an MHP or agency that provides a consultative or co-management service for a primary care practice, whether or not the service meets the complete definition of collaborative care. Such off-site integration may have certain advantages over on-site integration, such as the ability to serve larger and more dispersed populations as well as limited changes to existing infrastructure,13 reduced cost of travel for mental health specialists,14 less isolation of specialists from colleagues in their discipline, and fewer adjustments for mental health specialists in relation to practicing within medical culture and setting.11

Models of off-site integration vary primarily along two dimensions: direct vs. indirect (MHP sees the patient directly vs. provides consultation to PCP without seeing the patient directly) and in-person vs. remote (the patient or provider interaction happens face-to-face vs. via technology, such as by telephone, videoconferencing, or electronic messaging). Since models can vary along each of these dimensions, four intervention models of care are theoretically possible, as depicted in Figure 1: (1) direct in-person (e.g., in-person evaluations at specialty agencies, with coordination through primary care); (2) direct remote (e.g., telepsychiatry); (3) indirect in-person (e.g., when specialty and primary care providers convene in person for case conferences); and (4) indirect remote (e.g., telephone psychiatry consultation programs to PCPs).

Figure 1

Figure 1

Each of these models has advantages and disadvantages, as well as unique barriers and facilitators to implementation. For example, telepsychiatry (a direct remote model) limits travel demands and may facilitate providing more culturally and linguistically appropriate services,11 within less stigmatizing environments (e.g., primary care clinic, home) when on-site providers are not available.15 Certain families may also find it easier to share personal information via remote than via in-person assessments.16 In typical telepsychiatry models, however, mental health and primary care providers do not work with shared treatment plans and medical records, and providers do not regularly communicate about patients—all of which limit opportunities to coordinate care.11,15 In addition, remote models, compared to in-person delivery of service, may hinder the development of a therapeutic alliance.15 As another example, indirect models can enhance the care provided to patients with mild to moderate psychiatric conditions or symptoms in primary care and provide information and education to PCPs on treating these conditions. These models require providers to reserve time for consultations, however, and differences in office management and workflow style between primary care and MHC practices can be obstacles to synchronous communication.17 Indirect services also may not be billable to certain insurers, particularly when not associated with an office visit.17

Our goal was to review the existing literature on implementing these four models of off-site pediatric integrated MHC. We divided articles by model type and used the framework from Proctor and colleagues18 to extract whether and how each of eight implementation outcomes was measured in the articles reviewed: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability. We sought to understand what outcomes are being assessed and reported in the literature, and to identify factors that enable and hinder successful implementation and sustainability of off-site pediatric integrated MHC, including individuals involved, characteristics of the model, care processes, clinical setting, and broader context (e.g., political/economic/social context).

Back to Top | Article Outline

METHODS

We followed Arskey and O’Malley’s process for conducting scoping reviews using six steps,19 as reported in our previous review of co-located integrated care models, which used the same original search and a similar rigorous study-selection process.20 Unlike systematic reviews, which typically set out to answer a specific research question based on existing data, scoping reviews examine the state of research in a topic area, and identify knowledge gaps to guide further work. Our protocol involved (1) identifying our research question, (2) conducting a systematic search for studies, (3) reviewing search results to select relevant studies according to predefined inclusion/exclusion criteria, (4) “charting” the data by systematically extracting key information using a form, (5) collating and summarizing results, as well as discussing and agreeing on a “key implementation lesson” from each study, and (6) consulting with stakeholders for insights on findings and additional studies. Our team included a pediatrician (SP) and five child psychiatrists (AS, RP, MB, J Stratton, LW) with experience working in pediatric integrated care (including both direct and indirect, on- and off-site models), as well as both a psychologist (AB) and a telemental health program director (ES) with experience developing, implementing, and studying integrated care models.

Back to Top | Article Outline

Search Methods

The search strategy and keywords were described in detail in our previous review of co-located integrated care models.20 We included five umbrella concepts in this search, with related terms under each concept: (1) collaborative care (including terms such as integrated care, telepsychiatry, and child psychiatry access programs), (2) children/adolescents, (3) primary care (including the terms pediatrics and child health services), (4) mental health (including the term behavioral health), and (5) implementation outcomes (including the terms implement, accept, satisfaction, adopt, uptake, and utilization). Our broad search included both on-site programs (the focus of our previous review) and off-site programs. For this review, the coauthors manually reviewed full-text articles that were not designated as primarily co-located (and thus were not included in the prior review).

We searched the following electronic databases for relevant studies: Cochrane (up to 17 May 2018), Embase (up to 2 June 2018), MEDLINE (up to 29 May 2018), PsycInfo (up to 31 May 2018), and Web of Science (up to 2 June 2018). We also identified studies by checking references of the identified articles.

Back to Top | Article Outline

Inclusion/Exclusion Criteria

We included peer-reviewed, published studies that involved systematic data collection and analysis (quantitative or qualitative). These included clinical trials, observational studies, qualitative studies, program evaluations, and surveys, including those with and without comparison groups. We excluded studies not reporting on original data collection (e.g., reviews or editorials) and reports not published in peer-reviewed journals (e.g., conference abstracts, book chapters). We included studies reporting on implementation outcomes of integrated care interventions involving PCPs collaborating with off-site mental health specialists (e.g., psychiatrists, therapists) in the diagnosis or management of mental health disorders or symptoms in children or adolescents. Studies that included a combination of both on- and off-site MHPs were also included; however, we excluded studies that did not involve off-site specialists. Study populations had to be at least 50% under 18, with outcome data that applied specifically to child and adolescent mental health problems. We included studies of interventions for both common and severe psychiatric disorders as well as subthreshold mental health problems, but we excluded studies targeting only general health behaviors (e.g., treatment compliance for a medical condition). As before, we used Proctor’s framework18 for implementation outcomes (see Table 1).

Table 1

Table 1

Back to Top | Article Outline

Data Collection

Each abstract was initially screened independently by two reviewers for relevance to the study topic. Irrelevant abstracts were excluded, and potentially relevant articles were reviewed, independently by two reviewers, in full text. In both stages, disagreements were resolved through discussion, involving a third reviewer as needed. The following data were extracted from each study by a minimum of two reviewers and were inputted into an electronic form: author(s); year of publication; main intervention model (e.g., direct remote); secondary intervention models combined with the main model; PCP and specialist type; respective roles of PCP and specialist in managing behavioral health problems; intervention type and comparator (if any); study design; number of patients served/subjects included; population of patients served; funding source; limitations; implementation outcomes; and barriers/facilitators to implementation. In cases where more than one intervention model was used within the same program (e.g., direct remote and indirect remote components in a single program), we defined the main implementation model as the intervention received by the most patients. Differences in all extracted outcomes were resolved via discussion and consensus between the two data extractors and first authors (AS and RP).

Back to Top | Article Outline

Data Analysis

Data analysis was primarily qualitative in nature. First, we tabulated summaries of study characteristics and implementation outcomes. Of note, implementation outcomes were listed according to Proctor’s definitions but not necessarily by how they were named in an article (e.g., when authors reported on PCP satisfaction, we tabulated this outcome under acceptability). Next, for each study, a group of at least two reviewers discussed the study’s “key implementation lesson(s).” Themes were then identified across studies for each implementation outcome in Proctor’s framework.18 Relationships within and between studies were explored through reviewing tabulated data and themes. After the data and themes were identified, three authors (AS, RP, and AB) developed an overall synthesis of findings of included studies, presented in matrices of data contained in Supplemental Tables 1 and 2 (available as supplemental digital content at http://links.lww.com/HRP/A104 and http://links.lww.com/HRP/A105).13,22–61

Back to Top | Article Outline

RESULTS

The PRISMA flow diagram of articles screened and reasons for exclusion is depicted in Supplemental Figure 1 (available as supplemental digital content at http://links.lww.com/HRP/A106).62 Our initial search resulted in 5545 articles after removing duplicates, 458 of which were assessed for eligibility by full-text review. We found 39 original articles that described implementation outcomes from a total of 24 unique off-site, integrated programs for pediatric behavioral health care (in multiple cases, more than one article described outcomes from the same program or clinical trial; in one case, data from several state programs were summarized together). The interventions described used three different main intervention models: direct in-person (12 articles, 11 programs); direct remote (10 articles, 5 programs); and indirect remote (17 articles, 6 programs). None of the articles described a main indirect in-person model. Many articles included additional strategies beyond the main model (e.g., optional direct, in-person care at a specialty mental health agency as follow up after indirect remote consultation with PCP). Many programs also included an educational component for PCPs, but we report here on implementation outcomes specifically from patient care interventions.

Back to Top | Article Outline

Direct In-Person Models

Twelve articles described implementation of 11 different direct, in-person, off-site care models from five countries (England, Finland, Netherlands, New Zealand, and United States). Most programs were systematic partnerships between primary care sites and existing mental health agencies to provide care for patients. No study was randomized, but two were controlled, including the evaluation of a psychologist and patient navigator partnering with families, PCPs, and schools for managing attention-deficit/hyperactivity disorder (ADHD),38 and an expedited psychiatric consultation service for ADHD.40 The remaining studies were observational, with some including surveys. One other study evaluated a collaborative approach to ADHD treatment in the United States utilizing care-management liaising between primary care and a consulting child psychiatrist. Another observational study of direct in-person care evaluated a single-site child psychiatry consultation model in which the psychiatrists worked with local PCPs to provide consultation and collaborative treatment for children with anxiety, depression, and complex ADHD.42 Two studies described evaluations of the Primary Mental Health Worker (PMHW) program in England—one through interviews with stakeholders24 and the other through a survey of PMHWs.25 Three studies described implementation outcomes of interventions geared toward adolescents: both “Your Choice” in New Zealand37 and “SCREEN” in Finland27 were large observational studies of brief therapy in collaboration with primary care, while Gardner and colleagues34 tested a suicide-screening and -referral program in the American Midwest. The final three studies, two in Connecticut36,39 and one in the Midwest,41 described interventions focused on enhancing and formalizing connections and communication between sectors to improve delivery of collaborative care.

Back to Top | Article Outline

Acceptability and appropriateness of direct in-person models

Eight studies evaluated the acceptability and appropriateness (aka intervention “fit”) of direct in-person models for relevant stakeholders, including PCPs,30,36,39,40,42 primary care office staff,36 MHPs,24,25,39 parents,30,37,38 and patients.37 Overall, acceptability of interventions was high for all stakeholders. Interprofessional communication was an important element of increased satisfaction for both PCPs and MHPs, and was facilitated by the following: protocols for, and assistance with, information exchange; regular meetings/contact between PCPs and MHPs; and clear role definitions and protocols for shared care established ahead of time.30,36,39,40 PCPs reported that programs providing expedited access to psychiatric evaluations of designated patients (e.g., for ADHD evaluation/treatment recommendations, with planned transition back to primary care) both increased patients’ access to mental health services and improved the PCPs’ own skills in mental health care.40,42 Two studies describing the PMHW role in England contrasted satisfaction and appropriateness in each of three different models of integration based on primary affiliation and service location: outreach (community), specialty care, and primary care. PMHWs mainly working out of specialty settings reported the highest job satisfaction and least professional isolation, and did not encounter the same space concerns reported by others. Outreach PMHWs experienced significant tension in trying to reconcile the needs and expectations of providers in each setting (i.e., decreasing referrals to specialty care vs. connecting more patients to ongoing care) and in sorting out their actual role (i.e., as consultant vs. primary treater). Primary care-based PMHWs also experienced such tensions but benefitted from enhanced collaboration with primary care.24,25 Finally, in two other studies, providers expressed concern about unequal availability, related to insurance, of integrated care services to different patients.36,42

Back to Top | Article Outline

Adoption and penetration of direct in-person models

Three studies examined the adoption or penetration of direct in-person services.25,27,41 All three of the programs studied were population based (e.g., involving an entire county, health district, or, in the PMHW case, country). Two of these studies, one in England and one in the American Midwest, described increasing penetration over time. In the English study, the number of PMHWs increased over a seven-year period.25 In the American study, primary care and school-based mental health screening, as well as implementation of social-emotional classroom curricula, increased over a four-year period.41 The latter study also demonstrated population-level improvements associated with widespread adoption (increased graduation rates and decreased juvenile arrests).41 Differences in practice or patient characteristics contributed to variations in service adoption.27,41 For example, smaller practices with fewer pediatric patients were less likely, because of competing demands, to adopt mental health screening and referral.41 In Finland, adolescents referred from certain primary care centers were more likely to complete the SCREEN intervention and be referred for specialty care, suggesting that centers were referring patients with differing levels of need and interest.27

Back to Top | Article Outline

Feasibility of direct in-person models

Ten studies examined feasibility of services.25,27,30,34,37–42 Seven studies reported higher engagement and retention in care than what would be expected in usual-care models.27,30,34,37,38,41,42,63 Despite excellent overall engagement and retention, Laukkanen and colleagues27 found that high-risk adolescent populations (i.e., those with substance abuse, trauma, or functional impairments) remained harder to retain in care than others.27 Financial barriers at both patient and practice levels also interfered with feasibility (and ultimate sustainability) of in-person care.30,41,42 Multiple studies emphasized the importance of administrative support for implementation, noting that more-successful sites had more support for tasks such as scheduling and assisting with information exchange.30,39 Written forms, guidelines, and algorithms (e.g., to guide medication titration, or to formalize information exchange among providers) improved feasibility when available and, when they were not, were cited as a strategy to improve future feasibility.30,36,39,40 Multiple studies showed that integrated interventions improved the feasibility of adequately managing ADHD in primary care.30,38,40,42

Back to Top | Article Outline

Sustainability of direct in-person models

Although no studies formally evaluated sustainability, authors of several studies reported that lack of funding would be a challenge to sustaining interventions that had been successfully implemented using grant funding.30,41 In one study, participation in collaborative care was incentivized via enhanced Medicaid reimbursements for enrolled clinics in Connecticut36—which could be tested as a strategy to improve sustainability of collaborative care more broadly. Of note, one program reported an increase in referrals for higher-level specialty care associated with their integrated program, which was contrary to their expectation that integrated care would decrease demand for specialty care.24

Back to Top | Article Outline

Direct Remote Models

Ten articles described implementation of five different, primarily direct remote models in the United States (4) and Canada (1), of which two models were tested in randomized, controlled trials (RCTs). Four of the articles described implementation outcomes from the same large RCT on telepsychiatry (videoconferencing) for ADHD collaborative care out of Seattle Children’s Hospital.45–48 Three articles reported on implementing a single videoconferencing telepsychiatry program, also based at Seattle Children’s Hospital.30–32 One article described a small, pilot videoconferencing telepsychiatry program in rural Georgia.44 The other two articles described telephone-delivered, direct interventions targeting parents of young children: an RCT of coaching calls to, plus a self-help booklet for, parents of children with behavior problems in Ontario, Canada;29 and an observational study of a call-in service for parents in Nebraska.43

Back to Top | Article Outline

Acceptability and appropriateness of direct remote models

Four studies examined acceptability of direct remote models. All of these studies evaluated acceptability to parents, and two of the four evaluated acceptability to referring PCPs.30–32,44 Overall satisfaction ratings by both parents and PCPs were high. Some studies suggested differences in parent satisfaction and intervention appropriateness by a patient’s age or presenting symptoms. For example, Myers and colleagues31 found that parents of adolescents endorsed lower satisfaction with telepsychiatry services than parents of school-age children. Polaha and colleagues43 found that parents were less likely to report that their needs were met by a telephone consultation service if they called regarding conduct problems versus toileting, anxiety, and sleep. Based on satisfaction, length of calls, need for repeat calls, and ability to implement evidence-based interventions over the telephone with parents for these complaints,43 the authors concluded that a call-in service was most appropriate for concerns about toileting, repetitive behaviors, sleep, anxiety, and, in younger children, other behavior concerns.

Back to Top | Article Outline

Adoption and penetration of direct remote models

Factors affecting the adoption and penetration of four different direct remote programs were reported.30–32,43–48 The three main barriers to adoption and penetration of direct remote models were (1) lack of resources, (2) lack of specialist availability, and (3) low visibility to PCPs and patients. For example, Jacob and colleagues44 concluded that more time and funding for in-person orientation visits with PCPs prior to implementation would have improved the referral rate from PCPs to a telepsychiatry consultation service in rural Georgia. Polaha and colleagues43 cited limited specialist availability (one morning per week), lack of service visibility, and sparse population served as potential explanations for low use of a parent call-in psychology service.43 Myers and colleagues30 reported that pediatricians were more likely than family physicians to refer their patients to a telepsychiatry program based at Seattle Children’s Hospital. The same research group reported that engaging site champions and local stakeholders was important to increase visibility and utilization of remote services, and may have explained discrepancies in referral rates across sites.30–32

Back to Top | Article Outline

Feasibility of direct remote models

Overall, programs reported better initial engagement and retention in remotely provided, short-term interventions than has been found in previously published studies of outpatient mental health care.29,45–48,63 One telepsychiatry study reported a high no-show/cancellation rate for initial appointments despite multiple reminder calls, which they attributed to multiple factors, including the difficulty of coordinating among multiple entities and the difficulties that patients encountered regarding transportation, even to a “local” telepsychiatry site.44 The other telepsychiatry interventions served larger populations and appear to have been better resourced. Telephone consultations were generally brief,43 although duration depended on the symptoms or concerns. With videoconferencing methods, lack of comfort with the technology acted as a barrier to interest in, and use of, telepsychiatry.44–48 Notably, however, in the Children’s ADHD Treatment Study, technical difficulties were reported only in a minority of sessions and were rarely severe.45–48 Availability and turnover of MHPs was a barrier to the feasibility of multiple interventions.43,45–48

Back to Top | Article Outline

Fidelity of direct remote models

Fidelity of intervention delivery was evaluated in two different RCTs of direct remote care.35–38,42 Both trials reported high fidelity to treatment protocols administered remotely (including psychiatrists’ use of a medication algorithm in the Children’s ADHD Treatment Study, and graduate student therapists’ use of specific coaching behaviors with parents by telephone).35–38,42 The Children’s ADHD Treatment Study reported that therapists being supervised remotely also maintained high fidelity to their therapeutic protocol.47

Back to Top | Article Outline

Sustainability and cost of direct remote models

Information pertaining directly to the sustainability and cost of two interventions were reported.30–32,44 In both programs, reimbursement for telepsychiatry sessions was the same as for face-to-face visits. Jacob and colleagues44 noted that costs of telemedicine setup and maintenance were not covered. Myers and colleagues30–32 noted that a higher percentage of Medicaid patients were served in the Seattle Children’s telepsychiatry program than in traditional face-to-face outpatient clinic visits in the same network, thus reaching more traditionally underserved patients but resulting in lower collections for telepsychiatry visits.

Back to Top | Article Outline

Indirect Remote Models

Seventeen articles reported on implementing primarily indirect remote models in the United States (16 articles) and Canada (1 article). All used observational study designs, including retrospective chart reviews and program evaluation through surveys and interviews. The interventions described were all regional, real-time telephone consultation services (sometimes collectively referred to as child psychiatry access programs [CPAPs]) with MHPs—usually child and adolescent psychiatrists. All of these programs except one20 involved a component of direct care (either remote or in-person), if needed, following indirect consultation with a PCP. Most programs also made available some training or educational programming to PCPs.13,23,51,52,57,59

Over half (10) of the studies in this category reported implementation outcomes from the Massachusetts Child Psychiatry Access Program (MCPAP),22,23,33,49,50,52–55,61 which was the first such regional consultation program. Other implementation outcomes came from program evaluations of subsequent CPAP programs in Maryland,60 Michigan,59 New York State,13,57 Washington State,56 and Alberta, Canada.51 One study reported on a survey of program directors from multiple CPAP programs across the United States.58 None of these studies described results of controlled trials.

Back to Top | Article Outline

Adoption and penetration of indirect remote models

These studies reflect the increasing adoption of regional CPAP programs across the nation since the inception of MCPAP in 2004. Most studies described high levels of PCP enrollment and utilization in all regions where programs were available. Multiple studies noted, however, inconsistent utilization across regions, practices, and providers.23,51 A few studies examined practice- and provider-level characteristics affecting adoption and penetration. In Massachusetts, larger practices tended to be early adopters but then made fewer calls per patient after adoption. Later adopters tended to use the program more frequently.23 Other studies found geographical variation in PCP utilization of consultation support. In Massachusetts, more calls came to MCPAP from practices closer to the central hub,23 and in New York, more calls came to the Child and Adolescent Psychiatry for Primary Care (CAP PC) program from the upstate region.13 On the provider level, Van Cleave and colleagues55 found that in Massachusetts, frequent callers to the MCPAP preferred treating patients in primary care and appreciated both timely guidance and referral assistance. Infrequent callers reported lack of comfort with indirect consultation, access to other consultation sources, and calling less as they gained more skills by virtue of the program.55

Back to Top | Article Outline

Acceptability of indirect remote models

Nine articles reported on acceptability of CPAP programs to PCPs; in addition to reporting generally high acceptability, these studies found that PCPs felt much better equipped to meet patients’ mental health needs and felt an increase in skills and confidence in meeting those needs after implementation of CPAP programs.13,33,51–53,56,57,59,61 Timely/point-of-care consultation was an important aspect of high PCP satisfaction. The sole study that evaluated acceptability of a CPAP program to parents found moderate to high parent satisfaction with the MCPAP service.22 Parents reported lower levels of satisfaction, however, with the wait time for subsequent community referrals, and only about half of the parents reported clinical improvement in their children. No studies reported on the acceptability of these programs to MHPs.

Back to Top | Article Outline

Feasibility of indirect remote models

CPAP programs were most commonly used to address ADHD, anxiety, and depressive symptoms23,30,49,54 in school-age children and adolescents.13,53,54 PCPs in Massachusetts generally preferred other consultation methods for preschoolers, including developmental assessment centers, early intervention, schools, and community MHPs.53 Questions about medication, diagnosis, and assistance with community referrals were common across programs. Difficulty accessing ongoing psychiatric care or counseling was reported consistently as a challenge to feasibility of CPAP programs.22,57 One barrier to the feasibility of indirect remote models was PCPs’ concern that the model requires them to play more of a role in managing mental health problems than they feel appropriate.55 Multiple programs reported high symptom severity and functional impairment for a substantial minority of patients reviewed in consultations, many of whom were being managed by primary care physicians alone.49,60 In Massachusetts, PCP requests for community referrals have increased with time,52 and PCPs identified fear of feeling pressure to manage more than they were comfortable with as a barrier to seeking telephone consultation.55 One study reported that no lawsuits have been filed regarding these programs in the United States, which might be considered as an indication of program feasibility and potential for expansion.58

Back to Top | Article Outline

Fidelity of indirect remote models

Only one study reported on the fidelity of indirect remote consultation, noting that in quarterly blinded assessments of recommendations given from the Washington State Partnership Access Line, consultants had 95% fidelity to peer-reviewed, best-practice guidelines for child mental health.56

Back to Top | Article Outline

DISCUSSION

In this review, we sought to summarize the literature on implementing off-site pediatric integrated care. We identified 39 original research articles describing three distinct main models of off-site integrated care targeting children and adolescents: direct in-person, largely consisting of partnerships between primary and specialty care sites; direct remote, largely consisting of telepsychiatry and telephone-based consultation directly with patients; and indirect remote, largely consisting of primarily of child psychiatry access programs in which a remote child psychiatrist provides advice on cases to PCPs, generally at the point of care. We examined how implementation of each model was evaluated, and identified barriers and facilitators to successful program implementation.

Overall, all three integrated care models were highly acceptable to providers and families and were well utilized. Themes common to all models included higher PCP acceptability when the following were present: strong interpersonal, interdisciplinary connections and communication among providers; timely availability and reliability of specialty services; additional support beyond one-time consultation; and availability of standardized care algorithms. Adequate planning (including discussions with participating staff about program scope, methods of communication, and provider expectations) and ongoing administrative support improved feasibility, adoption, and penetration of interventions across models. Adoption and penetration also varied based on program visibility—facilitated by on-site practice champions, early adopters, and (in some cases) proximity to specialty hub—and the characteristics of primary care practices (e.g., larger size) and providers (e.g., pediatricians vs. family medicine providers) across models. Certain populations, including patients with less complicated ADHD, depression, or anxiety, were more amenable to off-site integrated models. Conversely, PCPs felt less comfortable treating certain patients, including preschoolers and those with those with conduct problems, significant functional impairment, multiple comorbidities, substance use, or autism spectrum disorder, with most off-site integrated care models. Funding and adequate reimbursement were barriers to sustainability in all models, particularly because reimbursing direct care interventions at “usual rates” ignores the need for start-up costs, administrative support (in some cases including technological assistance), and because indirect services (such as provider-to-provider consultations or coordination of care) have not been billable historically. While most programs used grant funding or state funding to cover these costs, others partnered with insurance companies to develop innovative models to begin addressing these barriers, such as the Massachusetts MCPAP program and Connecticut’s Enhanced Care Clinic program.17,22,23,33,36,49,50,52–55,58 The lack of specialist availability was a barrier to feasibility of direct models, and concerns about access to specialty referrals (with PCPs potentially being left with the responsibility for taking on care) were a barrier to feasibility of indirect models. Integrating school assessments and communicating with school staff presented challenges in studies including this component, regardless of the model.

Some important commonalities have been found in the factors that facilitate off-site versus on-site integrated behavioral health models for pediatric care—which we reviewed elsewhere.20 Strong interdisciplinary communication and strong collaboration between specialty providers and PCPs was important for high provider acceptability of both on- and off-site interventions. Thus, having a strong connection is a common motivator and driver of, and expectation within, successful implementation of all pediatric integrated care programs. While on-site models have an advantage in this realm because providers share a space (and often a medical record system), this review suggests that specific strategies can facilitate adequate communication even when space is not shared. In both on-site and off-site models, formalizing communication about patients and recommendations (e.g., by developing specific forms, systems, and expectations) improved interdisciplinary communication and satisfaction, and should be incorporated into any program. One study reviewed here showed that MHPs working in specialty settings tended to be less interested in communicating with PCPs about patients than vice versa but also that formalizing communication increased appreciation of these partnerships.39 In addition, algorithms to determine appropriateness for consultation (e.g., restricting to certain diagnoses or severity) and to assist with adherence to evidence-based treatment protocols enhanced the feasibility and effectiveness of both off-site and on-site models. Conversely, one study found that programs with enough flexibility to enable MHPs to help address crises as they arose also enhanced acceptability. Finally, some on- and off-site models led to increased referrals for specialty or emergency care.24,64 In both cases, the increases were likely a natural consequence of improved efforts to screen and identify mental health problems.65 It is reasonable to conclude that integrating mental health care with primary care (on- or off-site) is not a proven strategy for reducing utilization of specialty care, although it may improve identification of mental health problems and triage of complex patients to specialty care. In fact, reducing specialty referrals may not be an appropriate or acceptable goal of integrated programs, and may increase interpersonal tensions and decrease satisfaction for all involved.24

Certain barriers to implementation of on-site pediatric behavioral health care could be overcome by off-site collaborative models. First, practices that do not have the space or infrastructure to support an on-site MHP could still consider partnering with off-site clinicians to provide similar services. In some models, use of space was a source of concern among PCPs and staff.24 Second, professional isolation is an important cause of decreased job satisfaction for on-site MHPs,66 which can be overcome if they maintain a connection to their specialty “home.”25 The reach of off-site programs that can cover many pediatric practices is also appealing and well illustrated by the widespread adoption of child psychiatry access programs, in particular. Studies of CPAPs highlight variable adoption by practice and provider, however, both within programs (e.g., comparing one “regional hub” to another) and across programs (e.g., comparing one CPAP to another). While more research examining factors contributing to this variability is needed, authors of studies on the variability of MCPAP use have hypothesized that increasing adoption of on-site, embedded behavioral health providers may contribute to lower utilization by some (in particular, larger) practices.23 Nevertheless, practices with embedded providers who are not psychiatrists may utilize the service more because they are more able to carry out recommended behavioral health intervention and co-management.52 This point underscores the complementary roles that on-site and off-site MHPs could play in integrated care.

Off-site models also introduce new barriers not present in on-site models. The cost of, and comfort with, using technology to access specialty providers, along with the availability of technical support, are factors that may influence the use of direct remote strategies and potentially increase disparities in access to care. In indirect models, the continuing availability of “long-term” services in the community remains a concern, as is the severity of mental health problems that PCPs might be expected to manage without direct specialist involvement.17,22,23,33,49,50,52–55,58,59 The visibility of on-site services can suppress utilization of off-site services, even if formally complementary—an “out of sight, out of mind” phenomenon.30,31,35,44,55 Furthermore, without on-site providers, ongoing case consultation (including informal discussions that arise from being in proximity during clinic hours) may occur less frequently. Based on our findings, the critical task of relationship building is possible with off-site providers, but it may be more challenging and may require more effort. Finally, insurance barriers were more common in off-site models, and can lead to frustration from PCPs that their patients do not have equal access to the integrated off-site service.

In most interventions described in this review, one of the three models was primary, but components of other models were incorporated into care algorithms (e.g., the ability to “step up” to a direct in-person evaluation or short-term treatment if indirect remote consultation is not enough, or to seek indirect remote specialty advice following direct in-person evaluation if needed for continued guidance). The reviewed articles rarely mentioned indirect in-person arrangements even as a secondary model component (such as in-person case consultation between a PCP and a MHP), which is likely to be more common in on-site integration models. Even in off-site models, however, periodic in-person case conferences could help for coordination and learning in more-complex cases, promote relationship-building, and increase service visibility. Some well-developed programs do such case consultation remotely through videoconferencing—such as Project ECHO, a model for promoting education and increasing clinical capacity that has been replicated over 300 times in 35 countries.67

Our review has several limitations. Although the review process of article selection was rigorous, it is possible that we missed some relevant studies and thus lost potentially important information. We did not systematically assess the quality of the evidence, although at least two team members discussed each one of the studies and came to consensus on the key implementation lesson that could be learned from each study and applied in practice. This process, while helpful for data interpretation, does introduce subjectivity to our synthesis of the data. Due to the heterogeneity of programs, populations, and outcome measures, our summary of the data is descriptive. While all studies reported at least one implementation outcome, in many cases (about 40%) these outcomes were secondary measures, and the authors did not use an implementation science framework. Most outcomes were assessed retrospectively (e.g., with satisfaction surveys or interviews), and most studies did not randomize based on specific implementation strategies. In fact, few studies were randomized at all. This resulted in limited depth of information and limited use of standard measures. In addition, implementation outcomes were not consistently named in the literature according to Proctor’s definitions—which required us to rename outcomes from some studies to fit our paradigm. Certain outcomes, including fidelity, sustainability, and cost, were infrequently measured across interventions and models, most likely because they are the most difficult and costly to measure. While acceptability was measured more frequently, it was usually from a PCP perspective, sometimes from the patient or family perspective (in direct models only), and rarely from the perspective of MHPs (only in direct models). While adoption of programs by pediatricians and primary care practices was often measured, few data were reported on adoption of integrated programs by mental health specialists. Understanding the perspective of, and challenges reported by, specialists working in these models is critical to their expansion.

Back to Top | Article Outline

CONCLUSION

This is the first review of the implementation of off-site pediatric integrated behavioral health programs, which have the potential to expand integrated care without substantial infrastructure changes and to reach a broader population of patients than on-site programs. Despite some limitations, our review shows that these interventions are feasible, acceptable, and often adopted widely when there has been adequate planning, administrative support, and focus on communication and connection between primary care and specialty providers. Future experimental trials of interventions designed to test implementation strategies prospectively will be necessary to allow for a more rigorous exploration of fidelity, implementation cost, penetration, and sustainability. Studies that consider the perspectives of specialty providers, as well as those that test guidelines, protocols, and application of these models to specific populations, will be important for successful future expansion of these programs.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

The authors acknowledge the generous assistance of Syeda Hasan in creating Figure 1 of this manuscript.

Back to Top | Article Outline

REFERENCES

1. Costello EJ, Mustillo S, Erkanli A, Keeler G, Angold A. Prevalence and development of psychiatric disorders in childhood and adolescence. Arch Gen Psychiatry 2003;60:837–44.
2. Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry 2015;56:345–65.
3. Health Resources and Services Administration/National Center for Health Workforce Analysis; Substance Abuse and Mental Health Services Administration/Office of Policy, Planning, and Innovation. National Projections of Supply and Demand for Behavioral Health Practitioners: 2013–2025. 2015. https://bhw.hrsa.gov/sites/default/files/bhw/health-workforce-analysis/research/projections/behavioral-health2013-2025.pdf
4. Bringewatt EH, Gershoff ET. Falling through the cracks: gaps and barriers in the mental health system for America’s disadvantaged children. Child Youth Serv Rev 2010;32:1291–9.
5. Busch SH, Horwitz SM. Access to mental health services: are uninsured children falling behind? Ment Health Serv Res 2004;6:109–16.
6. Kurdyak P, Stukel TA, Goldbloom D, Kopp A, Zagorski BM, Mulsant BH. Universal coverage without universal access: a study of psychiatrist supply and practice patterns in Ontario. Open Med 2014;8:e87–99.
7. Butler M, Kane RL, McAlpine D, et al. Integration of mental health/substance abuse and primary care. Evid Rep Technol Assess (Full Rep) 2008;(173):1–362.
8. Wissow LS, van Ginneken N, Chandna J, Rahman A. Integrating children’s mental health into primary care. Pediatr Clin North Am 2016;63:97–113.
9. Heneghan A, Garner AS, Storfer-Isser A, Kortepeter K, Stein REK, Horwitz SM. Pediatricians’ role in providing mental health care for children and adolescents: do pediatricians and child and adolescent psychiatrists agree? J Dev Behav Pediatr 2008;29:262–9.
10. Horwitz SM, Storfer-Isser A, Kerker BD, et al. Barriers to the identification and management of psychosocial problems: changes from 2004 to 2013. Acad Pediatr 2015;15:613–20.
11. Fortney JC, Pyne JM, Turner EE, et al. Telepsychiatry integration of mental health services into rural primary care settings. Int Rev Psychiatry 2015;27:525–39.
12. Health AAOCAAPC on HCA and ETFOM. Improving mental health services in primary care: reducing administrative and financial barriers to access and collaboration. Pediatrics 2009;123:1248–51.
13. Kaye DL, Fornari V, Scharf M, et al. Description of a multi-university education and collaborative care child psychiatry access program: New York State’s CAP PC. Gen Hosp Psychiatry 2017;48:32–6.
14. Yilmaz SK, Horn BP, Fore C, Bonham CA. An economic cost analysis of an expanding, multi-state behavioural telehealth intervention. J Telemed Telecare 2018;1357633X18774181.
15. Fortney JC, Pyne JM, Mouden SB, et al. Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial. Am J Psychiatry 2013;170:414–25.
16. Hilty DM, Ferrer DC, Parish MB, Johnston B, Callahan EJ, Yellowlees PM. The effectiveness of telemental health: a 2013 review. Telemed J E Health 2013;19:444–54.
17. Sarvet BD, Wegner L. Developing effective child psychiatry collaboration with primary care: leadership and management strategies. Child Adolesc Psychiatr Clin N Am 2010;19:139–48; table of contents.
18. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health [Internet] 2009;36:24–34. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3808121/
19. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8:19–32.
20. Platt RE, Spencer AE, Burkey MD, et al. What’s known about implementing co-located paediatric integrated care: a scoping review. Int Rev Psychiatry 2018;30:242–71.
21. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65–76.
22. Dvir Y, Wenz-Gross M, Jeffers-Terry M, Metz WP. An assessment of satisfaction with ambulatory child psychiatry consultation services to primary care providers by parents of children with emotional and behavioral needs: the Massachusetts Child Psychiatry Access Project University of Massachusetts Parent Satisfaction Study. Front Psychiatry 2012;3:7.
23. Van Cleave J, Le T-T, Perrin JM. Point-of-care child psychiatry expertise: the Massachusetts Child Psychiatry Access Project. Pediatrics 2015;135:834–41.
24. Macdonald W, Bradley S, Bower P, et al. Primary mental health workers in child and adolescent mental health services. J Adv Nurs 2004;46:78–87.
25. Hickey N, Kramer T, Garralda E. Developing the Primary Mental Health Worker role in England. Child Adolesc Ment Health 2010;15:23–9.
26. Karsh B-T. Beyond usability: designing effective technology implementation systems to promote patient safety. Qual Saf Health Care 2004;13:388–94.
27. Laukkanen E, Hintikka JJ, Kylmä J, Kekkonen V, Marttunen M. A brief intervention is sufficient for many adolescents seeking help from low threshold adolescent psychiatric services. BMC Health Serv Res 2010;10:261.
28. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract 2008;14:117–23.
29. Reid GJ, Stewart M, Vingilis E, et al. Randomized trial of distance-based treatment for young children with discipline problems seen in primary health care. Fam Pract 2013;30:14–24.
30. Myers KM, Vander Stoep A, McCarty CA, et al. Child and adolescent telepsychiatry: variations in utilization, referral patterns and practice trends. J Telemed Telecare 2010;16:128–33.
31. Myers KM, Valentine JM, Melzer SM. Child and adolescent telepsychiatry: utilization and satisfaction. Telemed J E Health 2008;14:131–7.
32. Myers KM, Valentine JM, Melzer SM. Feasibility, acceptability, and sustainability of telepsychiatry for children and adolescents. Psychiatr Serv 2007;58:1493–6.
33. Straus JH, Sarvet B. Behavioral health care for children: the Massachusetts Child Psychiatry Access Project. Health Aff (Millwood) 2014;33:2153–61.
34. Gardner W, Klima J, Chisolm D, et al. Screening, triage, and referral of patients who report suicidal thought during a primary care visit. Pediatrics 2010;125:945–52.
35. Myers K, Stoep AV, Thompson K, Zhou C, Unützer J. Collaborative care for the treatment of Hispanic children diagnosed with attention-deficit hyperactivity disorder. Gen Hosp Psychiatry 2010;32:612–4.
36. Pidano AE, Marcaly KH, Ihde KM, Kurowski EC, Whitcomb JM. Connecticut’s enhanced care clinic initiative: early returns from pediatric-behavioral health partnerships. Fam Syst Health 2011;29:138–43.
37. Clark TC, Johnson EA, Kekus M, et al. Facilitating access to effective and appropriate care for youth with mild to moderate mental health concerns in New Zealand. J Child Adolesc Psychiatr Nurs 2014;27:190–200.
38. Power TJ, Mautone JA, Marshall SA, et al. Feasibility and potential effectiveness of integrated services for children with ADHD in urban primary care practices. Clin Pract Pediatr Psychol 2014;2:412–26.
39. Greene CA, Ford JD, Ward-Zimmerman B, Honigfeld L, Pidano AE. Strengthening the coordination of pediatric mental health and medical care: piloting a collaborative model for freestanding practices. Child Youth Care Forum 2016;45:729–44.
40. Hassink-Franke LJA, Janssen MMM, Oehlen G, et al. GPs’ experiences with enhanced collaboration between psychiatry and general practice for children with ADHD. Eur J Gen Pract 2016;22:196–202.
41. Huber BJ, Austen JM, Tobin RM, Meyers AB, Shelvin KH, Wells M. Overcoming barriers to rural children’s mental health: an interconnected systems public health model. Adv Sch Ment Health Promot 2016;9:219–41.
42. Fallucco EM, Blackmore ER, Bejarano CM, et al. Collaborative care: a pilot study of a child psychiatry outpatient consultation model for primary care providers. J Behav Health Serv Res 2017;44:386–98.
43. Polaha J, Volkmer A, Valleley RJ. A call-in service to address parent concerns about child behavior in rural primary care. Fam Syst Health 2007;253:333–43.
44. Jacob MK, Larson JC, Craighead WE. Establishing a telepsychiatry consultation practice in rural Georgia for primary care physicians: a feasibility report. Clin Pediatr (Phila) 2012;51:1041–7.
45. Myers K, Vander Stoep A, Lobdell C. Feasibility of conducting a randomized controlled trial of telemental health with children diagnosed with attention-deficit/hyperactivity disorder in underserved communities. J Child Adolesc Psychopharmacol 2013;23:372–8.
46. Stoep AV, Myers K. The Children’s Attention-Deficit Hyperactivity Disorder (ADHD) Telemental Health Treatment Study: methodology for conducting a trial of telemental health in multiple underserved communities. Clin Trials 2013;10:949–58.
47. Myers K, Vander Stoep A, Zhou C, McCarty CA, Katon W. Effectiveness of a telehealth service delivery model for treating attention-deficit/hyperactivity disorder: a community-based randomized controlled trial. J Am Acad Child Adolesc Psychiatry 2015;54:263–74.
48. Rockhill CM, Tse YJ, Fesinmeyer MD, Garcia J, Myers K. Telepsychiatrists’ medication treatment strategies in the Children’s Attention-Deficit/Hyperactivity Disorder Telemental Health Treatment Study. J Child Adolesc Psychopharmacol 2016;26:662–71.
49. Connor DF, McLaughlin TJ, Jeffers-Terry M, et al. Targeted child psychiatric services: a new model of pediatric primary clinician–child psychiatry collaborative care. Clin Pediatr (Phila) 2006;45:423–34.
50. Aupont O, Doerfler L, Connor DF, Stille C, Tisminetzky M, McLaughlin TJ. A collaborative care model to improve access to pediatric mental health services. Adm Policy Ment Health 2013;40:264–73.
51. Lipton H, Raivio E, Perrault E, et al. Integrating children’s mental health in primary health care. Can J Community Ment Health 2008;27(2):153–63.
52. Sarvet BD, Ravech M, Straus JH. Massachusetts Child Psychiatry Access Project 2.0: a case study in child psychiatry access program redesign. Child Adolesc Psychiatr Clin N Am 2017;26:647–63.
53. Sheldrick RC, Mattern K, Perrin EC. Pediatricians’ perceptions of an off-site collaboration with child psychiatry. Clin Pediatr (Phila) 2012;51:546–50.
54. Hobbs Knutson K, Masek B, Bostic JQ, Straus JH, Stein BD. Clinicians’ utilization of child mental health telephone consultation in primary care: findings from Massachusetts. Psychiatr Serv 2014;65:391–4.
55. Van Cleave J, Holifield C, Perrin JM. Primary care providers’ use of a child psychiatry telephone support program. Acad Pediatr 2018;18:266–72.
56. Hilt RJ, Romaire MA, McDonell MG, et al. The Partnership Access Line: evaluating a child psychiatry consult program in Washington State. JAMA Pediatr 2013;167:162–8.
57. Gadomski AM, Wissow LS, Palinkas L, Hoagwood KE, Daly JM, Kaye DL. Encouraging and sustaining integration of child mental health into primary care: interviews with primary care providers participating in Project TEACH (CAPES and CAP PC) in NY. Gen Hosp Psychiatry 2014;36:555–62.
58. Hobbs Knutson K, Wei MH, Straus JH, Sarvet B, Masek BJ, Stein BD. Medico-legal risk associated with pediatric mental health telephone consultation programs. Adm Policy Ment Health 2014;41:215–9.
59. Marcus SM, Malas NM, Quigley JM, et al. Partnerships with primary care for the treatment of preschoolers. Child Adolesc Psychiatr Clin N Am 2017;26:597–609.
60. Platt R, Pustilnik S, Connors E, Gloff N, Bower K. Severity of mental health concerns in pediatric primary care and the role of child psychiatry access programs. Gen Hosp Psychiatry 2018;53:12–8.
61. Sarvet B, Gold J, Bostic JQ, et al. Improving access to mental health care for children: the Massachusetts Child Psychiatry Access Project. Pediatrics 2010;126:1191–200.
62. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700.
63. Gopalan G, Goldstein L, Klingenstein K, Sicher C, Blake C, McKay MM. Engaging families into child mental health treatment: updates and special considerations. J Can Acad Child Adolesc Psychiatry 2010;19:182–96.
64. Hacker KA, Penfold RB, Arsenault LN, Zhang F, Soumerai SB, Wissow LS. Effect of pediatric behavioral health screening and colocated services on ambulatory and inpatient utilization. Psychiatr Serv 2015;66:1141–8.
65. Hacker K, Penfold R, Arsenault LN, Zhang F, Soumerai SB, Wissow LS. The impact of the Massachusetts behavioral health child screening policy on service utilization. Psychiatr Serv 2017;68:25–32.
66. Briggs RD, German M, Schrag Hershberg R, Cirilli C, Crawford DE, Racine AD. Integrated pediatric behavioral health: implications for training and intervention models. Prof Psychol Res Pract 2016;47:312–9.
67. Dearing JW, Cruz S, Kee K, Larson RS, Rahm AK. Project ECHO review and research agenda. Diffusion Associates, 2019. http://www.diffusionassociates.com/pdfs/echo.pdf
Keywords:

child psychiatry; consultation; pediatrics; remote consultation

Supplemental Digital Content

Back to Top | Article Outline
© 2019 President and Fellows of Harvard College