Secondary Logo

The role of organizational context in the implementation of a statewide initiative to integrate mental health services into pediatric primary care

King, Melissa A.; Wissow, Lawrence S.; Baum, Rebecca A.

doi: 10.1097/HMR.0000000000000169

Background: Although there is evidence that mental health services can be delivered in pediatric primary care with good outcomes, few changes in service delivery have been seen over the past decade. Practices face a number of barriers, making interventions that address determinants of change at multiple levels a promising solution. However, these interventions may need appropriate organizational contexts in place to be successfully implemented.

Purpose: The objective of this study was to test whether organizational context (culture, climate, structures/processes, and technologies) influenced uptake of a complex intervention to implement mental health services in pediatric primary care.

Methodology/Approach: We incorporated our research into the implementation and evaluation of Ohio Building Mental Wellness Wave 3, a learning collaborative with on-site trainings and technical assistance supporting key drivers of mental health care implementation. Simple linear regression was used to test the effects of organizational context and external or fixed organizational characteristics on program uptake.

Results: Culture, structure/processes, and technologies scores indicating a more positive organizational context for mental health at the project’s start, as well as general cultural values that were more group/developmental, were positively associated with uptake. Patient-centered medical home certification and use of electronic medical records were also associated with greater uptake. Changes in context over the course of Building Mental Wellness did not influence uptake.

Conclusion: Organizational culture, structures/processes, and technologies are important determinants of the uptake of activities to implement mental health services in pediatric primary care. Interventions may be able to change these aspects of context to make them more favorable to integration, but baseline characteristics more heavily influence the more proximal uptake of program activities.

Practice Implications: Pediatric primary care practices would benefit from assessing their organizational context and taking steps to address it prior to or in a phased approach with mental health service implementation.

Melissa A. King, PhD, MPAff, is Principal, People’s Health Solutions, Los Angeles, California. E-mail:

Lawrence S. Wissow, MD, MPH, is Professor, Division of Child and Adolescent Psychiatry, Johns Hopkins School of Medicine, Baltimore, Maryland.

Rebecca A. Baum, MD, is Faculty, Section of Developmental Behavioral Pediatrics, Nationwide Children’s Hospital, Columbus, Ohio.

BMW Wave 3 was supported by the Ohio Colleges of Medicine Government Resource Center with funding from the Ohio Department of Health and the Ohio Department of Medicaid. This research was additionally supported by National Institute of Mental Health Grant P20MH086048 for the Johns Hopkins Center for Mental Health Services in Pediatric Primary Care.

The authors have disclosed that they have no significant relationship with, or financial interest in, any commercial companies pertaining to this article.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (

Mental health conditions are the leading cause of childhood disability in the United States (Halfon, Houtrow, Larson, & Newacheck, 2012). By supporting families through critical windows of development, pediatricians may disrupt the modes by which mental conditions originate and evolve, helping youth to achieve better health. Yet only one in five youth with mental conditions are identified by their pediatricians, and only a fraction of these youth go on to receive treatment (Simonian, 2006).

Despite evidence that mental health services can be effectively delivered in primary care settings (Tice et al., 2015), pediatricians have reported few changes in the amount of mental health care provided over the past decade (Stein et al., 2016). This lag reflects barriers at multiple levels, from inadequate training (Green, Hampton, Ward, Shao, & Bostwick, 2014), to organizational cultures marked by stress and resistance to change (Bloom & Farragher, 2013), to financial disincentives (O’Donnell, Williams, & Kilbourne, 2013). Overcoming such barriers is essential given the many advantages to addressing mental health in the primary care setting. Primary care practices—whether in schools or hospitals or community centers—are places that youth and families visit regularly, making them natural points of access to care. Their philosophy of promoting and tracking child health and development complements the task of detecting emerging mental health issues as they arise. Through trusting alliances, pediatric staff may engage youth in brief interventions, introduce them to mental health providers, create bridges to community resources, and reduce stigma by treating mental health as part of whole health.

Overcoming barriers to integration requires what are known as complex interventions—that is, interventions with multiple components that act interdependently and that facilitate change in equally complex systems. Primary care’s adoption of medical home and accountable care models and the switch from paper to electronic medical records (EMRs) are examples of changes that have required complex interventions (for more examples, see the Agency for Healthcare Research and Quality Academy, Patient-Centered Primary Care Collaborative, and SAMHSA-HRSA Center for Integrated Health Solutions).

To date, efforts to implement complex interventions in primary care have focused more on “colder” organizational structures, processes, and technologies (e.g., workflows, checklists, and screening tools) and less so on the “warmer” human elements of these living systems, most notably culture and climate. This is problematic because, by definition, complex interventions additionally involve changes to the deeper values and experiences that determine why people work and interact the way they do. Failure to address each of the relevant features of a practice’s organizational context is thought to stymie implementation (Nembhard, Alexander, Hoff, & Ramanujam, 2009).

This study was part of the Building Mental Wellness (BMW) Wave 3 Learning Collaborative, a statewide effort to improve detection and management of child and youth mental health conditions among primary care providers in Ohio. BMW Wave 3 was modeled on the Institute for Healthcare Improvement Breakthrough Series and included on-site trainings that engaged all staff—from front desk to physicians—to support transformative change. The objective was to test whether a practice’s organizational context was associated with uptake.

Prior research in primary care settings has provided some evidence of links between organizational context and intervention uptake. Structure/processes and technologies have been found to relate to the implementation of new models for depression management (Chang et al., 2013) and of colocated mental health care (Guerrero, Heslin, Chang, Fenwick, & Yano, 2015). Culture and climate have in turn been found to influence buy-in to different types of quality improvement (QI) programs (Kaissi, Kralewski, Curoe, Dowd, & Silversmith, 2004) and implementation of computerized tools for screening and brief intervention (Carlfjord, Andersson, Nilsen, Bendtsen, & Lindberg, 2010). In a study by Dickinson et al. (2014), work culture and change culture, as well as staff agreement on culture ratings, predicted uptake of diabetes management interventions in primary care. Still, organizational context has not yet been fully explored as having a role in pediatric primary care’s transition to a larger role in mental health care.

Back to Top | Article Outline


We define organizational context as attributes of the workplace environments and interorganizational networks in which primary care staff operate. Building upon the seminal work of Glisson (2002), we identify four context domains. Culture refers to shared values and norms that set behavioral expectations within a practice (Schein, 2010). Climate refers to staff members’ shared perceptions of what happens within a practice and its impact on their experiences and well-being (James & Sells, 1981 ; Schneider, Ehrhart, & Macey, 2013). Research in child social services has provided evidence that culture and climate are distinct but related constructs (Glisson & James, 2002). Structure is used to refer to standardized policies and procedures and the configuration of roles and authority in an organization; the word processes is appended to emphasize that structure encompasses planned sequences of events in health care delivery. Technologies refer to the built environment, tools, equipment, and other resources used by staff as they carry out their work.

When measuring organizational context, one may have a specific outcome in mind (e.g., does Somecity Pediatrics have a workplace culture that is conducive to mental health care) or seek a general sense of characteristics that influence how people work and interact (e.g., does Somecity Pediatrics have a workplace culture that emphasizes flexibility or stability). We sought an understanding of both and therefore employed two context measures: a mental health-specific survey of culture, climate, structures/processes, and technologies (King, 2016; Measure 1) and a more general measure of change orientation based on the competing values framework (CVF; Quinn & Kimberly, 1984; Measure 2).

Measure 1, described in greater detail by King (2016), is a survey of contextual attributes that theory suggests would facilitate implementation of mental health services in pediatric primary care. Structure/processes and technologies items were modified from the American Academy of Pediatrics (AAP) Mental Health Practice Readiness Inventory (AAP, 2010). These items map to elements of the chronic care model (Wagner, Austin, & Von Korff, 1996), such as community linkages, delivery system redesign, child and family support, and decision support. Culture items reflect attributes such as research evidence relating to a mental health being valued (Van Patter Gale & Schaffer, 2009) and responsibility for mental health care being shared across all roles and levels (Grumbach & Bodenheimer, 2004). Climate items were informed by scales used in primary care (Becker & Roblin, 2008) and child social service settings (Glisson et al., 2008) and reflect attributes such as stress/chaos (Ohman-Strickland et al., 2007) and burdens such as caseload and paperwork (Rubin Stiffman et al., 2001). Measure 2 served to place practices on the continuum of the CVF, with research suggesting that more group/developmental values would be conducive to QI implementation (Shortell et al., 1995).

We had several hypotheses about how organizational context would relate to uptake.

Hypothesis 1 (based on Measure 1): Practices with culture and climate attributes that prior research suggests would facilitate mental health care implementation, as well as structures/processes and technologies that align with the chronic care model, were expected to relate to greater uptake.

Hypothesis 2 (based on Measure 2): Practices with a general culture emphasizing all-staff participation and risk-taking, corresponding to CVF ratings of a more group- and developmental-oriented culture, were expected to demonstrate greater uptake.

In addition to these context measures, we considered practice characteristics that are external or fixed in the sense that participants were not asked to change them over the course of BMW: participation in an accountable care organization, certification as a patient-centered medical home (PCMH), colocation of mental health specialist(s) on-site, payer mix, health system affiliation, weekly patient volume, past QI experience, use of EMRs, and suburban/urban/rural location. We made hypotheses about how external or fixed context characteristics were expected to support or hinder change.

Colocation and accountable care organization and PCMH status were expected to predict positive uptake, given the preexistence of care coordinating mechanisms and greater potential for payment models that facilitate equitable reimbursement for mental health care (O’Donnell et al., 2013). Conversely, a payer mix characterized by a high percentage of publicly insured patients was hypothesized to have a negative association with change, given potentially greater barriers to reimbursement for mental health services among Medicaid-enrolled patients (Mauch, Kautz, & Smith, 2008).

EMR use and prior QI experience were hypothesized to predict greater uptake, because they might be seen as markers of past experience with change and vehicles for sharing information (Okhuysen & Bechky, 2009). Rural practices were hypothesized to experience less change, given barriers they face to making changes to structures/processes such as specialist proximity and referral assistance, even though surveys show clinicians in rural practices tend to provide more mental health care (Penfold, Kelleher, Horwitz, & Stein, 2005).

Lastly, we did not hypothesize the direction of any relationship between uptake and weekly patient volume (proxy for organization size) or health system affiliation. On the one hand, resource dependency theory might suggest that external resources provide incentive for practices to risk-take and innovate. Goldberg and Mick (2010) found that primary care practices that are larger in size or affiliated with a health care system tend to have greater resource availability or “slack” and are therefore more likely to implement change. However, group cultures tend to predominate in primary care (Dowswell, Harrison, & Wright, 2001), and to the extent that being part of a larger organization or system might influence values to be less group-oriented or exert pressure to conform, these factors could also be detrimental.

Back to Top | Article Outline


BMW Wave 3 Learning Collaborative

BMW Wave 3 was coordinated by the AAP, Ohio Chapter, with the goal of engaging pediatric primary care practices in mental health service implementation. From October 2013 through May 2015, 21 practices endeavored to adopt a package of activities in five areas (resources, tracking, mental health promotion and screening, integration planning and implementation, and practice-based interventions; see Supplemental Digital Content 1, These activities and associated engagement processes and structures were planned to target three key drivers of integration:

  1. skills to support prevention, early identification, and management of mental health concerns (individual staff level);
  2. organizational culture and climate that support the delivery of family-centered mental health services (organization level), and
  3. organizational structures/processes and technologies known to support mental health service delivery, as exemplified by the chronic care model.

Content for the activities was based on the AAP core competencies for mental health, as well as communication and brief intervention practices for supporting mental health care delivery in primary care (Brown & Wissow, 2012). In the first month, practices engaged in survey evaluations of their organizational context for mental health and participated in an all-staff learning session with other practices. Team members specializing in QI helped practices reflect upon their Office Inventories (see Measures) and identify strengths and areas for improvement to plan practice changes and actualize them through Plan, Do, Study, Act cycles. Physician trainers conducted a series of site visits that involved interactive, all-staff trainings in core mental health-related skills. Over the course of the 18 months, practices had monthly meetings with the QI team to discuss and receive formal recognition for progress toward their BMW goals. A repeat evaluation of organizational context was conducted in the final month.

Back to Top | Article Outline

Research Design

BMW Wave 3 was itself a QI initiative, but components of its evaluation were approved as human subjects research by the Nationwide Children’s Hospital Institutional Review Board (IRB13-00397). In this quasiexperimental study, we used simple linear regression to test whether organizational context at baseline and changes to context from baseline to postintervention were associated with intervention uptake. Survey data were collected at baseline (Month 1) and postintervention (Month 18) for organizational context and postintervention (Month 18) for uptake.

Back to Top | Article Outline


Outcome variable

Program uptake was measured at the primary care practice level. QI coordinators met with staff from each practice monthly to discuss their progress and agree upon the level of completion of each of 17 discreet activities that were targets of BMW (see Supplemental Digital Content 1, In rare instances where a practice had completed an activity prior to the start of BMW (e.g., create a resource directory), staff were asked to identify an alternative goal in that area (e.g., update the existing resource directory) or received credit if the goal had been fully implemented. Each activity was rated on a scale of 1–4, where 1 = no changes begun, 2 = changes being tested, 3 = improvements implemented, and 4 = fully implemented/sustained. At the end of the intervention period, activity ratings were added up, and practices were assigned a continuous star score (possible range: 17–68) based on their degree of implementation of each activity.

Back to Top | Article Outline

Independent variables

Organizational Context Measure 1 was a revised version of the AAP (2010) Mental Health Readiness Inventory, described herein as the “Office Inventory.” The AAP tool was modified to include the four context domains detailed in the Theory section. Items were worded in a such a way that higher scores signify the presence of an attribute hypothesized to be supportive of mental health practice; higher domain scores in turn represent a context more favorable to program uptake.

Structures/processes (15 items) and technologies (5 items): operationalized as indices of items indicative of the presence of absence of structures (e.g., “my practice has staff roles assigned to effectively monitor patients’ progress…”), processes (e.g., “my practice coordinates with youth, families, schools…”), and technologies (e.g., “I use psychosocial history and validated screening and assessment tools…”) supportive of mental health practice.

Culture (6 items): operationalized as a scale measuring artifacts (e.g., “my coworkers are supportive of people facing mental health challenges”) and values (e.g., “people in my practice value research relating to the importance of early detection and treatment of mental health conditions”; Schein, 2010). Whereas espoused values are thought to be a measure of culture, some conceptualize observable artifacts (i.e., behaviors, practices, routines) as the “linking mechanism” between culture and climate (Ostroff, Kinicki, & Muhammad, 2013).

Climate (10 items): operationalized as a scale with one or more items for each climate dimension identified by James and Sells (1981), including stress, time constraints, and role clarity (role level); trust and autonomy (job level); support from coworkers (workgroup level); and safety to express ideas (leader and organization level).

Domain scores: Structure/processes, culture, and climate domains were measured on an ordinal scale, where 1 = strongly disagree and 10 = strongly agree. Item scores were converted to percentages, ranging from 0 to 100%, and domain scores were in turn calculated for each respondent as an average of items within each domain. The technologies (5 items) domain was measured on a continuous scale of percentage of agree/disagree items rated “agree.”

Total context scores: Research suggests that, although culture and climate represent distinct constructs, they tend to be correlated (Glisson & James, 2002). Pearson’s correlation coefficients—as well as linear regressions of outcomes on our four domains followed by assessment of variance inflation factors—were used to assess relationships between the four domains and establish the validity of calculating total context scores to be used in lieu of individual domain scores in multivariable models. Change in total context score may then be calculated as the difference between total context scores at baseline and postintervention.

Organizational Context Measure 2 was an application of the CVF (Quinn & Kimberly, 1984) included in the Office Inventory. The CVF places organizations on a continuum of four culture types: human relations model (group), open systems model (developmental), internal process model (hierarchical), and rational goal model (rational). Each type varies on “competing values,” which include internal versus external focus, stability versus flexibility, and different ways of approaching work (e.g., flexibility vs. planning) and goals (e.g., growth vs. productivity). Respondents were asked to distribute 100 points across sets of statements, where there is one statement in each set reflecting each of the four cultural types. The average percentage of points assigned to group or developmental values was calculated for each practice. A combined score of 50% or more is thought to indicate a group/developmental emphasis conducive to QI implementation (Shortell et al., 1995).

External or fixed organizational characteristics, as described in the introduction, were captured at the practice level through registration forms completed by the designated practice leads at baseline. Nine practice characteristics were measured using responses completed by the designated practice lead: high percentage of publicly insured patients (dichotomous, 0 or 1); health system affiliation (dichotomous, 0 or 1); PCMH status (dichotomous, 0 or 1); accountable care organization status (dichotomous, 0 or 1); geographic location (categorical, 0 = urban, 1 = suburban, 2 = rural); practice’s weekly volume (continuous); one or more colocated mental health specialists (psychiatrist, psychologist, counselor, social worker; dichotomous, 0 or 1); prior experience with a QI project (dichotomous, 0 or 1); and use of EMRs to track patient progress and/or outcomes (dichotomous, 0 or 1). The breakpoint for high versus low/average percentage of publicly insured patients of 30% or more was based on inspection of the metric’s distribution and AAP survey data. High versus low/average weekly volume was assigned based on whether a practice’s volume was one or more standard deviations above the learning collaborative mean.

Back to Top | Article Outline

Data Analysis

Climate and culture are attributes of an organization, but in this as in many studies, they are measured by surveying individuals, whose perceptions may vary. When individual ratings tend to agree they may be aggregated to yield a single rating at the organization level. We used the rWG ( J ) index, developed for assessing agreement when a group of judges (in our case, staff) rate the same organization (practice) on multiple parallel items (James, Demaree, & Wolf, 1984), to measure within-practice inter-rater agreement on each domain. The index represents the reduction in variance reflected if one compares the average observed variance in staff scores across items to the variance one would expect if there were a complete lack of agreement. It ranges in value from 0.00 (no agreement) to 1.00 (perfect agreement). When the rWG ( J ) index indicates substantial agreement, it may be used to justify aggregating individual responses to represent an attribute of a higher-level construct; in our case, a primary care practice (LeBreton & Senter, 2008).

We tested the effect of organizational context on BMW Wave 3 program uptake at the practice level. Outcomes were modeled using simple linear regression as a function of culture, climate, structures/processes, technologies, baseline total context score, change in total context score, percent group/developmental culture, interrater agreement, and external or fixed practice characteristics. We lacked the power to combine all independent variables into an adjusted model given the modest sample of practices. To ease interpretation, percentages were transformed into whole numbers by multiplying them by 100 prior to inclusion in regression models. Analyses were conducted using the statistical analysis software Stata (StataCorp, 2011).

Back to Top | Article Outline


Description of Practice Characteristics

Staff from 29 facilities, grouped into 21 practices, elected to participate in BMW. Eleven of these facilities were school-based health centers (SBHCs) from a single Ohio metropolitan area. Although the SBHCs were located on school property, they functioned as stand-alone family medicine practices that also served the surrounding community, and they self-organized to participate under one of three groups—preschool to 6th grade, preschool to 8th/12th grade, and high school only—yielding three distinct practices. The SBHCs did not engage in individual program uptake ratings and were not included in our analysis. The remaining 18 facilities were community- or hospital-based pediatric or family medicine practices located in rural (n = 4), suburban (n = 8), and urban (n = 6) settings. Two urban practices dropped out prior to beginning the program, and one suburban practice dropped out after the first few months. The latter was a small practice with one participating clinician who reported being unable to obtain buy-in from practice leadership. The characteristics of the 15 remaining practices that completed the BMW program and participated in uptake ratings are described in Table 1.

Table 1

Table 1

Nine practices (60%) reported being affiliated with a health system, two (13%) participated in an accountable care organization, and two (13%) were certified as a PCMH. More than half (n = 9) reported past experience with QI projects. All but two reported using EMR to track patient progress and/or outcomes. Ten of 15 practices reported a high percentage (≥30%) of publicly insured patients (range: 25%–98%). Practices served on average 322 patients per week (range: 75–775) and had on average 16 total staff (range: 6–45), five of whom were pediatricians or nurse practitioners (range: 2–11). Half (n = 7) reported having one or more specialists (psychiatrist, psychologist, counselor, social worker) colocated on-site.

Back to Top | Article Outline

Changes in Organizational Context

All 15 practices included in our analysis participated in the Office Inventory, although one (ID 8 in Table 1) participated at baseline but not postintervention. A total of 341 surveys were completed: 201 at baseline and 140 postintervention. Distribution of roles was similar across time points, with approximately three quarters of respondents self-reporting being in clinical roles baseline (72% or n = 144) and post (75% or n = 105; Table 2).

Table 2

Table 2

Aggregated organizational context domain scores at baseline and postintervention are displayed in Supplemental Digital Content 2 (see Supplemental Digital Content 2, Interrater agreement (rWG ( J )) based on the uniform theoretical null distribution was moderate to very strong in all but 1 of the 14 practices participating at both times, based on LeBreton and Senter's (2008, p. 836) heuristic for interpreting estimates. This, combined with moderate to strong between-practice differences (ICC Type 1; see Supplemental Digital Content 3,, justified using an average of individual staff scores within each practice to represent organizational context. Across practices, an average of 60% of points was assigned to group and developmental values on the CVF (min: 0.41, max: 0.83, median: 0.56), indicating a substantial emphasis on these, as opposed to hierarchical or rational, values. The presence of this skew was additionally supported by a Blau index of 0.64 (min: 0.48, max: 0.74, median: 0.65), confirming the distribution of points across the four value types was not uniform.

Changes in every item score were statistically significant, with the only exception being the climate domain. Changes measured for three climate items were significant or nearing significance, accounting for clustering by practice: “I have the opportunity to make full use of my knowledge and skills in caring for patients with mental health concerns” (p = .000); “I’m unsure of my role in caring for patients with behavioral, developmental, or emotional concerns” (p = .05); and “I am able to provide continuity of mental health care to my patients” (p = .000).

The lower an office’s baseline context score, the greater the degree of change the practice experienced, as demonstrated by significant negative associations between baseline context domain scores and change in domain scores (culture: r = −.724, p = .000; climate: r = −.178, p = .001; structures/processes: r = −.772, p < .000; and technologies: r = −.898, p = .000). We observed that the context changes that practices with lower baseline scores were able to achieve brought their post scores close to the starting scores of their higher-uptake peers (see Supplemental Digital Content 2,

Pearson’s correlation coefficients revealed moderate correlations between context domains, providing evidence of convergent internal construct validity. Linear regressions of implementation outcomes on context domains, followed by assessment of variance inflation factors, confirmed the presence of collinearity. These findings supported the calculation of total context scores for each practice as an average of the four domain scores at baseline and post.

An observation made during exploratory data analysis was a significant increase in interrater agreement (rWG ( J )) over the course of the learning collaborative. Average staff consensus on context domains increased an estimated 14 percentage points from baseline to postintervention.

Back to Top | Article Outline

Organizational Context for Mental Health: Hypothesis 1

Uptake was variable, with the 15 practices earning an average of three out of five stars (min: 1, max: 5). Average continuous star score was 62 (min: 53, max: 68, median: 65, SD: 5.46), with higher scores corresponding to a greater number of stars achieved. Results of simple linear regression with robust standard errors used to assess the relationship between continuous star score and organizational context are presented in Table 3. A practice’s baseline score going into the intervention, but not changes in context occurring over time, were found to relate to uptake. A coefficient of 0.536 (95% CI [0.112, 0.906]) suggests that a 10-point increase in total context score at baseline corresponds to a 5.36-point increase in continuous star score (p = .02). This increase is more than one full standard deviation on the continuous star scale and is evidence of an effect of organizational context for mental health care on uptake. This and significant coefficients for culture, structure/processes, and technologies support our hypothesis that practices with baseline attributes that prior research suggests would facilitate uptake would experience greater change (climate neared but did not reach significance).

Table 3

Table 3

Back to Top | Article Outline

Group and Developmental Values: Hypothesis 2

A coefficient of 0.200 (95% CI [0.037, 0.363]) for percent group/developmental suggests that for every additional 10 percentage points assigned to these values, a practice’s continuous star score increases by 2.00 points. This supports our hypothesis that practices with general cultures emphasizing all-staff participation and risk-taking would demonstrate greater uptake.

Back to Top | Article Outline

External or Fixed Characteristics: Hypothesis 3

Practices certified as PCMHs prior to enrollment achieved a continuous star score that was 4.92 points higher (95% CI [1.10, 8.74]) than their non-PCMH peers (p = .02), and practices where staff used EMR to monitor patient outcomes achieved a continuous star score than was 7.77 points higher (95% CI [3.24, 12.3]) than those without EMR (p = .006). Of the six practices that achieved only one or two out of five stars, none used EMR or were PCMH-certified. This supports our hypothesis that external or fixed practice characteristics, including PCMH certification and use of EMR to monitor patient outcomes, would support program uptake.

Back to Top | Article Outline


BMW was a complex intervention that helped primary care staff enhance their organizational contexts for health care integration. It was designed with the knowledge that empowering participants to understand their culture and climate for mental health and to engage in small cycles of change could support uptake of practices that primary care staff have historically found difficult. We found that culture—in addition to structures/processes and technologies that are the more common targets of complex interventions—was predictive of program uptake. Moreover, organizational context was amenable to change, particularly in practices with multiple barriers to overcome, as evidenced by lower context scores at baseline. External or fixed practice characteristics that were not targets of the BMW intervention (use of EMR and PCMH status) were also found to support uptake. Only baseline context (not change in context) was significant, suggesting that, although practices succeeded in making changes, it was their relative strengths and weaknesses going into the program that were the primary drivers of uptake. Taken together, findings point to the benefit of taking steps to understand and enhance organizational context in preparation for mental health service implementation.

Back to Top | Article Outline

Changes in Organizational Context

Although practices with lower total context scores to start had a harder time completing program activities and achieved fewer stars, the improvements they made lifted them close to where their peers began in terms of having a context supportive of mental health care (see Supplemental Digital Content 2, This may have a positive influence on their ability to implement similar activities in the future. That consensus on context scores increased over time is consistent with the complex intervention’s focus on all-staff participation and fostering cross-role consensus. The value of a learning collaborative like BMW is not just in the potential to improve a single metric, such as rates of mental health screening, but also in the opportunity to make more fundamental changes to how staff work together, supporting positive service and client outcomes more broadly and preparing staff to successfully engage in future QI initiatives.

Back to Top | Article Outline

Organizational Context for Mental Health

Of the “warmer” elements of organizational context (culture and climate), only culture was associated with variation in uptake. The mental health-specific measure of culture included mental health being positioned as a primary organizational goal and strategy and changes to promote mental health being supported and rewarded. Because change is difficult and involves growing pains, it is intuitive that practices where management value and reward mental health care would have an advantage. As Campbell (2009) wrote in Creating a Winning Organizational Culture, good leaders find ways to ensure values resonate with staff, linking “the values, norms, and philosophies of the organizational culture to those of the individual, to enact organizational change” (p. 341). All-staff participation was another cultural attribute measured. Its importance as a predictor of uptake points to the value of empowering front line staff who are passionate about mental health and who might not typically have an opportunity to play a leadership role.

Our measure of climate did not reach significance as a predictor of program uptake. This finding corroborates evidence that climate attributes such as chaos may fail to predict primary care intervention uptake even as the related construct of culture does (Dickinson et al., 2014). It may also be an artifact of the relatively low between-practice variability in climate domain scores. We question whether organizational climate might instead predict individual level outcomes such as staff attitudes and skill building, exerting a contextual effect on behavior change, a premise that we are testing in concurrent research using multilevel models.

The “colder” elements of structures/processes and technologies that we examined were found to relate to uptake, as hypothesized. These domains directly mapped to elements of the chronic care model, known determinants of readiness for mental health service delivery (AAP, 2010).

That baseline total context score, but not change in context, related to uptake is intuitive in that it takes time to make context changes that could in turn support ongoing program activities. Our findings suggest that if organizational context is addressed as part of a complex intervention to implement mental health in primary care, practices with lower context scores to start are unlikely to complete as many activities as their peers, although they may set themselves up for future success. An intervention designed to address organizational context prior to or in a phased approach with integration activities could provide an opportunity to test this premise.

Back to Top | Article Outline

Group and Developmental Values

Results supported our hypothesis that practices with a greater skew toward more group/developmental values would in turn experience greater uptake. As expected, these general cultural values related to uptake in a similar way as higher mental health-specific culture scores. Group-oriented values that foster sharing and integration of ideas and opinions across roles have been positively associated with participation and teamwork in primary care (Hann, Bower, Campbell, Marshall, & Reeves, 2007). Developmental-oriented cultures have in turn been associated with risk-taking and growth (Shortell et al., 1995).

Back to Top | Article Outline

External or Fixed Characteristics

We saw earlier that structures/processes and technologies that map to the chronic care model and represent the infrastructure for health care delivery related positively to program uptake and related clinician confidence-building. The chronic care model is the organizing framework for most medical homes, and it was therefore intuitive that PCMH certification should also be positively related to these changes. This relationship may also speak to practices’ ability to seek out efficiencies by positioning mental health integration as part of broader institutional goals.

Likewise, we expected EMR to support program uptake by enabling coordination and information sharing. Having an EMR in place is particularly helpful when it comes to screening, outcomes monitoring, and follow-up. Data can be collected and reviewed more efficiently, making it easier to gauge progress toward goals. It is also possible that having EMR in place from the start was a marker for past success with organization change.

Back to Top | Article Outline


There are several limitations to our study. Sample size for our analysis of the moderating effects of organizational context on practice-average change in uptake was small (n = 15), precluding our ability to assess the effects of organizational context and external or fixed characteristics together in an adjusted model. We entered into statistical analyses with a priori hypotheses and set data analysis plans to help guard against the “vibration of effects” to which studies with modest sample sizes are susceptible (Button et al., 2013). With regard to external validity, because our sample of practices was diverse, we have no reason to believe that experiences would have been drastically different in other Ohio practices with similar characteristics. Findings are general enough to provide a lens through which to identify potential facilitators and barriers to integration, although specific aspects may differ in states or regions with very different health care delivery systems, policies, cultures, and histories.

Back to Top | Article Outline

Practice Implications

The paucity of complex interventions that address organizational context in tandem with individual and system level determinants of behavior change may help explain why primary care staff have historically been resistant to mental health implementation. There is growing evidence that changes to organizational culture and climate can be brought about by complex interventions and that these changes can in turn improve youth mental health outcomes (Glisson, Hemmelgarn, Green, & Williams, 2013). Such studies have elucidated the role of context as a potential moderator and mediator of organization change. They show that, rather than passively being controlled for statistically or manipulated as part of study design, it may be actively framed as a target of health interventions, and that doing so may be instrumental to the uptake of new practices. Our findings contribute to this literature by suggesting that clinic and hospital administrators planning for mental health integration would be well advised to address culture and climate in tandem with processes and technologies that are the more typical focus. This might include an assessment of practice context for mental health, creative use of data feedback and dialogue, and opportunities for all-staff engagement in organization change.

  • Assessment of practice context might involve walkarounds, surveys, interviews and other forms of group storytelling and dialogue. Some surveys, such as the Office Inventory, are specific to mental health implementation in primary care, whereas others, such as applications of the CVF, take stock of more general practice attributes. A number of surveys are available free of charge through organizations such as the SAMHSA-HRSA Center for Integrated Health Solutions (
  • Data feedback and dialogue can support practices in identifying strengths and areas for improvement. Activities to “build the case” prior to program start and to facilitate intergroup dialogue and inquiry may be useful in practices where buy-in or participation is limited or where differing experiences or conflict across roles impedes working toward a common vision. These will require technologies for engagement that honor the unique experience that each staff member brings to the table (see, e.g., the engagement streams developed by Sandy Heierbacher and other members of the National Coalition for Dialogue and Deliberation,
  • All-staff participation in context change is essential and might involve convening staff across roles (from front desk to medical assistants to clinicians) in a common space to share ideas and develop consensus regarding what a culture of mental health would look like and what each person’s role will be in working toward that vision. Notable examples of holistic approaches to transforming the cultures of clinics and hospitals to be more supportive of staff and client mental health alike include Sandra Bloom’s Sanctuary Model ( and Angelica Thieriot’s Planetree (

Organizational behavior research shows that the most common reason health innovations are not implemented or disseminated is a failure to anticipate and address factors at each level that facilitate or create roadblocks to change. Although some external or fixed characteristics are permanent or will require change on behalf of government, insurers, and managed care entities to address, there are many strategies that providers can take to make their organizational contexts more conducive to mental health care. By putting change into context, we may help address the lag in incorporating mental health services into primary care, supporting the role of pediatric primary care as a gateway to preventive services, treatment, and recovery.

Back to Top | Article Outline


The authors would like to thank the entire Building Mental Wellness (BMW) Wave III implementation team, including Sean O’Hanlon, Heather Maciejewski, Cathy Jaworski, and John Duby, for their collaboration and insights. They would also like to acknowledge the helpful assistance of Jill Marsteller, PhD, MPP, the first author’s dissertation committee chair, and valued poster feedback from the Organizational Theory in Health Care Association.

Back to Top | Article Outline


American Academy of Pediatrics. (2010). Mental health practice readiness inventory. Pediatrics, 125(Suppl.), S129–S132.
Becker E. R., & Roblin D. W. (2008). Translating primary care practice climate into patient activation: The role of patient trust in physician. Medical Care, 46(8), 795–805.
Bloom S. L., & Farragher B. (2013). Restoring sanctuary: A new operating system for trauma-informed systems of care. New York, NY: Oxford University Press.
Brown J., & Wissow L. (2012). Rethinking the mental health treatment skills of primary care staff: A framework for training and research. Administration and Policy in Mental Health, 39(6), 489–502.
Button K. S., Ioannidis J. P., Mokrysz C., Nosek B. A., Flint J., Robinson E. S., & Munafò M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.
Campbell R. J. (2009). Creating a winning organizational culture. The Health Care Manager, 28(4), 328–343.
Carlfjord S., Andersson A., Nilsen P., Bendtsen P., & Lindberg M. (2010). The importance of organizational climate and implementation strategy at the introduction of a new working tool in primary health care. Journal of Evaluation in Clinical Practice, 16(6), 1326–1332.
Chang E. T., Rose D. E., Yano E. M., Wells K. B., Metzger M. E., Post E. P., … Rubenstein L. V. (2013). Determinants of readiness for primary care-mental health integration (PC-MHI) in the VA Health Care System. Journal of General Internal Medicine, 28(3), 353–362.
Dickinson L. M., Dickinson W. P., Nutting P. A., Fisher L., Harbrecht M., Crabtree B. F., … West D. R. (2014). Practice context affects efforts to improve diabetes care for primary care patients: A pragmatic cluster randomized trial. Journal of General Internal Medicine, 30(4), 476–482.
Dowswell G., Harrison S., & Wright J. (2001). Clinical guidelines: attitudes, information processes and culture in English primary care. International Journal of Health Planning and Management, 16(2), 107–124.
Glisson C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253.
Glisson C., Hemmelgarn A., Green P., & Williams N. J. (2013). Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child & Adolescent Psychiatry, 52(5), 493–500.
Glisson C., & James L. R. (2002). The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23, 767–794.
Glisson C., Landsverk J., Schoenwald S. K., Kelleher K., Hoagwood K. E., Mayberg S., … The Research Network on Youth Mental Health. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health, 35(1–2), 98–113.
Goldberg D. G., & Mick S. S. (2010). Medical home infrastructure: Effect of the environment and practice characteristics on adoption in Virginia. Medical Care Research and Review, 67(4), 431–449.
Green C., Hampton E., Ward M. J., Shao H., & Bostwick S. (2014). The current and ideal state of mental health training: Pediatric program director perspectives. Academic Pediatrics, 14(5), 526–532.
Grumbach K., & Bodenheimer T. (2004). Can health care teams improve primary care practice? JAMA, 291(10), 1246–1251.
Guerrero E. G., Heslin K. C., Chang E., Fenwick K., & Yano E. (2015). Organizational correlates of implementation of colocation of mental health and primary care in the Veterans Health Administration. Administration and Policy in Mental Health, 42(4), 420–428.
Halfon N., Houtrow A., Larson K., & Newacheck P. W. (2012). The changing landscape of disability in childhood. Future Child, 22(1), 13–42.
Hann M., Bower P., Campbell S., Marshall M., & Reeves D. (2007). The association between culture, climate and quality of care in primary health care teams. Family Practice, 24(4), 323–329.
James L. R., Demaree R. G., & Wolf G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69(1), 85–98.
James L. R., & Sells S. B. (1981). Psychological climate: Theoretical perspectives and empirical research. In Toward a psychology of situations: An interactional perspective (pp. 275–295). Hillsdale, NJ: Erlbaum.
Kaissi A., Kralewski J., Curoe A., Dowd B., & Silversmith J. (2004). How does the culture of medical group practices influence the types of programs used to assure quality of care? Health Care Management Review, 29(2), 129–138.
King M. A. (2016, February). The role of organizational context in the implementation of mental health services in pediatric primary care: Concepts, mechanisms, and intervention. Baltimore, MD: Johns Hopkins University.
LeBreton J. M., & Senter J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852.
Mauch D., Kautz C., & Smith S. (2008). Reimbursement of mental health services in primary care settings (DHHS Publication No. (SMA) 08-4324, pp. 1–48). Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. Retrieved from
Nembhard I. M., Alexander J. A., Hoff T. J., & Ramanujam R. (2009). Why does the quality of health care continue to lag? Insights from management research. Academy of Management Perspectives, 23(1), 24–42.
O’Donnell A. N., Williams M., & Kilbourne A. M. (2013). Overcoming roadblocks: Current and emerging reimbursement strategies for integrated mental health services in primary care. Journal of General Internal Medicine, 28(12), 1667–1672.
Ohman-Strickland P. A., John Orzano A., Nutting P. A., Perry Dickinson W., Scott-Cawiezell J., Hahn K., … Crabtree B. F. (2007). Measuring organizational attributes of primary care practices: Development of a new instrument. Health Services Research, 42(3 Pt. 1), 1257–1273.
Okhuysen G. A., & Bechky B. A. (2009). Coordination in organizations: An integrative perspective. Academy of Management Annals, 3(1), 463–502.
Ostroff C., Kinicki A. J., & Muhammad R. S. (2013). Organizational culture and climate. In Weiner I. B., Schmitt N. W., Highhouse S. (Eds.), Handbook of psychology: Industrial and organizational psychology (2nd ed. Vol. 12, pp. 643–676). Hoboken, NJ: John Wiley & Sons, Inc.
Penfold R. B., Kelleher K. J., Horwitz S. M., & Stein R. E. (2005, May). Pediatric survey of fellows: American Academy of Pediatrics Division of Health Services Research. In Pediatric Academic Societies Annual Meeting. Retrieved from
Quinn R. E., & Kimberly J. R. (Eds.). (1984). Paradox, planning, and perseverance: Guidelines for managerial practice. In Managing Organization Transitions (pp. 295–313). Homewood, IL: Dow Jones-Irwin.
Rubin Stiffman A., Striley C., Horvath V. E., Hadley-Ives E., Polgar M., Elze D., & Pescarino R. (2001). Organizational context and provider perception as determinants of mental health service use. The Journal of Behavioral Health Services and Research, 28(2), 188–204.
Schein E. H. (2010). Organizational culture and leadership (4th ed.). San Francisco, CA: Jossey-Bass.
Schneider B., Ehrhart M. G., & Macey W. H. (2013). Organizational climate and culture. Annual Review of Psychology, 64, 361–388.
Shortell S. M., O’Brien J. L., Carman J. M., Foster R. W., Hughes E. F., Boerstler H., & O’Connor E. J. (1995). Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Services Research, 30(2), 377–401.
Simonian S. J. (2006). Screening and identification in pediatric primary care. Behavior Modification, 30(1), 114–131.
StataCorp. (2011). Stata Statistical Software: Release 12. College Station, TX: StataCorp LP.
Stein R. E., Storfer-Isser A., Kerker B. D., Garner A., Szilagyi M., Hoagwood K. E., … McCue Horwitz S. (2016). Beyond ADHD: How well are we doing? Academic Pediatrics, 16(2), 115–121.
Tice J. A., Ollendorf D. A., Reed S. J., Shore K. K., Weissberg J., & Pearson S. D. (2015). Integrating behavioral health into primary care. Institute for Clinical and Economic Review. Retrieved from
Van Patter Gale B., & Schaffer M. A. (2009). Organizational readiness for evidence-based practice. The Journal of Nursing Administration, 39(2), 91–97.
Wagner E. H., Austin B. T., & Von Korff M. (1996). Organizing care for patients with chronic illness. Milbank Quarterly, 74(4), 511–544.

implementation science; mental health; organization change; organizational context; pediatric primary care

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved