As the largest hepatitis C virus (HCV) care provider in the United States, the Department of Veterans Affairs (VA) has invested in clinical, professional, and other resources to disseminate and implement evidence-based, high-quality HCV care.1 In 2015, direct-acting antiviral medications (DAAs), with cure rates >90%, revolutionized the field and rapidly replaced older, less efficacious therapies as the new standard of HCV care.2 VA was the first national health care system to establish a goal of evaluating all enrollees with HCV for potential treatment with these medications.
To improve HCV identification and management and disseminate evidence-based practices and innovations in HCV care, VA formed the national Hepatitis C Innovation Team (HIT) quality improvement (QI) collaborative.3,4 The HIT Collaborative’s primary aim was to increase the uptake of the new DAAs for HCV through efforts by a leadership team of HCV and QI experts supporting regional HITs and local sites in the selection of strategies to improve HCV care.5
Our evaluation of the HIT Collaborative conceptualized the site-level strategies being used to increase HCV treatment as implementation strategies, or “methods or techniques used to enhance the adoption, implementation, and sustainability of a clinical program or practice.”6,7 In 2015 the Expert Recommendations for Implementing Change (ERIC) project compiled a taxonomy of 73 discrete implementation strategies.8–10 We used the ERIC taxonomy to develop a survey to assess which implementation strategies VA sites deployed to improve HCV treatment. We previously used correlational statistical methods and found that sites using more strategies had more Veterans initiate HCV treatment.6 While that work was an important step, implementation strategies are typically conducted in combination with one another, and there is considerable debate in the field about multifaceted versus single component (discrete) strategies.11
Configurational Comparative Methods (CCMs) are a class of research methods that remain relatively new to implementation science.12 Unlike commonly-used regression analytic methods that quantify the strength of a relationship between variables, CCMs draw upon Boolean algebra and set theory to answer questions such as “What condition or combinations of conditions are consistently present when the outcome is present?”13,14 CCMs, through Boolean minimization and optimization, yield solutions are known as a “minimal theory”—a unique combination of nonredundant conditions whose joint presence links directly to an outcome of interest.15,16
CCMs operate from a theoretical framework that is distinct from other quantitative approaches. Correlation-based and regression-based methods, for example, draw upon an “interventionist” model, assessing the incremental effect of a unit difference in independent variable X on the values of dependent variable Y, controlling for all other variables. CCMs, by contrast, use Boolean algebra (rather than linear algebra) and rely on a “regularity theory” of causality, which states that A is a cause of B if and only if A is part of the set of conditions AX that, ceteris paribus, is regularly followed by B.17,18 The regularity framework is especially well-suited for the analysis of causal complexity (ie, the joint presence of conditions) and equifinality (ie, multiple solution paths to the same outcome). Within this framework, the specific regularity theory employed by CCMs is the “INUS” theory of causation formulated by Mackie, which states that causes are difference-makers of their effects, and that causal structures do not contain redundant elements.19 Causal inference with CCMs thus requires researchers to proceed thoughtfully and carefully (and not mechanically) when interpreting results, taking into account that configuration-outcome connections, while not inherently causal, become “difference-makers” only when they meet consistency and coverage requirements; reliably distinguish a set of cases with an outcome from another set of cases without that outcome; are nonredundant; and remain consistent with logic, theory, and prior knowledge.
CCMs, which include Qualitative Comparative Analysis (QCA) and more recently Coincidence Analysis (CNA), have been applied in political science, sociology and education for decades, and are starting to make inroads into health services research and implementation science.20 CCMs, for example, were prominently used to identify conditions aligned with successful implementation in a recent Cochrane Review of school-based interventions for asthma self-management.21
The aim of this evaluation was to use CCMs to identify solution pathways, or specific combinations of implementation strategies, that distinguished sites with higher HCV versus lower HCV treatment rates. Ultimately, the goal of this work was to use empirical data to inform high-yield implementation strategy selection and increase the efficiency and effectiveness of VA’s HCV implementation efforts.
Implementation Strategy Data
These data come from the first year (FY15) of a mixed-methods evaluation of a national QI program—the HCV Innovation Team (HIT) Collaborative. This program evaluation was deemed to be a QI program by the VA Pittsburgh Healthcare System IRB and approved as such by VA HIV, Hepatitis and Related Conditions Program Office. All participation in the evaluation was voluntary.
In brief, we emailed a survey to a key informant from each VA medical center (site) treating HCV (n=130) to understand the implementation strategies being employed by the site to implement HCV treatment in the prior fiscal year. The survey has been previously published and included a list of the 73 possible implementation strategies, classified into previously defined clusters (eg, financial, infrastructure) using examples that were applicable to HCV, to improve the readability and understandability of the survey.6,7 Participants were asked whether their site was using the strategy (yes/no) and were also asked to provide demographic information.
The primary outcome was the number of Veterans initiating HCV treatment per year at a site. Treatment data were derived from administrative data through VA’s Corporate Data Warehouse. A patient was counted as starting treatment if they had a DAA prescription in the electronic medical record. Once treatment data were obtained for each site, descriptive statistics were used to characterize treatment initiations. The median number of treatment starts for the sites was calculated, and sites in the upper 2 quartiles were characterized as “higher treating” while sites in the lower 2 quartiles were characterized as “lower treating.” We also conducted a sensitivity-type analysis to assess how well the CCM model performed when the outcome was site viremic volume.
Each one of the 73 implementation strategies from the survey served as a dichotomous factor where respondents could select the strategies as either implemented or not implemented in the past fiscal year. Respondent and site characteristics were also included as dichotomous factors: site complexity (high vs. low), specialty (gastroenterology vs. other), provider degree (PharmD vs. other), and tenure (0–9 vs. 10+ y). Within factors, conditions are thus the specific values a factor takes on.
The analyses with CCMs were conducted using the R packages “cna” and “QCApro” as well as the software applications R and R Studio.22,23
The original dataset contained 73 dichotomous factors, with no compelling a priori theoretical reason to select certain implementation strategies over others for inclusion in model development: each strategy had a plausible connection to the outcome. To achieve data reduction, we applied the “minimally sufficient conditions” function in cna to look across all 73 factors and all 80 cases at once and examined the Boolean output to identify strategy configurations with the strongest apparent connections to the outcome using the process described below. We then used that configuration-level information to guide the selection of a smaller subset of factors to include in the model iteration.
This phase of the analysis involving data reduction and initial factor selection required intensive computation. We considered all 1-strategy, 2-strategy, 3-strategy, and 4-strategy configurations across the 73 dichotomous factors that were instantiated within the dataset and met the cutoff threshold (described below); there were over 1 million theoretically possible combinations. Computational limitations precluded examining ≥5 strategy combinations with our 73-factor dataset.
We then generated a “condition table” to list and organize the CNA output. In a condition table, rows represent configurations of conditions that meet a specified consistency level (percentage of cases with outcome condition covered by solution vs. all cases covered by configuration), while column variables include outcome, condition, consistency, coverage (percentage of cases with outcome condition covered by solution vs. all cases with outcome condition) and complexity (number of discrete conditions in a configuration). We began running these analyses by setting the consistency level to 100% and the coverage level to 25% (to avoid overfitting individual cases). If we did not find any configurations that met our dual-threshold (ie, consistency score of 1.0 and a coverage score of ≥0.25), we then iteratively lowered the specified consistency level by 0.05 (eg, from 1.0 to 0.95, etc.) and repeat the process of generating a new condition table and examining it again, continuing this process until at a given consistency threshold we found configurations that met the consistency threshold as well as the coverage threshold of ≥0.25.
Next, we sorted the condition table by complexity and coverage and identified the configurations with the highest coverage scores for 1-object configurations, 2-object configurations, 3-object configurations, and 4-object configurations. We began with 1-object configurations, looking to see if the 1-object configuration with the maximum coverage score met all 4 of the following criteria: configuration met the consistency threshold; configuration met the coverage threshold; coverage score for that configuration uniquely distinguished it from all other configurations sharing the same complexity level (ie, there was separation between the configuration with the top coverage score and the next-highest coverage score); and the configuration aligned with logic as well as our own theoretical and clinical knowledge.
We then proceeded to examine the condition table for 2-object configurations, 3-object configurations, and finally 4-object configurations, starting with the smallest configurations and working upwards to ensure that redundant conditions could not appear. Using this approach, we inductively analyzed the entire dataset and reduced it to a smaller subset of candidate factors suitable for model iteration.
To remain consistent within our overall CCMs approach, we used a data reduction strategy that operated entirely within a “regularity theory” framework, rather than applying other well-established methods such as Blinder-Oaxaca decomposition and conditional inference trees, because those methods use linear algebra (as opposed to Boolean algebra) and operate within a fundamentally different “interventionist” framework).24–27
We next proceeded with model development. We introduced the subset of identified factors iteratively into solutions using the model-building functions within the “cna” package. If the model with the initial subset of factors did not meet an overall coverage score of ≥0.80, we then identified factors that appeared across multiple cases unexplained by the initial model. We then systematically searched across the entire condition table to identify all configurations with these specific factors, again using a bottom-up approach by first identifying prioritized configurations with lower complexity levels and higher coverage scores. Using this strategy, we identified a second subset of candidate factors to introduce into our model and retained those factors if the performance of the overall model improved in terms of consistency and coverage. We selected a final model based on overall model coverage (ie, ≥0.80), high consistency (ie, as close to 1.0 as possible) and clarity (ie, no model ambiguity).28
To reduce dimensionality, we created meta-factors for each of the solution paths and then calculated overall model consistency and coverage using both the cna and QCApro packages in R.
In an additional analysis, once solution paths were identified with CCMs, each of the strategies in the final model was compared with the remaining strategies to discover if the solution strategies differed by any ERIC feature (cluster theme, importance rating or feasibility rating) using the definitions per Waltz et al.9
We then examined solution versus nonsolution site differences including site complexity (classified as levels 1a, 1b, 1c, 2, or 3, with level 1a being the most complex and level 3 being the least complex, based on variables such as volume, patient risk score, complex, clinical programs, research dollars, and teaching programs), specialty location (gastroenterology, infectious disease, other), HCV prevalence, HCV screening rates, respondent degree, and respondent years with the VA with t tests and χ2 tests.29
Eighty of the 130 VA medical centers that were surveyed provided responses (62%). In brief, sites represented a range of complexity ratings (66% higher, 34% lower), and specialties (41% gastroenterology/hepatology, 21% infectious disease, 16% pharmacy, 10% primary care, and 11% other). Respondents included clinical pharmacists (44%), physicians and physician assistants (20%), nurse practitioners (16%), and other staff (20%).
The 40 sites characterized as “higher-treating” had ≥197 treatment starts and represented the upper 2 quartiles, while the 40 sites in the lower 2 quartiles with <197 treatment starts were characterized as “lower-treating.” Respondents reported using a mean of 25±14 strategies (range: 1–59). Only 3 strategies were not implemented by any site (“provide financial disincentives,” “alter patient fees,” and “use capitated payments”); these 3 were excluded from further analysis.
Solution Configurations Associated With Higher Hepatitis C Virus Treatment
We identified 5 distinct “solution paths,” comprised of 10 discrete implementation strategies that collectively explained 80% (32/40) of higher HCV treatment starts at a consistency level of 100% (40/40), with no model ambiguity. The presence of any one of these 5 paths was sufficient to produce higher treatment starts:
- Path 1: (S24: Local technical assistance) OR
- Path 2: (S34: Facilitate the formation of groups of providers and foster a collaborative learning environment AND S45: Recruit, designate, train leaders) OR
- Path 3: (S22: Develop resource sharing agreements AND S61: Develop a formal implementation blueprint) OR
- Path 4: (S56: Visit other sites outside your medical center to try to learn from their experiences AND S71: Intervene with patients/consumers to promote uptake and adherence to HCV treatment) OR
- Path 5: (S18: Create new clinical teams AND S47: Share the knowledge gained from QI efforts with other sites AND S70: Engage in efforts to prepare patients to be active participants in HCV care).
No single strategy was necessary for higher treatment, as there were 5 different paths to the outcome featuring 10 different implementation strategies. One strategy—“S24: Local technical assistance”—was sufficient in itself, as the presence of S24 was always accompanied by the presence of the outcome. The 4 other solution paths were conjunctions of 2–3 strategies, thus individual implementation strategies in these other 4 paths were all “INUS” conditions: they were individually each Insufficient but Necessary parts of terms (or solution paths) that were themselves Unnecessary but Sufficient for the outcome. All 5 terms in the final solution each had a coverage score of 0.25–0.40 (ie, covered ≥10 different “high-treatment” cases) and explained ≥1 unique case. No site or respondent conditions explained more “high-treatment” cases than final solution paths.
Figure 1 shows a matrix display of all 80 sites, the 5 solution paths and the 10 implementation strategies. Each row represents a case or Veterans Affairs medical center (80 sites). Shaded cells depict presence of conditions, such as “higher HCV treatment starts” or a specific implementation strategy. Black dots represent sites covered by the solution, and in the final column represent individual sites covered by the solution (at least 1 of the 5 paths).
None of the 40 lower treatment sites had any of the 5 solution paths. Among the 40 higher treatment sites, 32 were solution sites (ie, had any of the 5 solution paths resulting in 19 configurations). Among the 32 higher treatment solution sites, 1 site had all 5 solution paths, 4 had 4, 4 had 3, 5 had 2, and 5 had 1. Table 1 illustrates the mean number of treatment starts at sites with different solution path configurations and combinations of paths. The mean number of HCV treatment starts did not differ between the 19 solution path configurations (P=0.586) or the number of solutions paths (1–5) per site (P=0.494).
Table 2 shows the ERIC features of each of the 10 strategies in the solutions, including cluster name, and importance and feasibility ratings. The 10 implementation strategies represent 6 of the 9 ERIC clusters (engage patients, provide interactive assistance, develop stakeholder relationships, train and educate stakeholders, support clinicians, and use evaluative and iterative strategies). The 3 clusters not represented were: change infrastructure, utilize financial strategies, and adapt and tailor to context.
High-treatment Site Characteristics
Solution sites (ie, sites with higher-treatment rates) had significantly higher complexity ratings (97% vs. 56% were level 1, P<0.001) and HCV prevalence (9.2% vs. 7.8%, P=0.005) than nonsolution sites. Respondents from solution sites did not significantly differ from those representing nonsolution sites in terms of specialty (P=0.773), degree (P=0.642), or years with the VA (P=0.409). Solution sites used, on average, more strategies than nonsolution sites (36 vs. 18, P<0.001).
We conducted an additional analysis to assess how well the CCM model performed when the dataset was limited to the interquartile range for viremic volume, the total number of Veterans potentially eligible for treatment at the start of FY15 at each site. The 40 sites that fell within the interquartile range for viremic volume (636, 1744) were almost evenly distributed between the “higher treatment” (n=19) and “lower treatment” (n=21) groups, indicating variation in viremic volume across the 2 groups. The 5 CCM solution pathways identified for the overall dataset also performed well in this analysis, with a consistency level of 100% (14/14) and a coverage level of 74% (14/19).
This evaluation, to our knowledge, is unique in its application of CCMs to examine combinations of ERIC-defined implementation strategies. We present an application of CCMs to assess a large dataset comprised of 80 cases (sites) and 73 factors (implementation strategies) and identified strategy configurations directly linked to higher treatment starts. In this example, we identified 5 distinct solution pathways that distinguished higher treating from lower treating sites across a national sample with 100% consistency.
CCMs provide a mathematical, cross-case approach that can serve to complement existing quantitative and qualitative methods. We used CCMs to refine our previously published findings using traditional regression analytic methods and address the call to incorporate causal theory in research questions and methods.30,31 Using traditional methods, we found in an earlier publication that 28 of the 73 strategies were individually significantly and positively associated with HCV treatment starts and together explained a low percentage of the variance of site-level HCV treatment starts.6 However, recommending 28 strategies to sites is not practical, efficient, or cost-effective. Using CCMs, we identified 10 difference-making strategies with 1–3 strategies in each pathway; 8 of the 10 CCM-identified strategies had been previously found to be individually associated with higher HCV treatment starts. The strategies identified by CCMs were directed across multiple levels (eg, intrapersonal, interpersonal, and organizational), and to different audiences (patient, provider, team, and leaders). The 5 solution paths represented 4 larger themes: (1) developing implementation plans; (2) utilizing local technical assistance and expertise; (3) facilitating knowledge exchange by participating in a collaborative, engaging leaders, creating new clinical teams, and learning from other sites’ experiences with QI efforts; and (4) engaging patients as active participants in initiating and maintaining connection to care.
Overall, our findings are consistent with implementation science literature reporting that strategies often do not operate in isolation, that their value may be maximized when they operate synergistically, and that their impact can sometimes be conditional on one another.32 In the context of our findings, CCMs specified that one strategy (local technical assistance) was sufficient for the outcome and the other 9 by themselves were neither necessary nor sufficient for the outcome. While it would appear that “technical assistance” could be a “single-component” or stand-alone strategy, technical assistance itself may encompass many elements and context should be considered in interpreting this result. VA provided several structural and universal changes at a national level (eg, placing medications on the formulary) such that local technical assistance may not be sufficient in a different system where these structural changes are not in place. Similarly, financial strategies did not appear to be important in this sample, but individual sites did not need to use financial strategies because the umbrella structure of VA had already applied them across the system.
While scholars recognize the need for tools to help guide strategy selection for real-world application, discerning how strategies work in combination or in parallel with one another is not well understood.33,34 These analyses would appear to be most useful in the context of an active implementation effort. To apply CCMs, there must be sites that have used a variety of strategies and combinations of strategies and a clinical metric to assess their success. Configuration-outcome connections are not inherently causal and the “difference-making” strategy configurations we identified in our findings had to satisfy multiple criteria, including being able to meet consistency and coverage requirements; reliably distinguish a set of cases with an outcome from another set of cases without that outcome; be nonredundant; and remain consistent with logic, theory, and prior knowledge. While we can conclude that there is evidence for these strategy configurations, experimental work is ultimately needed to determine their relative effectiveness.
This evaluation had several limitations. Our assessment of implementation strategies was based on self-report from one respondent per site. Three implementation strategies were dropped from the analysis because they were not used by any of the 80 facilities and thus excluded from consideration. Also, we cannot discern the timing or sequencing of implementation strategies in this analysis. Furthermore, the HCV treatment outcome and implementation strategies were dichotomized for these analyses, rather than including more nuanced assessments that account for the intensity of the strategy or continuous outcomes. Such approaches require a more detailed understanding of the strategies employed and different analyses and are areas of future study.
VA is a high resource setting which provided a national QI collaborative, and this may limit the generalizability of our results. HCV treatment is a relatively low complexity intervention, which provided a clear outcome measure but may limit the generalizability of these findings to other evidence-based practices and innovations with higher complexity.
CCMs appear to be particularly well-suited to assess how implementation strategies work together in medical care settings. Using a CCMs approach, we developed a model identifying specific combinations of implementation strategies that were sufficient for producing high HCV treatment starts in 80% of cases with 100% consistency. Our findings complement previous results that found HCV treatment starts were significantly associated with a set of 28 individual strategies; these 2 distinct analytic methods together point to a smaller set of difference-making strategies that work together in specific combinations. Future work will evaluate subsequent years of the HIT Collaborative using CCMs and other methods, allowing us to further understand how implementation strategies work in practice to improve health care delivery.
The authors express their indebtedness to Alrik Thiem and Michael Baumgartner, two of the world’s leading methodologists in CCMs. While they were not involved in any way with this study, our work builds directly on their groundbreaking advances in configurational analysis, which include new approaches to conducting Qualitative Comparative Analysis as well as the development of Coincidence Analysis as an entirely new method within the larger CCMs family.
1. Belperio PS, Chartier M, Gonzalez RI, et al. Hepatitis C care in the Department of Veterans Affairs: building a foundation for success. Infect Dis Clin North Am. 2018;32:281–292.
2. Backus LI, Belperio PS, Shahoumian TA, et al. Real‐world effectiveness of ledipasvir/sofosbuvir in 4365 treatment‐naive, genotype 1 hepatitis C‐infected patients. Hepatology. 2016;64:405–414.
3. Belperio PS, Chartier M, Ross DB, et al. Curing hepatitis C virus infection: best practices from the US Department of Veterans Affairs. Ann Intern Med. 2017;167:499–504.
4. Park A, Gonzalez R, Chartier M, et al. Screening and treating hepatitis C in the VA: achieving excellence using lean and system redesign. Fed Pract. 2018;35:24–29.
5. Hagg HW, Workman-Germann J, Flanagan M, et alHenriksen K, Battles JB, Keyes MA. Implementation of systems redesign: approaches to spread and sustain adoption. Advances in Patient Safety: New Directions and Alternative Approaches (Vol 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Research and Quality (US); 2008. Available at: https://www.ncbi.nlm.nih.gov/books/NBK43727/
6. Rogal SS, Yakovchenko V, Waltz TJ, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implement Sci. 2017;12:60.
7. Proctor EK, Powell BJ, McMillen JC. Implementation strategies
: recommendations for specifying and reporting. Implement Sci. 2013;8:139.
8. Waltz TJ, Powell BJ, Chinman MJ, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implement Sci. 2014;9:39.
9. Waltz TJ, Powell BJ, Matthieu MM, et al. Use of concept mapping to characterize relationships among implementation strategies
and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:1.
10. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies
: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.
11. Weiner BJ, Lewis MA, Clauser SB, et al. In search of synergy: strategies for combining interventions at multiple levels. J Natl Cancer Inst Monogr. 2012;2012:34–41.
12. Miech EJ, Damschroder L. Qualitative comparative analysis and implementation research: an introduction. Veterans Administration Health Services Research & Development National CyberSeminar; 2016.
13. Thiem A. Conducting configurational comparative research with qualitative comparative analysis: a hands-on tutorial for applied evaluation
scholars and practitioners. Am J Eval. 2017;38:420–433.
14. Baumgartner M, Epple R. A coincidence analysis
of a causal chain: the Swiss minaret vote. Sociol Methods Res. 2014;43:280–312.
15. Baumgartner M. Parsimony and causality. Qual Quant. 2015;49:839–856.
16. Cragun D, Pal T, Vadaparampil ST, et al. Qualitative comparative analysis: a hybrid method for identifying factors associated with program effectiveness. J Mix Methods Res. 2016;10:251–272.
17. Baumgartner M. Regularity theories reassessed. Philosophia (Mendoza). 2008;36:327–354.
18. Rohlfing I, Zuber CI. Check your truth conditions! Clarifying the relationship between theories of causation and social science methods for causal inference. Sociol Methods Res. 2019:0049124119826156.
19. Mackie JL. The Cement of the Universe: A Study of Causation. Oxford, UK: Clarendon Press; 1974.
20. Palinkas LA, Mendon SJ, Hamilton AB. Innovations in mixed methods evaluations. Annual Rev Public Health. 2019;40:423–.
21. Harris K, Kneale D, Lasserson TJ, et al. School‐based self‐management interventions for asthma in children and adolescents: a mixed methods systematic review. Cochrane Database Syst Rev. 2019;1:CD011651.
22. Ambuehl M, Baumgartner M. cna: causal modeling with coincidence analysis
. R package version 2.1.1; 2018.
23. Thiem A. QCApro: professional functionality for performing and evaluating qualitative comparative analysis. R package version 1.1-1; 2016.
24. Jann B. The Blinder-Oaxaca decomposition for linear regression models. Stata J. 2008;8:453–479.
25. Sharaf MF, Rashad AS. Regional inequalities in child malnutrition in Egypt, Jordan, and Yemen: a Blinder-Oaxaca decomposition analysis. Health Econ Rev. 2016;6:23.
26. Hothorn T, Hornik K, Zeileis A. Unbiased recursive partitioning: a conditional inference framework. J Comput Graph Stat. 2006;15:651–674.
27. Ferré Á, Poca MA, de la Calzada MD, et al. A conditional inference tree model for predicting sleep-related breathing disorders in patients with Chiari malformation type 1: description and external validation. J Clin Sleep Med. 2019;15:89–99.
28. Baumgartner M, Thiem A. Model ambiguities in configurational comparative research. Sociol Methods Res. 2017;46:954–987.
29. Veterans Health Administration. VHA facility complexity model; 2015. Available at: http://opes.vssc.med.va.gov/FacilityComplexityLevels/Pages/default.aspx
. Accessed December 12, 2016.
30. Chambers DA, Vinson CA, Norton WE. Advancing the Science of Implementation Across the Cancer Continuum. New York, NY: Oxford University Press; 2018.
31. Williams NJ, Beidas RS. Annual research review: the state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field. J Child Psychol Psychiatry. 2019;60:430–450.
32. Proctor E, Hooley C, Morse A, et al. Intermediary/purveyor organizations for evidence-based interventions in the US child mental health: characteristics and implementation strategies
. Implement Sci. 2019;14:3.
33. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21:1055–1080.
34. Kirchner JE, Waltz TJ, Powell BJ, et alBrownson RC, Colditz GA, Proctor EK. Implementation strategies
. Dissemination and Implementation Research in Health: Translating Science to Practice, 2nd ed. New York, NY: Oxford University Press; 2017:245–266.