Research evidence use is a key component of evidence-based decision making (EBDM) in public health. EBDM involves the use of evidence of disease burden to identify and prioritize issues and to select programs and policies previously shown to improve population health. EBDM also involves assessment of community strengths and needs, incorporation of community preferences, implementation of selected evidence-based programs and policies, sound evaluation, and use of evaluation findings to improve implementation.1,2 With funders increasingly requiring use of evidence-based policies and programs (EBPPs) shown to improve population health, and evidence increasingly available, public health practitioners are expected to use evidence in grant applications, selection and justification of strategies, priority setting, implementation, and evaluation.
In previous cross-sectional studies with public health practitioner decision makers, several organizational supports facilitated evidence use for public health decision making if present and, if lacking, served as barriers. These included the following: organizational culture and climate; access to evidence; on-the-job opportunities to build skills in acquiring, assessing quality of, and interpreting evidence; leadership; clear organizational processes for evidence use, such as performance management systems; provision of time to gather and use evidence; positive attitudes of managers; manager sponsorship of the project; opportunity to generate locally relevant evidence; and monitoring of internal decision-making processes.3–10 Studies in mental health and human services organizations had similar findings.11,12 In one of the few longitudinal studies, Dobbins and colleagues13 found that tailored targeted research evidence messaging increased the number of evidence-informed strategies used by health departments.
Leadership support and access to research evidence may be essential to promote and ensure research evidence use.3–8,14–16 The presence of supportive organizational climates and cultures was also identified by interview participants as key to evidence use.3,17,18 In addition to positive manager attitudes, leaders can develop policies and set criteria to monitor evidence use processes, provide staff time for evidence use, and ensure staff receive training in use of evidence.5 Leaders can also facilitate needed organizational access to evidence.4,10,19,20 Many public health agencies lack access to full-text journal articles.16,19 Several reviews noted limited availability of relevant, clear, timely, and reliable research evidence.4,8,10
The purpose of this study was to identify patterns of organizational supports and associations with research evidence use for job tasks among public health practitioners working in chronic disease prevention and health promotion in 12 US state health departments and partnering organizations. While earlier cross-sectional studies measured organizational supports and tested relationships with evidence use, as described earlier, the present study is one of the first to longitudinally test relationships of organizational supports with research evidence use. Latent class analysis (LCA), a model-based approach, is a useful tool to identify groups with homogeneous response patterns in a heterogeneous population with observed variables.21 The latent class model can identify categorical latent variables from a complex array of observed categorical data, comparable with factor analysis.
The present study was part of a larger 12-state study.22–24 The Washington University in St Louis institutional review board approved the study. Organizational context and resources are hypothesized to facilitate staff adoption of EBDM processes, including research evidence use.22,25
Six state public health department chronic disease prevention units were randomly selected and invited to participate, and 6 state health department chronic disease units were selected from states closest in state population to reduce bias from differences in state chronic disease funding. Via e-mail recruitment, the study team invited all professional staff members in the 12 state health department chronic disease units and a purposive sample of the staff in partnering organizations to participate in a baseline survey in 2014 and a second survey in 2016. Partners were included because they are key to implementation of EBPPs and to increase the sample size. Chronic disease directors identified key organizations and staff their programs partner with for chronic disease prevention, including local and district health departments; universities; health care facilities; and coalitions and voluntary and community-based health organizations (see Supplementary Digital Content, available at http://links.lww.com/JPHMP/A480). State enrollment and survey timing were staggered to accommodate study team schedules. Follow-up via e-mail and phone increased the postsurvey response to 70.5% of baseline; a total of 909 participants completed both surveys. A small portion of survey participants (16.2%) attended in-state EBDM training shortly after the initial survey, which included encouragement and training on how to use research evidence.
Survey development was based on a literature review,26 the study team's prior research, 5 rounds of expert input, cognitive response interviews with 11 former chronic disease directors, and test-retest reliability with 75 state health department staff members working in chronic disease prevention in a variety of states. Details and the conceptual framework have been provided previously.22–24
The 65-item survey addressed demographics, research evidence use, and organizational supports. Participants were asked “how often do you use research evidence to” do each of 6 job tasks, where 0 = seldom or never, 1 = sometimes, 2 = often, 3 = always, and 8 = not relevant to my role. The 6 job tasks were as follows: “write a grant application”; “plan or conduct a needs assessment”; “select policies, programs, or other types of interventions”; “justify selection of interventions to funders, agency leadership, or external partners”; “evaluate interventions”; and “develop materials for local public health agencies or external partners.” After removal of “not relevant to my role” responses, mean frequencies were calculated for each job task at time 1 (see the Supplementary Digital Content Table, available at http://links.lww.com/JPHMP/A480). Research evidence use at time 1 (2014) and time 2 (2016) summary variables were calculated for each individual as the mean of research evidence use frequencies in 6 job tasks after removal of “not relevant to my role” responses. Dichotomous summary variables were then created as the highest tertile of summary research evidence use scores versus all else (vs the middle and lowest tertile scores) because the dependent variable was skewed to the right (to the “agree” side), and odds ratios are readily interpretable. Perceived organizational supports addressing 4 domains identified through a literature review26 were assessed in thirteen 7-point Likert items, where 1 = strongly disagree and 7 = strongly agree. Table 1 shows item wording for the 4 domains: supervisory EBDM expectations; access to evidence and resources for EBDM; participatory decision making; and agency leadership encouragement. Organizational support item scores were dichotomized to strongly agree/agree (6 or 7) versus others.
Data management was conducted using SPSS 24. LCA was employed in SAS 9.4 to identify distinct, mutually exclusive subgroups of participants based on the 13 dichotomous variables of organizational supports.27,28 Six models were fitted to the data for 2 to 7 latent classes. Model selection took parsimony, interpretability, fit statistics, and model identification into account. Akaike information criterion, Bayesian information criterion, and adjusted Bayesian information criterion were used to compare the relative fit of the models. Lower values indicate a better fitting model. To check model identification, parameters were estimated from 100 random starting values for each model.21 Model identification was the proportion of the maxima in all log-likelihood solutions. Inclusive LCA is a recent method that includes the outcome variable (mean research evidence use at time 2 across tasks) as a covariate and may be superior to noninclusive LCA.28 By adjusting for the outcome variable and other characteristics, inclusive LCA can produce more accurate parameter estimates and improve estimate attenuation of the association between latent class membership and the outcome. After model selection, the optimal latent class model was then adjusted for the outcome variable, the 12 participating states, and job position.
To investigate the relationship between research evidence use at time 2 and class membership, 2 commonly used classify-analyze approaches were employed.27,28 The first approach is the maximum probability assignment, which assigns the class membership to individuals based on their maximum posterior probabilities. The second approach is referred to as multiple pseudo-class draws, which accounts for the uncertainty of class membership by randomly assigning class membership based on the distribution of posterior probabilities. Typically, the random assignment is repeated 20 times, and subsequent analysis is conducted in each repeat (as done here). Results are then combined according to rules of multiple imputation for missing data.29 Class membership was then incorporated as a predictor into a generalized linear mixed model in PROC GLIMMIX, with dichotomous research evidence use at time 2 as the outcome variable. The model included state as a random effect because of the sampling design and adjusted for individual characteristics (eg, research evidence use at baseline, education, position). Compared with the maximum probability assignment approach, the multiple pseudo-class draws approach takes the uncertainty of class membership into account27 and was therefore assumed to be less biased. However, some studies indicated that the maximum probability assignment approach had better performance on eliminating biases of point estimation.27,28 It has also been suggested that the performance of both approaches would be similar in the inclusive LCA.28 All analyses were conducted in SAS 9.4 with PROC LCA package.30
Analyses were conducted with 872 of the 909 participants with complete research evidence use and organizational support data (70.5% of baseline sample). Most (79.2%) were female, nearly two-thirds (65.6%) had a graduate degree, and nearly half (48.9%) were 50 years or older. Of the 872 survey participants, 49.3% worked in state health departments, 11.4% in local or district health departments, 9.8% in universities, and 29.5% in other partnering organizations. On average, participants had worked in their current position for 5.8 ± 5.5 years, in their agency 10.1 ± 7.8 years, and in public health 16.4 ± 12.7 years. Nearly half (47.6%) worked in program manager positions, 19.6% in leadership positions, 28.1% in specialist positions such as health educators, epidemiologists, or program evaluators, and 4.7% in other positions. Participants most commonly used research evidence to select or justify interventions (see the Supplemental Digital Content Table, available at http://links.lww.com/JPHMP/A480). Perceptions of organizational supports are shown in Table 1.
Latent class model selection
The LCA model fit statistics are presented in Table 2. The models with 5 or more latent classes had poor identification and therefore could not be the optimal models. The model with 4 latent classes was selected as the best model, as it had adequate model information criteria, good identification, and interpretability. Entropy was 0.86, which indicates low error in class membership assignment based on the maximum posterior probabilities. The inclusive LCA was adjusted for the outcome variable research evidence use at time 2 (P < .001), state (P < .001), and job position (P < .001).
The item response probabilities of organizational support indicators among 4 classes are shown in the Figure. Class 1 (24.1% of the total sample) was labeled as unsupportive workplace, with low probabilities in all domains of organizational supports. Class 2 (18.2% of the total sample) was labeled as low agency leadership support, with low probabilities in agency leadership support and moderate probabilities in the other 3 domains of organizational supports. Class 3 (31.5% of the total sample) was labeled as high agency leadership support, with high probabilities in agency leadership support and moderate probabilities in the other 3 domains of organizational supports. Class 4 (26.2% of the total sample) was labeled as supportive workplace, with high probabilities in all 4 domains of organizational supports. The proportion of participants in each class scoring in the highest tertile evidence use was 22.8%, 31.4%, 42.8%, and 44.5%, respectively. We found the same classes and patterns of organizational supports in separate basic LCAs in the subsample of only state health department and local health department staff and in the subsample of only state health department staff.
Relationship between research evidence use at time 2 and class membership
The odds ratios of scoring the highest tertile research evidence use at time 2 with 2 different classify-analyze approaches are summarized in Table 3. The unsupportive workplace class was set as the reference. In the generalized linear mixed modeling, research evidence use at baseline (P < .001) and having a master or doctorate degree in any field (P = .007) were incorporated as covariates. Other individual characteristics, such as gender (P = .42), job position (P = .99), and public health master's or doctorate (P = .13), and having a nursing degree (P = .15), were not incorporated in the final models. According to the results from the maximum probability assignment approach, participants in the high agency leadership support class were 2.08 (95% CI, 1.35-3.23) times more likely to score the highest tertile research evidence use at time 2 than the unsupportive workplace class. Participants in the supportive workplace class were 1.74 (95% CI, 1.10-2.74) times more likely to score the highest tertile research evidence use at time 2 than unsupportive workplace class members. However, the low agency leadership support class does not differ from the unsupportive workplace class in research evidence use at time 2 with odds ratio = 1.15 (95% CI, 0.69-1.92). The multiple pseudo-class draws approach produced similar results (Table 3).
In the present study, one of the few longitudinal inquiries on this topic, LCA methods were useful in identifying meaningful patterns of organizational supports for research evidence use with actionable implications. Participants working in organizational environments with high perceived agency leadership encouragement, supervisory expectations, access to evidence and resources, and participatory decision making at baseline were nearly twice as likely to frequently use research evidence for multiple job tasks at time 2 compared with participants with low perceived supports in these 4 domains.
Those in organizations with high perceived agency leadership encouragement, middle-high supervisory expectations, and middle other supports were also twice as likely to have the highest use of research evidence at time 2 compared with those in unsupportive workplaces. This finding implies that leadership support at the agency level in combination with moderate levels of work unit supervisory expectations for EBDM, access to evidence and resources for EBDM, and participatory decision making can further the use of research evidence in practice, including selection and justification of interventions.
Supervisory expectations alone, without high agency leadership encouragement, did not predict research evidence use. In a related interview study, state health department middle managers and program managers working in chronic disease prevention deemed it essential to also set up internal procedures and policies within agencies to support use of EBDM processes including use of research evidence in decision making.31 Examples included centralized surveillance databases with easy-to-use data access platforms readily available to staff and external agencies, internal performance management systems, agency-wide digital access to intervention evidence, internal policies and procedures requiring evidence-based justification of intervention selections, and use of research evidence in grant applications, requests for proposals, and contracts with external agencies.
High agency leadership in combination with high or moderate additional organizational supports was associated with increased research evidence use at time 2 in this study. Earlier studies also found leadership support a key facilitator of research evidence use, broader evidence use, and EBDM.3–8,14–16 In interviews in 6 European Union countries, Van de Goor and colleagues5 found leadership essential. As found in the present study, interview participants in Victoria, Australia, deemed senior managers the most influential. They stated that senior management sponsorship of projects helped ensure that research evidence would be used in decision making.7 In Denmark, a survey with 98 local health managers found health manager emphasis on evidence use was significantly correlated with evidence use for prioritization.32
In addition to leadership support, earlier studies identified organizational access to evidence as a facilitator to evidence use or a barrier if lacking.4,10,19,20 In the present study, perceptions of moderate or high access to evidence and other resources in combination with other domains were associated with increased evidence use. Van de Goor and colleagues5 also found lack of locally useful evidence and intervention cost-effectiveness evidence was a barrier.5 Ability and confidence to assess, interpret, and use the research also affected the gathering and use of research evidence. These aspects were not addressed in the present study but are important. Recommendations included collaborative learning approaches to build skills in research use,20 practitioner engagement in creation of local evidence,33 increased collaboration between researchers and practitioners so that research is geared to meet practitioner decision-making needs,8 and use of stories to highlight key messages in succinct and memorable ways.8,33
Research alone is not enough to drive the complex and politically influenced processes of EBDM.3,4,17 Political priorities, policymaker beliefs, community views, social norms, budget constraints, influences of external funder organizations and special interest groups, and opinions of selected experts and managers all play a part in public health decisions, including policy development.3,8,17,32 During the study period, federal funding agencies increasingly required evidence-based practice and the national movement toward accreditation of public health departments accelerated. Accreditation applications required health assessment, strategic planning, and a stronger focus on evidence-based practice. A better understanding of policy-making processes and influences, more practice-based research, more locally relevant research, and stronger trust, relationships, and linkages between researchers and decision makers are recommended in the literature to better address the political contexts in which public health decision making takes place.4,17,20,34
Additional influences noted in the literature include individual factors such as skills and confidence in one's ability to gather and use evidence3,6,7,14; characteristics of the available research itself, including complexity, lack of timeliness, and especially relevance for the specific locality, population group, or programmatic purpose4,5; and relationships and collaboration of practitioners with researchers such as through knowledge brokers.4,10 Interview participants also noted that the high staff turnover in health departments is disruptive to using evidence in decision making.4
Type of organization and job position were not associated with the frequency of public health research evidence use in the present study. However, in an Australian study with agencies that develop public health policies, Zardo and Collie6 found that senior managers were more likely to use research evidence than those in middle manager or nonmanager positions.6 Similar to the findings of Zardo and Collie,6 the present study found those with a master's or doctoral degree were more likely to frequently use public health research evidence than those without graduate degrees.
This study has several limitations. The data were self-reported responses from an online survey. With funders’ emphasis on EBPPs, there could be social desirability bias in participants’ responses. Frequency of evidence use was only asked for research evidence, although practitioners use a variety of types of evidence.23,34,35 Many participants responded that 1 or more of the 6 listed job tasks were not relevant to their job positions; other uses of evidence may have been missed. Additional organizational supports for research evidence use such as departmental quality improvement sections and performance management data systems were not asked. A more ideal study design would have been to survey staff from a single agency type, but our sample of state health department staff or any other single agency type was too small for inclusive adjusted LCA. Models were not adjusted for organizational demographics (such as the number of employees) and structures, although the staff from diverse organizations were included in the sample. However, we found the same classes and patterns of organizational supports in basic LCAs, with the subsample of only state and local health department staff and with the subsample of only state health department staff. And 37 individuals with missing data in research evidence use or organizational supports were excluded from analyses. In addition, the latent class model cannot handle missing data in covariates. In generalized linear mixed modeling, individuals with missing data in any predictors were excluded from analyses by PROC GLIMMIX. This study did not explore how research evidence use in decision-making processes may be affected by external political influences such as special interest advocacy groups or programmatic requests from elected officials.
Four distinct latent classes of organizational supports were identified with LCA in a sample of chronic disease prevention staff from 12 state public health departments and partnering organizations. The results suggest that agency leadership support may be a crucial element of organizational supports to encourage research evidence use. Other organizational supports such as supervisory expectations, access to evidence, and participatory decision making may need the backing of leadership to further research evidence use in public health job tasks.
There are several implications for research from this study and from the literature. Because of the limitations of self-report close-ended organizational support measures, next steps for research could include an expert practitioner panel to review and operationalize the classes found in this survey study to further identify gaps in organizational supports for EBDM and management practices to address gaps. Increased agency leadership encouragement to use research evidence in decision making can further public health practice. Availability of directly relevant and simply presented research evidence (including economic evaluation data on cost-effectiveness) as requested by interview participants in other studies5 could increase evidence use in public health decision making. Oliver and colleagues4,34,36 emphasize the need to learn more about public health decision making, policy-making processes, and how evidence is used at different stages of decision making. Learning more about these processes could help tailor approaches to increase research evidence use.
Implications for Policy & Practice
- Organizational supports identified in this study (eg, leadership encouragement, access to evidence, participatory decision making) are modifiable at relatively low cost.26
- Organization support measures used in this study can also be applied in practice settings to identify strengths and gaps in agency processes and commitment to EBDM.
- Public health agencies can increase the use of research evidence in grant applications, selection and justification of EBPPs, and evaluation through agency leaders valuing and encouraging EBDM and supervisors communicating expectations to staff.24
- Leaders can also increase access to evidence and resources for EBDM, provide skill-building opportunities, and ensure staff participation in decision making.
- Organizations supporting the public health workforce can disseminate evidence (including practice-based and locally relevant) through governmental Web sites, national guidelines, archived webinars, and workshops, which earlier studies found participants preferred as evidence sources.23,34
1. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental concept for public health practice. Annu Rev Public Health. 2009;30:175–201.
2. Brownson RC, Baker EA, Deshpande A, Gillespie KN. Evidence-Based Public Health. 3rd ed. New York, NY: Oxford University Press; 2018.
3. Armstrong R, Waters E, Moore L, et al Understanding evidence: a statewide survey to explore evidence-informed public health decision-making in a local government setting. Implement Sci. 2014;9:188.
4. Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14:2.
5. van de Goor I, Hämäläinen RM, Syed A, et al Determinants of evidence use in public health policy making: results from a study across six EU countries. Health Policy. 2017;121(3):273–281.
6. Zardo P, Collie A. Predicting research use in a public health policy environment: results of a logistic regression analysis. Implement Sci. 2014;9(1):1–10.
7. Zardo P, Collie A, Livingstone C. Organizational factors affecting policy and programme decision making in a public health policy environment. Evid Policy. 2015;11(4):509–527.
8. Orton L, Lloyd-Williams F, Taylor-Robinson D, O'Flaherty M, Capewell S. The use of research evidence in public health decision making processes: systematic review. PLoS One. 2011;6(7):e21704.
9. Li V, Carter SM, Rychetnik L. Evidence valued and used by health promotion practitioners. Health Educ Res. 2015;30(2):193–205.
10. Kneale D, Rojas-Garcia A, Raine R, Thomas J. The use of evidence in English local public health decision-making: a systematic scoping review. Implement Sci. 2017;12(1):53.
11. McBeath B, Jolles MP, Carnochan S, Austin MJ. Organizational and individual determinants of evidence use by managers in public human service organizations. Hum Serv Organ. 2015;39(4):267–289.
12. Williamson A, Makkar SR, McGrath C, Redman S. How can the use of evidence in mental health policy be increased? A systematic review. Psychiatr Serv. 2015;66(8):783–797.
13. Dobbins M, Hanna SE, Ciliska D, et al A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies. Implement Sci. 2009;4:61.
14. Zardo P, Collie A. Type, frequency and purpose of information used to inform public health policy and program decision-making. BMC Public Health. 2015;15:381.
15. Dodson EA, Baker EA, Brownson RC. Use of evidence-based interventions in state health departments: a qualitative assessment of barriers and solutions. J Public Health Manag Pract. 2010;16(6):E9–E15.
16. Jacobs JA, Dodson EA, Baker EA, Deshpande AD, Brownson RC. Barriers to evidence-based decision making in public health: a national survey of chronic disease practitioners. Public Health Rep. 2010;125(5):736–742.
17. Armstrong R, Pettman TL, Waters E. Shifting sands—from descriptions to solutions. Public Health. 2014;128(6):525–532.
18. Pettman TL, Armstrong R, Pollard B, et al Using evidence in health promotion in local government: contextual realities and opportunities. Health Promot J Aust. 2013;24(1):72–75.
19. Harris JK, Allen P, Jacob RR, Elliott L, Brownson RC. Information-seeking among chronic disease prevention staff in state health departments: use of academic journals. Prev Chronic Dis. 2014;11:E138.
20. Salter KL, Kothari A. Knowledge “translation” as social learning: negotiating the uptake of research-based knowledge in practice. BMC Med Educ. 2016;16:76.
21. Collins LM, Lanza ST. Latent Class and Latent Transition Analysis: With Applications in the Social, Behavioral, and Health Sciences. Hoboken, NJ: Wiley; 2010.
22. Allen P, Sequeira S, Jacob RR, et al Promoting state health department evidence-based cancer and chronic disease prevention: a multi-phase dissemination study with a cluster randomized trial component. Implement Sci. 2013;8(1):1–28.
23. Jacob RR, Allen PM, Ahrendt LJ, Brownson RC. Learning about and using research evidence among public health practitioners. Am J Prev Med. 2017;52(3S3):S304–S308.
24. Brownson RC, Allen P, Jacob RR, et al Controlling chronic diseases through evidence-based decision making: a group-randomized trial. Prev Chronic Dis. 2017;14:E121.
25. Kramer DM, Cole DC. Sustained, intensive engagement to promote health and safety knowledge transfer to and utilization by workplaces. Sci Commun. 2003;25(1):56–82.
26. Brownson RC, Allen P, Duggan K, Stamatakis KA, Erwin PC. Fostering more-effective public health by identifying administrative evidence-based practices: a review of the literature. Am J Prev Med. 2012;43(3):309–319.
27. Lanza ST, Tan X, Bray BC. Latent class analysis with distal outcomes: a flexible model-based approach. Struct Equ Modeling. 2013;20(1):1–26.
28. Bray BC, Lanza ST, Tan X. Eliminating bias in classify-analyze approaches for latent class analysis. Struct Equ Modeling. 2015;22(1):1–11.
29. Rubin DB. Multiple Imputation for Nonresponse in Surveys. Hoboken, NJ: John Wiley; 2011.
30. Lanza ST, Dziak JJ, Huang L, Wagner AT, Collins LM. PROC LCA & PROC LTA Users' Guide (Version 1.3.1). University Park, PA: The Methodology Center, Penn State; 2013. https://methodology.psu.edu/node/1070
. Accessed June 5, 2017.
31. Allen P, Jacob RR, Lakshman M, Best LA, Bass K, Brownson RC. Lessons learned in promoting evidence-based public health: perspectives from managers in state public health departments [published online ahead of print March 2, 2018]. J Community Health. doi: 10.1007/s10900-018-0494-0.
32. Larsen M, Gulis G, Pedersen KM. Use of evidence in local public health work in Denmark. Int J Public Health. 2012;57(3):477–483.
33. Cairney P, Oliver K. Evidence-based policymaking is not like evidence-based medicine, so how far should you go to bridge the divide between evidence and policy? Health Res Policy Syst. 2017;15(1):35.
34. Oliver KA, de Vocht F. Defining “evidence” in public health: a survey of policymakers’ uses and preferences. Eur J Public Health. 2017;27(suppl 2):112–117.
35. Brownson RC, Fielding JE, Green LW. Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research [published online ahead of print November 20, 2017]. Annu Rev Public Health. doi:10.1146/annurev-publhealth-040617-014746.
36. Oliver K, Lorenc T, Innvaer S. New directions in evidence-based policy research: a critical analysis of the literature. Health Res Policy Syst. 2014;12:34.