Davis, Mary V. DrPH, MSPH, NCIPH; Vincus, Amy MPH; Eggers, Matthew MPH; Mahanna, Elizabeth MPH; Riley, William PhD; Joly, Brenda PhD; Fisher, Jessica Solomon MCP; Bowling, Michael J. PhD
Quality improvement (QI) has been identified as a key strategy to improve the performance of state and local public health agencies.1,2 Over the last decade, a number of QI strategies and programs have been implemented at the federal, state, and local levels.3 According to results from the 2008 National Association of County and City Health Officials (NACCHO) profile survey, 55% of local health departments (LHDs) reported implementation of formal QI efforts.4
A number of these QI strategies have included training for frontline staff. Many of the 16 Multi-state Learning Collaborative (MLC) grantees have provided QI training to LHD employees.5 To date, however, there has been little assessment of the effectiveness of different QI training approaches to prepare the public health workforce to successfully conduct QI efforts. Furthermore, QI is an LHD strategy to improve agency performance. Although training LHD employees in QI approaches and tools is critical to ensure that the agency has the capacity to conduct this work, transformational change in the LHD culture to implement QI is required to assure the necessary foundation for QI implementation and success.6
Workforce development training has been characterized by levels of cognitive learning whereby participant knowledge, skill, and ability to perform specific tasks increase as the level of instruction and opportunities to practice skills increase.7,8 Basic learning approaches are designed to increase content knowledge, awareness, and understanding. Midlevel approaches provide opportunities to apply concepts and skills in the content area. High-level approaches provide opportunities for synthesis and application of concepts and skills. Evaluations of training offerings typically examine participant improvements in knowledge and skills and ability to apply these in the workplace setting.9–11 Workplace supports and barriers to implementing acquired knowledge and skills should also be assessed to account for opportunities to enact these newly acquired skills.12
The purpose of this article is to examine the effectiveness of 3 NACCHO QI training approaches: webcasts (basic-level training), face-to-face workshops (midlevel training), and demonstration site projects (high-level training). As part of its efforts to support LHDs to conduct QI and prepare for voluntary accreditation, NACCHO provided these trainings to LHD staff. We examine the impact of these trainings on participant knowledge, skill, and ability to successfully participate in a QI project, and receptivity to learning more about QI. We hypothesized that the highest-level training, demonstration site projects, was most likely to result in greater gains in outcomes of interest.
NACCHO offered 3 types of QI training to LHDs:
1. A series of 5 webcasts offered between December 2007 and April 2009 available to any health department staff member.
2. Face-to-face workshops offered between July 2007 and February 2008 (at meetings of the state associations of county and city health officials in Colorado, Connecticut, Michigan, Ohio, and Nebraska).
3. Applied training offered to 41 health departments that participated in demonstration site projects in 1 of 2 rounds: September 2007 to May 2008 (round 1) or April to November 2008 (round 2). Local health departments completed a self-assessment and then used QI techniques to address weaknesses identified via the self-assessment. The training offered to demonstration sites included meetings with consultants; peer or collaborative network discussions; and formal, tailored trainings about QI.
Participants or their agencies self-selected into the various training modalities. In the case of the demonstration site trainings, the health department self-selected that the agency would participate. NACCHO provided registration data for 586 LHD staff who participated in webcasts and 129 participated in face-to-face workshops. NACCHO also provided information for the 41 contacts at each demonstration site. Demonstration sites were expected to have teams of employees participating in the project; therefore, we contacted each demonstration site to obtain a list of all employees who had worked on the project to include them in the survey pool. Registration information on participants was collected by NACCHO for nonresearch purposes. This information varied by training type and therefore could not be further used to inform this research.
Survey design and pretesting
We identified potential items from the literature and practice in the following domains: (1) QI knowledge and skill gain, skill application, receptivity, to learning about and implementing QI activities; (2) barriers and facilitators to implementing QI in LHDs; (3) successful implementation of QI projects since participating in NACCHO trainings; (4) accessing other QI resources; and (5) respondent characteristics including tenure working in the LHD and participation in other QI trainings.
We pretested a draft version of the survey questionnaire with 8 MLC representatives and NACCHO Public Health Infrastructure and Systems advisory committee members. Through pretesting, we revised questionnaire items on the basis of respondent feedback and shortened the entire questionnaire to be completed in approximately 15 minutes.
We fielded a 29-item Web-based survey in spring 2010. Any LHD staff who registered for any of the webcasts or workshops, or who participated in a demonstration site project, was invited to complete the Web questionnaire (n = 741). We used multiple, personalized contacts to notify and invite potential respondents to respond to the questionnaire, including mail prenotification, e-mail invitation, and multiple e-mail reminders.13 Among the 741 invited to complete the questionnaire, 96 were not reachable through e-mail or were not LHD employees, leaving 645 eligible questionnaire respondents.
The independent variable of interest was individual training modality participation. We categorized respondent participation as (a) single participation, where respondents reported participating in only 1 training modality or (b) “plus” participation where a respondent reported participating in 2 or more training modalities.
Dependent variables were assessed at the individual level and included (1) knowledge and skill gain, (2) skill application, (3) receptivity to additional QI training and activities, and (4) contribution of NACCHO training(s) on participant perceived ability to successfully participate in a QI project.
We assessed knowledge and skill gain, skill application, and ability to successfully participate in a QI project through single items with 6-point ordinal scales anchored at each end (from “none” to “a great deal”). For example, the measure of knowledge gain directed respondents to “please rate the degree to which you gained new knowledge as a result of participating in one or more of NACCHO's QI trainings.” We combined 3 items that measured receptivity to QI (learning more about QI concepts and tools, wanting to participate in QI projects, and wanting to lead QI projects) to create an 18-point index variable. The 3 measures were strongly and positively correlated with the total (0.72, 0.82, 0.70) and had a standardized α of .87.
We created an index variable of 8 items that assessed barriers to applying QI concepts in the health department. For each item, respondents were asked to rate on a scale of 1 (does not limit at all) to 6 (limits a great deal) the extent to which specific elements (such as lack of fiscal support) limited their ability to apply QI tools and concepts. All measures were positively correlated with the total (low = 0.45, high = 0.69) with a standardized α of .84. For facilitators to apply QI skills, we created a 6-item index variable to assess factors that, if present, have the potential to facilitate application of QI skills (using a scale of 1 does not help at all to 6 helps a great deal). All measures were positively correlated with the total (low = 0.44, high = 0.84) with a standardized α of .88. Lastly, we created an index variable of 3 items to assess senior management support for QI using the same scale as facilitators to QI. All measures were positively correlated with the total (0.76, 0.82, 0.84) with a standardized α of .91.
We conducted all descriptive analyses using SAS 9.2 PROC FREQ and PROC MEANS with multivariable models fit using PROC SURVEYREG (SAS, Cary, North Carolina). Complex survey design software accurately computed standard errors of estimates compensating for nonindependence of employees drawn from the same LHD. Descriptive statistics included calculating means and 95% confidence intervals for dependent variables. We calculated separate means and 95% confidence intervals for the 6 training participation approaches: single webcast, workshop, demonstration site, web plus, workshop plus, and demonstration site plus. We built bivariable and multivariable models. Multivariable models included the following potential control variables: barriers to implementing QI, facilitators to implementing QI, senior management support in implementing QI, other QI resources accessed, and other QI trainings accessed. Additional control variables included size of population served by LHD, LHD geographic jurisdiction, and LHD participation in other major QI activities such as the MLC and use of the National Public Health Performance Standards Program (NPHPSP) local agency tool. Local health department characteristic data were obtained from NACCHO. MLC participation and NPHPSP use were obtained from those programs. We specified a 10% change in regression coefficients for key independent variables as a criterion for including a potential control variable in a regression model. Statistical significance was assumed to exist at P ≤ .05.
We hypothesized that single demonstration site respondents would have the highest score on all dependent variables when compared with respondents who participated in other single training modalities. Single webcast participation was used as the referent category because there were sufficient numbers of respondents in this category and we posited that this would have the least impact on the outcomes of interest. We further hypothesized that demonstration site respondents who participated in another training modality would have higher scores on all dependent variables when compared with respondents who only participated in the demonstration site training modality. Workshop respondent scores were not included in the multiple training modality category models as there were an insufficient number of respondents in this category.
Three hundred seventy-eight individuals responded to the questionnaire for a response rate for all respondents of 58.6%.14 Data cleaning resulted in removal of 89 responses. Thus, 289 of the 300 respondents who work in 143 health departments completed all questionnaire items for a completion rate of 96%. Calculating response rate by training type was based on registration data and varied from 53% for demonstration site participants to 47% for webinar participants and to 25% for workshop participants. These response rates are lower than the total response rate as additional participants were added to the survey pool, particularly for demonstration site participants. Respondent tenure in a public health department was evenly spread across 1 to 10 years (34%), 11 to 20 years (31%), and more than 20 years (35%). Among the 225 respondents who reported job classification, the highest percentage (35%) reported being the health director or agency administrator. Other respondent job categories included directors of nursing and nurse supervisors, health educators, environmental health specialists, analysts, or consultants. Health directors were more likely to report participation in the workshops as NACCHO targeted this training to them. There was no meaningful pattern of participation by training type among other job classifications. The majority (55%) of the LHDs where respondents work serve populations between 40 000 and 499 999, 78% have a local governance structure, and the mean number of full-time equivalent employees is 177.
The remainder of the findings focus on differences among respondents who participated in different training types. Figure 1 shows the distribution of respondent participation in the 3 training modalities. The highest number of respondents (101) reported viewing 1 or more webcasts (and did not participate in any other modality), 65 respondents reported participating only in a demonstration site, and 17 respondents participated only in a workshop. Seventy-six respondents reported participating in a demonstration site project and viewing 1 or more webcasts; 16 reported participating in a workshop and viewing one or more webcasts; and 3 respondents reported participating in a workshop and a demonstration site project. Six respondents participated in all 3 training modalities (variations in numbers of participants in different analyses reflect cases with missing data on modality participation and/or LHD identifier information).
FIGURE . Individual ...Image Tools
Knowledge gain, skill gain, and application of skills by modality
Table 1 shows means and 95% confidence intervals for knowledge and skill gain, skill application, and receptivity to QI by training participation category. Single demonstration site participants reported the most gains in knowledge, skill, and skill application among single training participants. Workshop participants reported the least gains in these domains; confidence intervals for the workshop-related means were quite large, likely reflecting the small number of questionnaire respondents who participated in this training modality. In linear regression models comparing participants in single training categories (Table 2), mean scores for demonstration site participants were significantly higher for knowledge and skill gain in the adjusted models that included QI resources accessed. For skill application, the adjusted model included QI resources accessed and facilitators to implementing QI. Adjusted model R2s for all 3 models, however, accounted for only 10% to 15% of variance.
Respondents who participated in multiple training modalities had higher mean scores on knowledge and skill gain and skill application than those who only participated in a single training. The greatest gains were among respondents who participated in a demonstration site and another training modality, but these differences were not significant in regression models (Table 3).
Quality improvement receptivity
Among single training category respondents, single webcast category participants had the highest mean scores on the QI receptivity index (14.18) followed by single demonstration site participants (11.78). As presented in Table 2, only the unadjusted regression model resulted in significant differences among respondents in different single training categories, with Web site participants reporting significantly higher receptivity scores when compared with single workshop (−2.66, P = .01) and demonstration site respondents (−2.40, P < .001). The percentage of variance explained for the model was quite low (R2 = 0.09). Regression models examining differences between respondents in the single demonstration site and demonstration site plus webcast categories yielded significant differences for the unadjusted model only (2.78, P = .001) with an R2 = 0.13 (Table 3).
Ability to successfully participate in a QI project
Among single training category participants, single demonstration site respondents had the highest mean scores (4.02) followed by Web site participants (3.13) and workshop participants (3.00) on successful participation in a QI project. On this outcome, single demonstration site participant mean scores were significantly higher than single webcast participant mean scores in the adjusted regression models (1.18; P = .0001). This model also included QI resources accessed and barriers to implementing QI and explained 19% of the variance.
Participants in the demonstration site plus webcast category had the highest mean score for ability to successfully participate in a QI project of all participant categories (Table 3). In the unadjusted model, there was a significant difference between scores (0.57; P < .05); however, these differences were not significant in the adjusted model and the unadjusted model only explains 5% of variance.
This study is among the first to evaluate the effectiveness of public health QI training approaches. Questionnaire results confirm our primary hypothesis that demonstration site participants—a high-level training approach—would have the greatest gains on outcomes of interest, including knowledge and skill gain, skill application, and ability to participate in a QI project. Regression models showed that these relationships were significant when accounting for other factors, including QI resources accessed, facilitators to implementing QI, and barriers to implementing QI. An unexpected finding was that single web respondents had the highest scores on the QI receptivity index, but these findings were only significant in unadjusted models. This finding may be a result of higher interest among webcast participants who self-selected to participate. In addition, demonstration site respondents may believe they have sufficiently learned about QI.
Questionnaire results also provide support for the hypotheses that respondents who participate in multiple training types would report greater knowledge, skill gain, and ability to participate in a QI project. Although regression models were not significant; the trends were in expected directions.
Previous literature on accountability, performance improvement, and performance management frameworks have noted that perceived barriers, particularly costs (personnel and time) and lack of leadership support would prevent many LHDs in adopting these efforts.6,15 In our results, only 1 regression model included barriers to implementing QI, indicating that the barriers we studied had a limited effect on participants’ abilities to implement QI projects. Other QI resources accessed was the most predominant control variable included in our regression models. These resources include materials from NACCHO, the Public Health Foundation, the Public Health Accreditation Board, NPHPSP, and MLC.
These findings, however, may not be generalizable to all LHDs. Questionnaire respondents work in LHDs that serve larger populations, are more likely to have a local governance structure, and a greater number of full-time equivalent employees when compared with all LHDs who responded to the 2008 profile survey.4 Results from the profile survey indicate that LHDs that serve larger populations were significantly more likely to have engaged in QI. In contrast with our findings, the profile results indicate that LHDs with centralized (versus local) governance were more likely to have engaged in QI. Local health departments that serve smaller populations report concerns about agency abilities to engage in accountability activities, such as accreditation, because of small staff sizes and stretched budgets.16 Encouraging these LHDs to participate in QI may require different training models or specialized incentives.
The following are limitations to interpreting study results. The most significant limitation is that this is a posttest only study design with no comparison group. Thus, findings must be interpreted with caution. In addition, recall bias among questionnaire respondents is of concern. The early webcast and workshop modalities occurred 24 to 30 months prior to survey implementation. Those participants may not have remembered participating in those trainings or their recall of knowledge and skill gain may not be accurate. Furthermore, this time lag may have resulted in a small number of respondents in various categories limiting statistical power to detect significant differences between participants in different training modalities. In particular, the low response rate among respondents in the face-to-face workshop category hampered our ability to determine the true effectiveness of this training modality.
Questionnaire results may be highlighting selection biases of high performing LHDs and their employees. Among the 143 LHDs represented by questionnaire respondents, 54% of these LHDs have also participated in an MLC project and/or completed the NPHPSP local instrument. The existing QI culture in these LHDs may have played a factor in questionnaire respondent participation in QI trainings as well as ability to apply training content.12 The fact that many respondents participated in multiple training approaches may underscore their agencies quality improvement culture. Finally, while there were statistically significant differences on several outcomes among training modalities, the variance in outcome measures explained by these models was quite low despite our attempts to control for potential confounding variables. Future research may need to account for other factors.
Our findings suggest that a public health QI training framework should include both didactic training on QI content and opportunities for QI application. These results are consistent with recommended training frameworks for QI seen in health care. The Institute for Healthcare Improvement Breakthrough Series model is built on the premise that QI learning is accelerated when training includes didactic content, as well as direct application of knowledge. Learning is further advanced when teams from different agencies work on common solutions together.5,17 The MLC III promoted this model in the 16 participating states.5 Findings support the use of this approach in which the Robert Wood Johnson Foundation made a considerable investment. Future research should examine whether this approach can effectively increase successful participation in QI projects for staff in LHDs of all sizes.
1. Riley WJ, Beitsch L, Parsons HM, Moran JW. Quality improvement in public health: where are we now? J Public Health Manag Pract. 2010;16(1):1–2.
2. Institute of Medicine, ed. The Future of the Public's Health in the 21st Century. Washington, DC: National Academies Press; 2007.
3. Davis MV. Opportunities to advance quality improvement in public health. J Public Health Manage Pract. 2010;16(1):8–10.
4. Beitsch LMJD, Leep C, Shah GMS, Brooks RG, Pestronk RM. Quality improvement in local health departments: results of the NACCHO 2008 survey. J Public Health Manag Pract. 2010;16(1):49–54.
5. Gillen SM, McKeever J, Edwards KF, Thielen L. Promoting quality improvement and achieving measurable change: the lead states initiative. J Public Health Manag Pract. 2010;16(1):55–60.
6. Riley WJ, Parsons HM, Duffy GL, Moran JW, Henry B. Realizing transformational change through quality improvement in public health. J Public Health Manag Pract. 2010;16(1):72–78.
7. Alexander LK, Horney JA, Wallace JW, Davis M, Wilfert RA, MacDonald PD. Guiding principles of a comprehensive Internet-based public health preparedness training program. Poster presented at: American Public Health Association Annual Meeting; November 2006; Boston, MA.
8. Alexander LK, Horney JA, Markiewicz M, MacDonald PD. 10 guiding principles of a comprehensive Internet-based public health preparedness training and education program. Public health rep. 2010;5(suppl):51–60.
9. Davis MV, Sollecito WA, Shay S, Williamson W. Examining the impact of a distance education MPH program: a one-year follow-up survey of graduates. J Public Health Manag Pract. 2004;10(6):556–563.
10. Umble KE, Orton S, Ottoson JM, Barclay G. Evaluating the impact of the management academy for public health: developing entrepreneurial managers and organizations. J Public Health Manag Pract. 2006;12:436–445.
11. Kirkpatrick DL. Evaluating Training Programs. 2nd ed. San Francisco, CA: Berrett-Koehler Publishers Inc; 1998.
12. Ottoson JM, Patterson I. Contextual influences on learning application in practice: an extended role for process evaluation. Evaluation Health Prof. 2000;23(2):194–195-211.
13. Schaefer DR, Don A, Dillman. Development of a standard e-mail methodology: results of an experiment. Public Opin Q. 1998;62(3):378–397.
14. The American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 6th ed. Deerfield, IL: American Association for Public Opinion Research; 2009.
15. Davis PB, Solomon JMCP, Gorenflo G. Driving quality improvement in local public health practice. J Public Health Manag Pract. 2010;16(1):67–71.
16. Davis MV, Cannon MM, Corso L, Lenaway D, Baker EL. Incentives to encourage participation in the national public health accreditation model: a systematic investigation. Am J Public Health. 2009;99(9):1705–1711.
17. Massoud M, Nielsen G, Nolan K, Sevin C, eds. A Framework for Spread: From Local Improvement to System Wide Change. Cambridge, MA: Institute for Healthcare Improvement; 2006.
local health department; public health workforce; quality improvement; training
© 2012 Lippincott Williams & Wilkins, Inc.