Local health jurisdictions work on the front lines to combat threats to the health of their communities including those associated with disasters and other public health all-hazards everyday. In 2003, the National Association of County and City Health Officials (NACCHO) established the Advanced Practice Centers (APC) program in collaboration with the Centers for Disease Control and Prevention to provide local health departments (LHDs) with access to critical public health preparedness resources. For the last 7 years, the APC program has been at the forefront of public health preparedness and worked diligently to create more than 100 products that benefit LHDs. One intent of the program is that APCs save LHDs substantial time, energy, and money by creating products that are free and easily adaptable and can be quickly and easily integrated into local plans, procedures, and practices. These tools mean that LHDs do not have to reinvent the wheel through access to tried and true products that enhance public health workforce capabilities. During the 2010 to 2011 program year, the APC launched a quality improvement (QI) initiative that captured feedback of end users through a QI product and process evaluation. This column details the lessons learned from the yearlong initiative.
Quality Improvement and Public Health
Quality improvement has been broadly defined for public health as “the use of a deliberate and defined improvement process, such as Plan-Do-Check-Act, which is focused on activities that are responsive to community needs and improving population health. It refers to a continuous and ongoing effort to achieve measurable improvements in the efficiency, effectiveness, performance, accountability, outcomes, and other indicators of quality in services or processes [that] achieve equity and improve the health of the community.”1 (p6)
Quality improvement is especially important now in public health, as state and local health department funding has become more and more constrained and while expectations for performance in protecting and improving public health have not diminished. Awareness has increased regarding the needs for more effective prevention, for better access to services, and for public health to be prepared to respond in emerging challenges, such as the influenza A virus (H1N1) pandemic. This increased awareness has raised the public's expectations of public health performance, making continuous QI a necessity, not an option. Quality improvement is considered important enough by the Public Health Accreditation Board to make it part of an accreditation standard for local, state, and tribal health departments.2
The APC QI Initiative
The APC program consists of 8 APC sites (Table 1) established by a grant awarded through a competitive bid process from across the United States and representing jurisdictions with different sizes, population demographics, and preparedness priorities. Throughout 2010, APC sites developed 24 new and innovative preparedness products, ranging from a Blueprint for the Use of Volunteers in Hospitals and Rural Medical Centers, a Transitional Framework for Pandemic Readiness and Response, and an eMedCheck, with Blackberry, iPhone, iPod, and iPad applications.
To fully test the efficacy of these new APC products at the local level, the concept of the APC connector sites (connectors) initiative was launched. Connectors were also established through a national competitive bid process. Each new connector was paired with an existing APC site to work in conjunction with a consultant to plan, implement, and evaluate an APC product through a QI process (Table 2).
After formal announcement of the awards, NACCHO solidified the partnerships between the APC sites and connectors through a series of conference calls to familiarize the pairs with the products. The initiative launched in November 2010 with a 2-day QI training workshop for the 8 connectors. The session also introduced connectors to their QI coaches, who, along with the APC program director, would guide them through the initiative.
The QI Protocol
The QI protocol addressed the time constraint challenge posed by having less than a year to implement multiple Plan-Do-Check-Act cycles by using a stepwise Plan-Do-Check-Act approach. Instead of waiting to get through an entire product implementation cycle to analyze data, the connectors broke their product implementation process into key steps and attempted to use what they learned from evaluation at each step to inform and improve efforts at future steps.
The protocol was based on using leading indicators, organized by having each connector develop a chain of success, in which they identified implementation steps that offered useful evaluation points. Each evaluation point involved collecting and analyzing data on leading indicators of future success. The leading indicators could be, for example, measures of interim product benefits (eg, strengthen partnerships, develop staff skills, and design exercises faster or more efficiently) or measures of beneficial project attributes (eg, easy to use). For each evaluation point in the chain of success, the connectors defined the expected results or benefits, the time frame for evaluation, evidence to document success, indicators to measure success, and the major learning focus.
The QI coaches helped connectors develop QI-related evaluation plans, guided them through parts of their QI and evaluation process, as they needed it, and assisted in preparing product evaluation reports. At the start, the coaches facilitated meetings between each connector and its related APC. The first were online meetings to establish mutual expectations. The later meetings were in-person at a NACCHO event to review draft evaluation reports and ways APCs can improve products. The coaches did not collect any evaluation data, which was the responsibility of the connectors, and at no time did either coach go on site to observe projects or provide implementation assistance. Except for the group sessions and meetings at NACCHO events, all coaching was at a distance.
The intent of the project was to gauge if the selected APC product would likely impact a specifically identified preparedness challenge in the home jurisdiction of the connector. The QI process would be able to document that impact and qualify the lessons learned. Connectors did demonstrate that they could capture data although implementing the products to enable them to recommend product improvements. Some user departments even made their own product improvements that can be offered to future user departments. Improvements being identified for all 8 APC products do not mean that they were defective. All connectors found the products they tested valuable, and they all achieved benefits from using the products. Specific lessons learned improved capabilities and processes, expanded partnerships, demonstrated product versatility and dual use, and documented recommendations for product improvements.
New preparedness capabilities developed at very low cost
Some sites developed entirely new kinds of skills, preparedness plans, or resources, giving them important new capabilities without having to add staff or take on other major costs. They generally used time of existing personnel to gain capabilities they could not have developed without much greater staff time or cost if the APC products were not freely available. Given the restrictive funding environment of almost all LHDs, it is highly unlikely that they would have developed these capabilities without the use of the APC products. For example, the Southern Nevada Health District developed new capabilities in infectious disease response by developing a National Incident Management Systems–compliant Infectious Disease Emergency Response Plan. The Palm Beach County Health Department developed new capabilities for collaborating with community pharmacists for preparedness by using the Web site that the Montgomery County APC designed for that purpose.
Improved existing preparedness capabilities, processes, and plans
Connectors found that APC products enabled them to improve existing capabilities or make existing preparedness processes and plans more efficient or effective. For example, staff of Champaign-Urbana and partners reported that it took them less time, on average, to develop exercises on the basis of the toolkit used than it took them in the past to develop similar exercises. Gunnison Valley Health Hospital found that the emergency operations plan in the Mesa County APC toolkit identified an area they had not addressed in the hospital's emergency operations plan. They used that information to improve the Gunnison Valley Health Hospital's emergency operations plan. The Thurston County Public Health and Social Services Department expanded its surge capacity for public health outbreak investigations by using an APC product that provides a framework to train department staff and Medical Reserve Corps volunteers who do not normally play investigative roles but have skills transferrable for that purpose.
Partnership, organization, or staff development
A number of sites reported improved coordination or relationships with community partners as a result of using an APC product. In some cases, this involved existing preparedness partners for LHDs such as law enforcement agencies, fire departments, hospitals, medical reserve corps units, and emergency management agencies. In other cases, it involved new preparedness partners, such as with community pharmacists in Palm Beach County and schools and preschools in Oklahoma City County.
Product versatility and user choice: Users choose only parts of products and still benefit
All the APC products evaluated have multiple parts, whether multiple pages and resources on a Web site or multiple tools in a “toolkit.” Some are quite comprehensive, such as the Mesa County toolkit for using volunteers with some 136 tools, or the San Francisco Bay Area APC's infectious disease emergency response toolkit with 40 core templates and 178 job action sheets. Although connectors generally examined all parts of the products they tested, only Palm Beach County actually used and thoroughly evaluated all parts of the product. Other connectors selected the parts they found most applicable to their setting and, with occasional local adaptations, used those parts to good advantage and derived benefits as described earlier. The variety of tools or product “parts” that users can pick and choose makes the APC products especially versatile and expands the range of LHDs and communities that can derive benefits by using APC products.
Product versatility: Multiple preparedness uses and dual use
Several products were shown to have more than 1 way to be used to improve public health preparedness, including uses identified by the connectors that the APC did not necessarily anticipate. Gunnison County used parts of a toolkit designed for using volunteers in a rural hospital and applied the directions for use of volunteers in an alternate care sites. Southern Nevada determined that the San Francisco Bay Area APC's infectious disease emergency response toolkit is useful not only for infectious disease emergency response plans, for which it is designed, but also for other types of emergencies. They plan to use the APC toolkit formats and components to develop Radiation Emergency and Chemical Exposure Emergency response plans.
This project also revealed significant potential in most APC products tested for “dual use,” for both emergency preparedness and other public health applications. For example, Palm Beach County concluded that the Montgomery County APC's Prescription for Preparedness Web site could, with the addition of appropriate content, be used to engage pharmacists on topics of interest to pharmacists and public health in addition to all-hazard preparedness. Palm Beach County suggests that LHDs could succeed in engaging more pharmacists to help address additional public health outcomes, while also building pharmacists' support for preparedness.
Cerro Gordo County found that tools from the Seattle-King County APC's Continuity of Operations Plan toolkit are also beneficial for everyday operations. For example, tools such as the Line of Succession template and Delegation of Authority letters (nonmedical) can be valuable to help staff understand their roles and responsibilities in filling in for others in more common operational situations, not just when continuity of operations is threatened.
Recommendations for product improvements
Connectors were serious about their QI role of not only testing and evaluating APC products for their own use but also recommending improvements in the products for other LHDs that can use the products in the future. Recommendations included better-user orientation on products and their use (eg, product summaries and orientation videos), start-up guides (eg, presentations and videos), clearer contextual information on which tools to use, documentation of user requirements (eg, staff skills and internal systems), and clearer user expectations. Some connectors that used exercise materials provided in APC products found that the scenarios were not the best matches for their local jurisdiction. These sites recommended that additional scenarios be provided by the APC for user LHDs to choose from, so they can more likely find locally relevant scenarios.
Findings from this APC QI process have implications that can have a beneficial impact for LHDs as they build capacity and capabilities for public health preparedness.
Valuable user data can be captured at key “learning moments” when implementing APC products. Although APC products differ, all those tested had key points in their implementation process where connector sites collected data to aid their evaluation. In this project, they were called “evaluation points.” But they are more generally “learning moments” when critical user data can be captured to inform product improvement, even if complete evaluation projects are not done.
Identifying explicit testable benefits and beneficial attributes for each product is important. Although APCs generally identified a purpose or objectives of their products, they often were less clear about explicit benefits to be achieved by product users. Identifying explicit benefits that connectors could test for became an important early step in the coaching process. Similarly, it was important for coaches to work with connectors and APCs early in the process to identify “beneficial project attributes” (eg, ease of use and comprehensiveness) that users could be expected to experience.
Local health departments' sharing knowledge and tools to assist each other's learning is useful. From time to time during the project, connectors shared tools they used for QI and evaluation. Sometimes, the coaches saw common needs across sites and facilitated sharing of instruments used for user surveys or for tracking likely recommendations. In 1 case, 1 connector site volunteered to be the first to draft initial sections of an evaluation report based on an outline prepared by the coaches. This enabled other sites to have a better idea of how to prepare reports than if they had just seen a blank outline and enabled the outline to be improved before everyone used it. Sharing across LHDs should be promoted as a valuable way to improve products and how they are used.
Useful APC QI processes can be implemented without full-scale evaluations. If NACCHO and APCs can structure product implementation to take advantage of key learning moments, then valuable user data can be captured for QI without the burden of full-scale evaluations. The APCs could embed user survey instruments in some of the tools or parts or their products, or in product Web sites, for user departments to provide data to APCs at learning moments. The use of the same instruments by different users may assist in APC analysis and interpretation of user data.
One key lesson learned from this QI initiative is that continuous learning must be enabled and fostered. If practical sharing and communication tools can be provided, then APC product users can learn from each other and from APCs. They could quickly learn not only about products and how to use them but also about how other users have adapted the product and even gain access to the user adaptations, and APCs will obtain more user data at a faster pace. Also, APCs can learn from each other about QI processes or ways to obtain user data at learning moments in each other's product implementation plans. On the basis of the initial success of the QI pilot, 2 activities will launch during the upcoming 2012 program year: the QI Template for product review and the Community of Practice (COP).
QI template and uniform product review
Insight gained from the chain of success and learning moments documented throughout the QI initiative led to development of a QI template that guides end users through a modified Plan-Do-Check-Act process. The QI template prompts participants reviewing a specific APC product to answer a few questions about expectations for product use and experience in obtaining the product. During and after the implementation process, additional questions are aimed at gathering information about the outcomes of the experience.
Each of the existing 8 APC sites will reach out to 4 additional organizations to implement a peer QI review of APC products. Partner organizations will have the benefit of expert tutorials in using the product; they will increase the preparedness capacity of their organizations; and partnerships, both internal and external, will be built and strengthened. The uniform QI template will gather like information and continue to document usefulness of the APC product inventory.
Community of practice
The QI initiative underscored the need to keep conversations going and share lessons learned. An online COP will provide a platform for posting learning moments. The COP can be found under the community tab on the APC Web site (apc.naccho.org). The COP provides additional support for public health practitioners and preparedness partners. Practitioners can post inquiries and cite lessons learned regarding capabilities, functions, tasks, or resource element being implemented in the local jurisdiction. Practitioners can also pose questions about how a product was adapted, adopted, or modified to meet individual needs.
The intent of the APC program is to save LHDs substantial time, energy, and money by creating products that are free and easily adaptable and can be quickly and easily integrated into local plans, procedures, and practices. These actions, if successful, will in turn enhance public health workforce capabilities. Rather than just assume a correlation, the APC QI initiative documented that the process works.