Gorenflo, Grace G. MPH, RN; Klater, David M. MBA; Mason, Marlene BSN; Russo, Pamela MD, MPH; Rivera, Lillian MSN, PhD, RN
Increasingly, health departments are conducting performance improvement initiatives,1–6 and at least 3 health departments have successfully applied for their state's equivalent of the Malcolm Baldrige Award for Excellence (Baldrige program).7,8 Launched in 2011, the Public Health Accreditation Board's (PHAB's) voluntary national accreditation program now offers a platform for performance and quality improvement specifically tailored to public health practice. Among the requirements for attaining PHAB accreditation status is the use of a “fully functioning performance management system that is completely integrated into health department daily practice,” and the Baldrige criteria are specifically cited as one possible resource to this end.9The Quest for Exceptional Performance10 tool was developed to illustrate the complementary nature of these 2 nationally recognized programs and to help health departments capitalize on the connections between them. This article describes how The Quest for Exceptional Performance tool was developed and analyzes areas of alignment between the 2 programs to identify the most robust connections.
Malcolm Baldrige Award for Excellence
The Baldrige program was established in the late 1980s by Congress, in recognition of Secretary of Commerce Malcolm Baldrige, with an initial goal to enhance the competitiveness of US businesses in the global marketplace.11 Housed in the National Institute of Standards and Technology in the US Department of Commerce, the program has evolved to become a public/private partnership focused on improving the performance of a broad range of US organizations.
Nearly all states operate or are served by state or local Baldrige-based programs that are connected by a national network, the Alliance for Performance Excellence.12 Using the Baldrige criteria, these programs help numerous organizations begin and progress along their performance excellence journey. Since 2012, recognition from one of these programs is a prerequisite for Baldrige program applicants housed in a state with an Alliance program.
The Baldrige Criteria for Performance Excellence serve as the cornerstone of the program and are built around 7 “categories” focusing on critical aspects of management that contribute to performance excellence: leadership; strategic planning; customer focus; measurement, analysis, and knowledge management; workforce focus; operations focus; and results. The 7 Baldrige categories are further subdivided into “items.” For Baldrige categories 1 through 6, each item has several “areas to address” with corresponding questions regarding how the organization addresses the item. Baldrige Award applicants are required to submit narrative responses to the questions posed under each item, describing how the items have been addressed. The final category (Baldrige Category 7) requests information about results that have been achieved. Collectively, the categories, items, areas to address, and questions are referred to as the Baldrige “criteria.”
The national Baldrige program offers 3 versions of the assessment criteria, each of which is tailored to a different sector (education, health care, and business/nonprofit). No substantive difference exists among these versions; rather, they reflect language and examples specific to each sector.
To initiate the program's rigorous evaluation process, organizations submit a 50-page application that provides narrative responses to the assessment criteria. An independent Board of Examiners, comprising trained volunteers from all sectors of the economy, reviews the applications to assess the degree to which the criteria have been met. Approximately 20% of Baldrige applications pass this initial screening and undergo an intensive, weeklong site visit. Of the 1510 applications received in the program's 25-year history, only 95 organizations (approximately 6%) have received the award.13
Public Health Accreditation Board
The PHAB program is built around 12 “domains,” each of which has several “standards.” Each standard has several “measures” with a description of corresponding documentation that applicants must submit to demonstrate fulfillment of each measure. The assessment components are routinely referred to as PHAB “standards and measures.” The vast majority of measures are applicable to all types of applicants (ie, state, territorial, local, and tribal health departments). Even when a distinction is made, there are very few substantive differences—rather, minor adaptations typically reflect different levels of government.*
Similarities between the Baldrige and PHAB programs
The Baldrige program and PHAB have several striking similarities. The overall assessment framework of each is composed of 4 different levels (see the example provided in Table 1). In addition, both programs use teams of trained individuals to conduct the assessment process, site visit, and report development. Most importantly, both processes generate a description of opportunities for improvement to drive further improvement efforts. This is true regardless of whether Baldrige recognition or PHAB accreditation status is achieved by an applicant organization.
Differences between the 2 programs
In addition to the programs' similarities, it is also important to understand their differences. An organization seeking Baldrige recognition selects the version of the criteria most applicable to their work, while PHAB mandates that applicants adhere to the measures specifically designated for their level of government. While only a few organizations ever receive Baldrige Award recognition, if the initial trend continues (14 health departments have been accredited since the program's first recognitions in March 2013),14 it is anticipated that many public health agencies will attain PHAB accreditation. While a number of states have Baldrige-based programs that provide various recognition and learning opportunities, PHAB is a national-level program only at this time (although 3 states—Michigan, Missouri, and North Carolina—also have a public health agency accreditation program, all of which are specifically for local public health departments).15–17 Furthermore, the Baldrige program requires information in a narrative format not to exceed 50 pages, while PHAB requires specific documents. The Baldrige criteria are generic and PHAB's standards and measures are specific to public health. Each Baldrige area to address has several corresponding questions that cover a range of related issues, whereas PHAB's measures typically require 1 to 2 pieces of documentation with a more narrow focus; when making comparisons, therefore, one area to address in Baldrige may include questions that are each discretely addressed by a single PHAB measure.
Creating The Quest for Exceptional Performance tool
Establishing a framework to support meaningful comparisons between the Baldrige and PHAB programs involved characterizing similarities and differences between the programs' key elements, selecting the most appropriate version of the Baldrige program for health departments, completing an inventory of the PHAB measures, identifying the most suitable units of comparison, and determining how to capture and rate degrees of alignment.
The key components of each program were identified and described, including the evaluation system, required documentation, site visit, results, use of results, and periodicity of assessments. This information, coupled with a summary of the programs' respective histories and missions, provided context for the tool's users.
The 2011-2012 Business/Nonprofit version of the Baldrige program was selected for this effort, given its stated applicability to government agencies. The inventory of PHAB Standards and Measures, Version 1.0 yielded a list of 108 measures. The units of comparison were the 67 Baldrige areas to address* and the 108 PHAB measures.
“Alignment” was defined as similarity between the activities each program was seeking to measure. The following definitions were used to rate the degree of alignment:
- High: The intent and/or language of the Baldrige area to address and the PHAB measure are nearly identical.
- Medium: The intent and/or language of the Baldrige area to address and the PHAB measure are similar.
- Low: There is some overlap in the intent and/or language of the Baldrige area to address and the PHAB measure.
- Blank cell: No degree of alignment.
Alignments and their ratings were determined by 2 Baldrige examiners, both of whom have worked with public health departments on their Baldrige applications and one of whom was directly involved in the development of PHAB standards and measures and assisted in orienting the other examiner to the PHAB program. The Baldrige examiner with PHAB expertise systematically developed and populated the crosswalk, that is, a chart in numerical order of the PHAB domains* (see Table 2). This entailed identifying, measure by measure, any corresponding Baldrige areas to address and assigning a rating to each. No limit was placed on the number of corresponding alignments for each unit, so some PHAB measures were aligned with more than 1 Baldrige area to address. The other Baldrige examiner reviewed the completed crosswalk to make independent determinations about alignments and their ratings. The examiners then discussed discrepancies in alignments and discordant ratings and achieved final consensus through discussion.
The crosswalk was finalized after incorporating feedback from a 3-member advisory group comprising 1 state and 2 local public health leaders, all of whom had participated in their respective state-based Baldrige programs. Sample Baldrige applications were collected and included in the final tool.
Analyzing the alignments
The analysis investigated the strongest connections between the Baldrige and PHAB programs. This was measured by both the frequency and strength of alignments between the programs.
Frequency was calculated for each PHAB domain, on the basis of the percentage of measures in that domain with an alignment with a Baldrige area to address. The denominator was the number of measures† in the domain, and the numerator was the number of measures in the domain that had at least 1 alignment. Then frequency was calculated for each Baldrige category. The denominator was the number of measures in the analysis, and the numerator was the number of measures in the category that had at least 1 alignment.
Strength was calculated by assigning a weight to each alignment's rating, wherein high = 3, medium = 2, low = 1, and no alignment = 0. A sum of the weights was generated for each possible domain-category pair. For example, between PHAB Domain 1 (Assessment) and Baldrige Category 4 (Measurement and Analysis), there were 12 total alignments: 2 rated as high, 4 rated as medium, and 6 rated as low. The value of this domain-category pair was 20, which equals the summed weights of these alignments.
The average strength of alignment was then calculated for each PHAB domain. The denominator was the number of aligned measures across the domain, and the numerator was the total of the domain-category pairs' summed weights of alignment across the domain. The average strength of alignment was then calculated for each Baldrige category. The denominator was the number of aligned measures across the category, and the numerator was the total of the domain-category pairs' summed weights of alignment across the category.
The strongest connections were identified as the highest 2 scores of strength of alignment by domain-category pair, frequency across a domain, frequency across a category, average strength of alignment across a domain, and average strength of alignment across a category.
A total of 123 alignments were identified in the PHAB crosswalk. As shown in Table 3, the highest frequencies were seen in the following PHAB domains: Workforce (Domain 8), Evaluate and Improve (Domain 9), and Administration and Management (Domain 11), followed by Assessment (Domain 1). In the Baldrige program, the highest degrees of frequency were in Measurement and Analysis (Category 4) and Customer Focus (Category 3).
Regarding strength, of the 123 alignments, 39 were rated as high, 40 as medium, and 44 as low. The highest summed weights of alignment were seen in the PHAB Domain 5 (Policies and Plans)–Baldrige Category 2 (Strategic Planning) pair and the PHAB Domain 1 (Assessment)–Baldrige Domain 4 (Measurement and Analysis) pair. The greatest average strength was seen across PHAB Domains 5 (Policies and Plans), 9 (Evaluation and Improvement), and 11 (Administration and Management). The 2 greatest average strengths for the Baldrige categories were categories 2 (Strategic Planning) and 5 (Workforce).
The most substantial connections between the 2 programs are best understood within the context of both the frequency and strength of alignments. First and foremost, PHAB's Domain 9 (Evaluation and Improvement) is most closely aligned with the Baldrige program overall. Every PHAB measure in this domain has at least 1 alignment, every Baldrige category has at least 2 alignments with this domain, and the domain boasts the highest average strength of alignments. This considerable connection is a reflection of PHAB's intent to serve as a platform for performance and quality improvement.
Another significant connection can be found around the planning activities seen in PHAB Domain 5 (Policies and Plans) and Baldrige Category 2 (Strategic Planning). Although the frequency of alignments between them is among the lowest, the strength of the connections is the highest. While mostly attributed to the respective programs' focus on strategic plans, PHAB's requisite community health improvement plan and performance management system contribute some additional strong alignments—specifically with the “performance measurement” area to address contained in Baldrige Category 2.
Alignment of workforce development issues is best illustrated by collectively considering the results of the analysis for PHAB Domains 8 (Workforce) and 11 (Administration and Management) and their alignment to Baldrige Category 5 (Workforce). The intuitive connection between Domain 8 and Category 5 is borne out by the high frequency and average strength of the alignments, respectively. Moreover, Domain 11 includes measures about human resource systems, further strengthening connections in this area.
Assessment and monitoring activities present another noteworthy connection between the programs. The focus of PHAB Domain 1 (Assessment) on assessment provides the strongest single alignment to Baldrige Category 4 (Measurement and Analysis). Moreover, implementing the community health improvement plan and strategic plan as captured in Domain 5, and using a performance management system as per Domain 9 measures, are additional activities that enhance the connection to Category 4.
Efforts specifically geared toward understanding and addressing customer satisfaction offer another strong connection between the PHAB and Baldrige programs. Work required by PHAB under Domains 3 (Inform and Educate) and 4 (Engage with the Community), in particular, have the strongest alignments with Category 3 (Customer Satisfaction). Customer satisfaction activities described in Domain 9 are another strong point of connection. PHAB's attention to aspects of customer satisfaction reflect the nature of health departments as public service entities, and only 3 PHAB domains (planning, workforce, and governance) have no alignment with Category 3.
Finally, it was noted that 3 concepts within the Baldrige criteria are not captured within the scope of the PHAB standards and measures. The first is the Baldrige program's emphasis on guiding innovation. In addition, Baldrige requires performance comparisons and benchmarking with other high-performing organizations to identify best practices and help set performance improvement targets. Finally, Baldrige places a very strong emphasis on how the organization's processes (described in Baldrige categories 1-6) affect the organization's performance results (described in Baldrige category 7).
While the connections identified through the process of developing the crosswalks and engaging in the analysis are straightforward and intuitive to a large degree, the limitations of this work rest on its theoretical basis, and the lack of testing its application. Future analyses may be possible if a sufficient number of health departments seek Baldrige recognition and/or use Baldrige as a platform for a performance management system.
The strongest connections between PHAB standards and measures and the Baldrige criteria are in the areas of performance management, quality improvement, strategic planning, workforce development, assessment and analysis, and customer service. For health departments seeking Baldrige recognition, the crosswalk can facilitate the identification of operations that would be appropriate for responding to the questions posed by the Baldrige criteria. For health departments building a performance management system, the crosswalk can identify operations that could be assessed according to the questions posed in the corresponding Baldrige areas to address.
As part of PHAB's commitment to engage in its own ongoing quality improvement efforts,18 it may be useful for PHAB to examine how it relates to the well-established and nationally recognized Baldrige program. In the process, PHAB might consider including measures that address innovation, benchmarking against high-performing public health agencies, and results in future versions of its standards and measures.
1. Madamala K, Sellers K, Beitsch LM, Pearsol J, Jarris P. Quality improvement and accreditation readiness in state public health agencies. J Public Health Manag Pract. 2012; 18:(1):9–18.
2. Joly BM, Booth M, Shaler G, Mittal P. Assessing quality improvement in local health departments: results from the multi-state learning collaborative. J Public Health Manag Pract. 2012; 18:(1):79–86.
3. Corso LC, Lenaway D, Beitsch LM, Landrum LB, Deutsch H. The national public health performance standards: driving quality improvement in public health systems. J Public Health Manag Pract. 2010; 16:(1):19–23.
4. Madamala K, Sellers K, Pearsol J, Dickey M, Jarris P. State landscape in public health planning and quality improvement: results of the ASTHO survey. J Public Health Manag Pract. 2010; 16:(1):32–38.
5. Beitsch LM, Leep C, Shah G, Brooks RG, Pestronk RM. Quality improvement in local health departments: results of the NACCHO 2008 survey. J Public Health Manag Pract. 2010; 16:(1):49–54.
6. Berryman P, Solomon J, Gorenflo G. Driving quality improvement in local public health practice. J Public Health Manag Pract. 2010; 16:(1):67–71.
* The few exceptions are 5 measures specific to state applicants, and one measure each specific to tribal applicants, tribal and state applicants, and tribal and local applicants.Cited Here...
* Category 7 (Results) areas to address were not included in the analysis, as although some were deemed to have a level of alignment, the alignment did not include specific measures but rather was described in a very general way.Cited Here...
† While 108 measures were used in the crosswalks, 104 measures were used in the analysis. The 4 that were excluded were deemed to be duplicative of other measures.Cited Here...