Some of the most important training tools used in the field of emergency management, public health preparedness (PHP), and hospital preparedness are operations-based drills, functional exercises, and full-scale exercises.1–4 Exercises are simulated incidents in which agencies and individuals respond as though the incidents are real and actually occurring.5,6 Mock incidents allow agencies and organizations to validate response plans, policies, and procedures2,4,6; clarify organizational internal and external roles and responsibilities4,5; determine how well independent response systems integrate2; identify resource gaps in the operational environment2,5,6; and test agreements with other public and private entities.2,4–6 Accordingly, exercises help identify areas for improvement linked to performance standards7 within PHP8 and health care preparedness (HCP) capabilities.9 Recommended improvements may then be incorporated into public health emergency preparedness and hospital preparedness program work plans.
The Department of Homeland Security's (DHS) Homeland Security Exercise and Evaluation Program (HSEEP) provides a framework for both the design and evaluation of drills and exercises.10 Furthermore, HSEEP guidelines provide exercise designers and participating agencies with tools for systematically thinking through the exercise planning process. As exercises increase in complexity—with additional DHS core capabilities and PHP/HCP capabilities assessed via large-scale, multisite exercises involving an increasing number of agencies over a long duration—there are many complications in applying the HSEEP framework. The HSEEP framework and its related templates are not designed to manage the development and evaluation of multiple, operations-based, parallel exercises combined into 1 complex large-scale event.
The South Central Preparedness and Emergency Response Learning Center (PERLC) at the University of Alabama at Birmingham School of Public Health, in support of the Mississippi State Department of Health (MSDH) and The University of Mississippi Medical Center, has been working on applying HSEEP guidelines to complex multiorganizational, parallel exercises. The lessons learned from the planning of Mississippi's Emergency Support Function-8 (ESF-8) involvement in National Level Exercise 2011 (NLE 2011) were used to develop an expanded exercise planning model that not only utilizes HSEEP guidance but also accounts for increased complexity and is more functional for public health. The model was tested and refined during the planning and evaluation of Mississippi's State-level ESF-8 exercises in 2012 and 2013.
This article describes the Expanded HSEEP (E-HSEEP) model for use in the design of complex exercises where multiple response functions and capabilities are tested. Furthermore, this article demonstrates that E-HEEP facilitates the development of a comprehensive evaluation plan and shows that areas for improvement are more easily linked to PHP and HCP capabilities to better provide information that can be used to formulate future public health emergency preparedness and hospital preparedness program work plans.
The South Central PERLC is a Centers for Disease Control and Prevention–funded entity comprising partners from the Schools of Public Health at the University of Alabama at Birmingham and the Tulane University. The South Central PERLC is tasked with addressing competency-based education and training needs of the public health agencies in Alabama, Louisiana, and Mississippi. As part of its mission, beginning in October 2010, the South Central PERLC collaborated with the MSDH in the planning, execution, and evaluation of Mississippi's ESF-8 segments of the NLE 2011. The scenario was to respond to a 7.7 magnitude earthquake on the New Madrid Seismic Zone, affecting multiple states and seriously devastating the northern region of Mississippi. The ESF-8 segment of NLE 2011 focused primarily on the state's overall management of mass casualties and fatalities, as well as on the state's ability to shelter those with special medical needs.
The initial strategy for the MSDH's NLE 2011 planning was to employ HSEEP guidelines at all exercise stages to assess lessons learned, corrective actions, and other insights. The HSEEP provides a standardized methodology and terminology for exercise design, development, conduct, evaluation, and improvement planning.10 However, in attempting to retain HSEEP logic, it was determined that the framework was inadequate for the management of multiple parallel exercises that were combined into 1 complex, large-scale, “hybrid” operations-based exercise. Hybrid operation-based exercises include multiple types of operations-based exercises (eg, drills, functional exercises, and full-scale exercises) that test multiple PHP response functions and capabilities (specifically those outlined in the exercise objective priorities listed later). Therefore, the exercise planning team conducted a systematic analysis of and made modifications to the existing HSEEP framework for the purposes of NLE 2011 planning and in doing so, created an expanded exercise planning model that has been found to be beneficial for planning and executing similar complex large-scale public health exercises and in evaluating responses to actual public health emergencies.
National Level Exercise 2011 provided MSDH with an opportunity to test multiple components and annexes of the MSDH Concept of Operations Plan and the ESF-8 infrastructure in response to mass casualties and fatalities anticipated by a 7.7 magnitude earthquake. The NLE 2011 Exercise Plan defined 11 objectives deemed appropriate for evaluating public health emergency response procedures, identifying areas for improvement and achieving a collaborative attitude. The overall exercise objectives were aligned with DHS target capabilities (now core capabilities) and focused on the following priorities: (1) activation and deployment of state medical assistance teams; (2) activation and deployment of disaster transport assets including the activation of emergency medical service memorandums of understanding; (3) coordination of disaster transport assets for patient movement; (4) activation and use of the Mississippi patient tracking system to monitor patient movement; (5) adoption of effective command and control structure within the Public Health Coordination/Command Center and across operational units; (6) demonstration of effectiveness of intersite communications between operational units and Public Health Coordination/Command Center; (7) activation of emergency management assistance compact and deployment and integration of its resources into current operations; (8) activation of federal resources to provide medical supplies, equipment, and prophylaxis through the Strategic National Stockpile and the coordination of their subsequent distribution; (9) activation and operation of special medical needs shelters; (10) activation and deployment of state mortuary response teams; and (11) coordination of medical surge and mass casualty resources with Mississippi's hospitals. At the time of NLE 2011 planning, PHP capabilities had not yet been released by the Centers for Disease Control and Prevention and the HCP capabilities had not yet been released by the Department of Health and Human Services. However, plans for 2012 and 2013 state-level ESF-8 exercises in Mississippi included the alignment of exercise objectives with PHP and HCP capabilities (as well as their associated functions and tasks) to facilitate the prioritization of corrective actions and lessons learned. This allowed prioritized findings to be easily incorporated into public health emergency preparedness and hospital preparedness program work plans.
The E-HSEEP model was developed through changes in the HSEEP exercise planning process in the areas of exercise plan development, controller/evaluator handbook, evaluation plan, and after action report and improvement plan.
Exercise plan development
The Exercise Plan (ExPlan) template provided by HSEEP includes 4 chapters: “General Information,” “Exercise Logistics,” “Player Information and Guidance,” and “Evaluation and Postexercise Activities.” The ExPlan serves as the guiding document for all other HSEEP documents and is the primary source for exercise information. Eleven overall exercise objectives were drafted on the basis of the above listed priorities to guide MSDH's exercise planning for its participation in NLE 2011. These overall exercise objectives were placed in Chapter 1: “General Information” of the ExPlan and guided the planning of the exercise as well as the evaluation.
Although the layout of the ExPlan is logical and user-friendly for small-scale operations-based exercises, the magnitude of the planned ESF-8 play in NLE 2011 and the number of public health operational units and participating agencies did not easily conform to the HSEEP ExPlan layout. The MSDH and ESF-8 partners needed a document detailing all aspects of NLE play with information for participants, controllers, and evaluators of each subexercise. Factors further complicating ExPlan layout included differences in each public health operational unit's scope, duration, and venue of play. To guide exercise planning and evaluation, for each operational unit being tested (or subexercise), function-specific objectives were created that aligned to 1 or more of the overall exercise objectives (see the Figure).
Some of the operational units required multiple days of play to achieve their specific objectives, while others would require only a day or even just a part of day. Some of the subexercises were located at a single venue while others were spread over multiple venues. The exercise planning team decided that the best approach would be to introduce separate chapters in the ExPlan for each functional area as guided by the overall exercise objectives, combining elements of the existing HSEEP ExPlan Chapters 2 to 4 into a single chapter for each function. This approach reflected each function's subexercise within the larger exercise (see the Table).
The first 3 chapters maintained the same titles as HSEEP's Chapters 1 to 3 but only contained content common to all functional areas and operational units involved in the exercise. Beginning with Chapter 4, each operational area (ie, command and control; medical assistance teams; disaster transport operations; mortuary response team; receipt, staging, and storage; closed point of dispensing distribution; special medical needs shelter; and hospitals) was then assigned its own chapter containing function-specific information on exercise play, applicable capabilities, and function-specific objectives, exercise setup information, directions, access restrictions, rules for each exercise venue, and information on the Hot Washes to be held at each venue at the end of each operational period. Some functional areas were further broken into subareas on the basis of exercise participation. For example, as shown in the Figure, “Command and Control” chapter was divided into operational units responsible for command and control each with its own function-specific objectives including the ESF-8 desk at the State Emergency Operations Center, the Public Health Coordination/Command Center, the State Emergency Response Team (a forward Emergency Operations Center with an ESF-8 component), and a public health district command center. The final chapter of the ExPlan mirrored HSEEP's Chapter 4 and provided exercise-wide information on evaluation and postexercise activities.
The Controller/Evaluator (C/E) Handbook is a document that provides additional information and instructions to those who will be responsible for overseeing the exercise and evaluating the players' actions. For the ESF-8 play in NLE 2011, the planning and evaluation team designed the evaluation and, as a result, all evaluation materials to be organized according to the functional areas of exercise play. Therefore, an appropriate number of controllers and evaluators were assigned to each functional area of each subexercise. As multiple subexercises may be occurring simultaneously at a given venue, each controller and evaluator was issued exercise evaluation guides (EEGs), participant feedback forms (PFFs), and Hot Wash materials/instructions specific to his or her assigned functional area. To keep the paperwork organized, PFFs and Hot Wash materials were coded on the basis of the function, venue, and period of exercise play.
The Expanded HSEEP EEGs were developed from standard HSEEP templates provided for each of the identified DHS target (now core) capabilities. Customization was conducted for the overall exercise and each of the activated functional areas. The EEGs were tailored to remove extraneous activities outside the stated exercise objectives. Within each activated functional area, activities and tasks were further modified to capture observations specific to each exercise venue and operational period. The design team then consulted agency plans to ensure stated performance measures and metrics were included (eg, open point of dispensing within 60 minutes of health officer notification).
In state-level exercises conducted in 2012 and 2013, the EEGs were tailored to include the functions and task of the PHP and HCP capabilities that aligned with the exercise objectives. This alignment allowed specific performance measures pulled out of the PHP and HCP capabilities to be priority areas of observations for exercise evaluators. This information was captured and then used by the state health department in reporting to the Centers for Disease Control and Prevention and the Health and Human Services.
In addition to EEGs, the exercise planning and evaluation team prepared the PFFs to be distributed to players, controllers, evaluators, and observers. Participant feedback forms were distributed at the end of each segment of exercise play at each venue, resulting in considerable breadth and depth of feedback to capture observations specific to multiple objectives. Existing HSEEP PFF templates were utilized, however, with modifications. First, PFFs were created for each venue active during a given operational period. The venue, specific location of play, date, and operational period were added and prepopulated to ensure that they could be easily organized.
Given the complex nature of the exercise and the large number of functional level objectives to be measured across multiple sites, evaluation design had to be carefully planned to best allocate the evaluation staff. Accordingly, the exercise planning team worked with participating agencies to establish an exercise schedule matrix and determine the number of evaluators necessary for each of the exercise venues to ensure adequate staffing. Staffing assignments and individual packets containing the Master Scenario Events List, EEGs, C/E Handbook, and PFFs for each operational period were prepared for each evaluator.
Similarly, planners determined that 1 Hot Wash would not be sufficient for capturing participant feedback because of the number and variety of activated functions, the total number of players, and number of part-time players. Therefore, to capture the most accurate, targeted information, it was determined that mini-Hot Washes would be conducted at each venue at the end of each operational period. Participant feedback forms were designed to be distributed and collected at each mini–Hot Wash, with a larger Hot Wash held at the end. Finally, planning team, controller, and evaluator debriefs were conducted.
After action report and improvement plan
The final HSEEP element for the exercise is the After Action Report and Improvement Plan (AAR/IP). The AAR/IP is the final product for the exercise, detailing the outcomes of the exercise, identifying strengths and areas for improvement, and setting forth a plan and corrective actions for addressing any identified areas for improvement.
The evaluation team chose to once again modify the HSEEP's template to meet the scaling needs of the ESF-8 play in NLE 2011. Slight modifications were made to the executive summary and Section 1. The executive summary was expanded beyond the typical 3 strengths and 3 areas for improvement because of the large-scale and numerous functions to be activated during the exercise. Section 1, which provides exercise details, was modified to include the list of activated functions. The primary modifications were to Section 3, the analysis of capabilities. Rather than organize this section by target capability, the evaluation team felt that organizing Section 3 by function, as laid out in the ExPlan, would separate the larger exercise into a series of function-specific subexercises and enhance the utility of the report for MSDH and ESF-8 partners. The titles of the function-specific areas may be aligned with PHP and/or HCP capabilities at this level and still reflect the target capabilities selected for that function's subexercise. This alignment allowed the activities to remain in sync with the EEGs and for the analysis to proceed as outlined by HSEEP. As a result, this structure also followed through to the IP matrix, to allow those involved in each function to extract the section of the report and IP matrix relevant to their participation in the exercise, and still allowing the MSDH and ESF-8 partners to have a complete, comprehensive AAR/IP.
After the extensive NLE 2011 planning, actual public health emergencies occurred in Mississippi within weeks of scheduled play, preventing the MSDH and its ESF-8 partners from participating. However, the South Central PERLC was able to successfully apply the lessons learned from NLE 2011 planning to the evaluation of the Mississippi ESF-8 responses to the April 2011 tornado outbreak and the May 2011 Mississippi River flood events. As was planned for the evaluation of NLE 2011, function-specific Hot Washes were held with each functional area of the ESF-8 response to both of these events. Then the AAR/IPs for both events were laid out by functional area, addressing the strengths, lessons learned, and areas for improvement identified by participating responders. This organization allowed for the identification of 116 corrective actions across the tornado response and the flooding events that were then prioritized and considered when deciding on future actions to improve ESF-8 response capabilities.
Subsequent state-level exercises conducted in 2012 and 2013 also utilized the E-HSEEP exercise framework, successfully enabling MSDH to capitalize on its investment in conducting large-scale exercises. The MSDH was able to capture strengths, lessons learned, and areas for improvement and identify microlevel issues that may have been missed using the traditional HSEEP framework. For example, in 2012, more than 90 corrective actions were identified from an exercise that focused on the activation, mobilization, operations, and demobilization of 2 state medical assistance teams, a forward assessment and scene triage team, and a mobile field hospital unit. Furthermore, in 2013, more than 120 corrective actions were identified from an exercise that tested MSDH's volunteer management system and the mobilization and operations of a receipt, staging, and storage site; an open point of dispensing site; a community reception center site; a field mobile hospital site staffed by 2 medical assistance teams; and command and control between these sites.
Exercises are expensive and time-consuming, yet essential to the development of emergency response plans and capabilities. The E-HSEEP model for exercise planning and evaluation allowed MSDH to maximize the benefits of conducting exercises by examining issues of response that occurred at not only the macrolevel but also at the microlevel. Complex exercises create an opportunity to examine how each part functions not only independently but within the framework of a large-scale emergency.
In developing E-HSEEP, each of the essential HSEEP documents (ie, the ExPlan, C/E Handbook and materials including the Evaluation Plan, EEGs, feedback and Hot Wash forms, and the AAR/IP) was organized allowing for a more systematic and logical evaluation of the ESF-8 play and subsequently allowing for results specific to each of the multiple departments and agencies, or objectives, to be easily extracted from the final AAR/IP. In addition to increasing its utility and user-friendliness, the decision to lay out the ExPlan with a function-specific chapter for each operational area of exercise play improved exercise evaluation.
The South Central PERLC is creating an Excel-based E-HSEEP tool that will allow practice partners to track corrective actions. Corrective actions can be categorized by year and event type, and, by using the PHP and HCP capabilities, each corrective action can be assigned to a capability and any corresponding function(s), task(s), and resource element(s). In addition, each corrective action can be assigned a macrotheme and subthemes, which gives a verbal depiction of the main theme of the corrective action. The macrotheme represents broadest issue of the corrective action. For example, macrothemes can include staffing, training, policies/procedures, external partnerships, internal organizational issues, and equipment/resource needs. Subthemes provide a more detailed description of the corrective action to help define focus.
Once the information has been compiled, data will be analyzed using Excel filters and pivot tables to examine results and trends. This E-HSEEP tool will provide the ability to conduct many different types of analyses and comparisons. For example, comparisons can be drawn between actual events and exercises to identify any common trends in themes or in PHP/HCP capabilities being utilized. Results from past exercises can be compared to determine whether trends exist in the issues being identified and may help identify gaps in what is being tested and/or evaluated. In addition, exercises/events can be analyzed by year to examine trends in themes (eg, are the same themes an area for improvement each year?). As additional exercises and events occur, more information is entered into this tool, providing state partners a database of corrective actions.