Insights From Evaluation of Practice Experience
As called for in the National Health Security Strategy (NHSS), health departments currently employ “after action reports” to collect data on their experiences in actual public health emergency responses. The 2009 H1N1 pandemic revealed significant weaknesses in the use of such reports, especially in analyzing root cause and performance in terms of public health emergency preparedness (PHEP) capabilities.1 To address these deficiencies and to develop an effective, generalizable approach to learning from actual public health emergencies, we sought to understand how the concept of a “critical incident registry” (CIR), commonly used in health care and other industries, could be adapted to meet the NHSS call for new quality improvement methods and the needs of public health care practitioners.
We conducted a workshop with public health systems researchers and public health care practitioners in 2011, reviewed the literature on qualitative assessments of systems improvement, and examined CIR examples in other industries.2
A CIR uses a standard protocol to gather and analyze information about critical incidents that reveal a system's vulnerability. Collecting individual reports allows for cross-case analysis through descriptive studies of events entered, analytical studies of the relationship between contexts and mechanisms, and the identification of best practices in developing protocols. As an example of the potential place for a CIR, the 2009 H1N1 pandemic required the collaborative efforts of numerous partners; a root-cause analysis of this experience, based primarily on the state and local experience in Massachusetts, illustrates the importance of developing approaches across partners for managing when vaccine supply is uncertain and distributed over time, balancing clear and precise policies with flexible implementation.3 Furthermore, the review of the literature on qualitative assessments systems improvement research showed that when the focus is on improvement rather than accountability, qualitative assessment of system capabilities can be more useful than quantitative assessments. Examples from the aviation and other industries reflect the usefulness of qualitative assessments entered into a CIR specifically. Ensuring that such assessments are rigorous can be challenging, but a well-established body of social science methods provides a useful approach.4,5 The workshop and literature review identified several key critical characteristics needed for a PHEP CIR to be feasible and useful: (1) the scope of a CIR must include incidents that include a sufficient role for public health agencies, are “meaningful,” test 1 or more PHEP capabilities, and are sufficiently limited to isolate the issues; (2) the structure of the CIR should facilitate analysis of individual incidents as well as support cross-case analysis; (3) a framework for rigorous analysis of individual incidents and methods for cross-case analysis must frame the CIR; and (4) incentives to reporting include organizational improvement and systems learning, but these incentives must be made explicit so as to overcome disincentives in reporting.
A CIR could address 3 important goals in PHEP: organizational learning through analysis of individual incidents, sharing reliable lessons between like health departments, and providing a database for cross-event analysis for contextual relationships and best practices. To be effective, standard protocols are needed for analysis and reporting of critical events. Critical incident registries provide a means for public health care practitioners to review adverse events that test public health capabilities. Development of a PHEP CIR would provide a platform to more deeply analyze these events, understand how responses can be improved, and disseminate learning to other organizations.
1. Centers for Disease Control and Prevention, Office of Public Health Planning and Response. Public Health Preparedness Capabilities: National Standards for State and Local Planning. Atlanta, GA: Centers for Disease Control and Prevention; 2011.
2. Stoto MA, Kraemer JD, Piltch-Loeb R. A public health preparedness critical incident registry (CIR). Submitted. 2013.
3. Higdon MA, Stoto MA. The Martha's Vineyard Public Health System responds to 2009 H1N1. Int Public Health J. 2013;5(4). In press.
4. Gilson L, Hanson K, Sheikh K, Agyepong IA, Ssengooba F, Bennett S. Building the field of health policy and systems research: social science matters. PLoS Med. 2011;8:e1001079.
5. Yin R. Case Study Research: Design and Methods. Thousand Oaks, CA: Sage; 2009.