Secondary Logo

Journal Logo

More Than a Tick Box: Medical Checklist Development, Design, and Use

Burian, Barbara K. PhD*; Clebone, Anna MD; Dismukes, Key PhD*; Ruskin, Keith J. MD

doi: 10.1213/ANE.0000000000002286
Healthcare Economics, Policy, and Organization
Free

Despite improving patient safety in some perioperative settings, some checklists are not living up to their potential and complaints of “checklist fatigue” and outright rejection of checklists are growing. Problems reported often concern human factors: poor design, inadequate introduction and training, duplication with other safety checks, poor integration with existing workflow, and cultural barriers. Each medical setting—such as an operating room or a critical care unit—and different clinical needs—such as a shift handover or critical event response—require a different checklist design. One size will not fit all, and checklists must be built around the structure of medical teams and the flow of their work in those settings. Useful guidance can be found in the literature; however, to date, no integrated and comprehensive framework exists to guide development and design of checklists to be effective and harmonious with the flow of medical and perioperative tasks. We propose such a framework organized around the 5 stages of the checklist life cycle: (1) conception, (2) determination of content and design, (3) testing and validation, (4) induction, training, and implementation, and (5) ongoing evaluation, revision, and possible retirement. We also illustrate one way in which the design of checklists can better match user needs in specific perioperative settings (in this case, the operating room during critical events). Medical checklists will only live up to their potential to improve the quality of patient care if their development is improved and their designs are tailored to the specific needs of the users and the environments in which they are used.

Published ahead of print July 28, 2017.

From the *Human Systems Integration Division, NASA Ames Research Center, Moffett Field, California

Department of Anesthesia and Critical Care University of Chicago, Chicago, Illinois.

Published ahead of print July 28, 2017.

Accepted for publication May 17, 2017.

Funding: None.

The authors declare no conflicts of interest.

Reprints will not be available from the authors.

Address correspondence to Barbara K. Burian, PhD, Human Systems Integration Division, NASA Ames Research Center, Mail Stop 262-4, Moffett Field, CA 94035. Address e-mail to Barbara.K.Burian@nasa.gov.

Few clinicians appear to be undecided when it comes to medical checklists—these tools tend to be liked and embraced or disliked and avoided. Despite their demonstrated efficacy in some medical settings,1–7 perioperative checklists have yet to deliver many of their promised benefits,8–12 and some medical professionals complain of growing “checklist fatigue” and reject them outright.13,14 This can be attributed, at least in part, to poor design and content2,12,15–19; inadequate introduction and training20,21; unenthusiastic or incomplete application9,22–24; duplication with other safety checks10; poor integration with existing work flow11,12,25; and professional, institutional, and national cultural barriers.8,13,14,26,27 Most of these issues relate to the human factors of checklist development, design, and use.28,29 Checklists have a life cycle. They are conceived, designed, tested and evaluated, implemented, and revised or, when necessary, retired. Each of these phases requires critical decisions and actions; neglecting any of them will result in an ineffective checklist that users will dislike and ultimately ignore or discard.

Although the existing literature provides some extremely useful suggestions,2,16,18,30–38 there is currently no comprehensive, integrated framework to guide the development and design of robust, effective medical checklists. For example, both Project Check and the Healthcare Financial Management Association (HFMA) provide “checklists” for developing medical checklists.35,36 Both have their roots in the development of the World Health Organization (WHO) Surgical Safety Checklist and are therefore only partially relevant to non “time-out” checklists (eg, shift handovers, critical event response in operating rooms or intensive care units). Guidance is often limited in scope or broadly stated (eg, “Use simple sentence structure and basic language”)36 and often has not been updated to reflect recent research regarding human performance and cognitive psychology related to checklist design and use.19,39

Nonetheless, these resources provide a valuable starting point for considering checklist development and design. This article builds on this guidance to provide a comprehensive blueprint for the development or revision of medical checklists.a It is organized around 5 stages that should always be part of the checklist life cycle: (1) conception, (2) content determination and design, (3) testing and validation, (4) training and implementation, and (5) ongoing evaluation, revision, and possible retirement. Our blueprint incorporates important advances in the understanding of human performance, cognition, and team behavior to help advance the science of medical checklist development, design, and use to improve patient care.

The “checklist” label has been applied to everything from a tool that helps physicians figure tax deductions,47 to a guide for manuscript peer-review,48 to an algorithm for resuscitation during cardiopulmonary arrest,49 to the WHO Surgical Safety Checklist.32,50 We reviewed articles in the PubMed database published in the past 5 years to determine the degree to which recent medical literature addresses the checklist development life cycle, but limited our review to only checklists used in clinical practice. Search terms of article titles included “checklist” AND “develop*” (develop, developed, developing, development; N = 124), “checklist” AND “design*” (design, designed, designing; N = 15), and “checklist” AND “checking” (N = 2). Of these 141 articles, 50 were discarded (38 nonclinical checklist or non-English articles, 2 duplicates, 9 abstract only, 1 no abstract nor article available) leaving 91 articles that were subjected to a full text review. Thirty of these articles are specifically relevant to anesthesiologists: perioperative period (n = 11), critical care (n = 11), obstetric anesthesia (n = 4), and pain management (n = 4). Further findings are reported below.

Back to Top | Article Outline

DEVELOPING EFFECTIVE MEDICAL CHECKLISTS

First, we must clarify some terminology. Checklist development and design are quite different concepts and should not be used interchangeably. Development refers to the entire checklist life cycle—conception, design, evaluation, use, modification, and termination. Design is one phase of development (though highly interrelated to other phases) and pertains specifically to checklist appearance, format, layout, and functioning.

Similar confusion exists in the medical literature about checklist accomplishment methods (Do and Confirm, Read and Do, Flow), the formatting of action items (challenge-response, imperative statement), means of accomplishment (read aloud or silently),12,51 and the number of individuals involved during checklist accomplishment (1, 2, or more). “Do and Confirm” refers to checklists used to verify, after the fact, that certain actions have been performed or information has been gathered or recorded. “Read and Do” pertains to checklists used to guide actions step-by-step, in real time. (Some checklists, such as the WHO Surgical Safety Checklist,50 combine elements of both.) The “Flow” accomplishment method is similar, in practice, to “Read and Do” but pertains to accomplishment of embodied checklists.52,b Written action items can be formatted in the challenge-response style (Volatile anesthetics…Off) or as imperative statements (Turn off volatile anesthetics). Any of the 3 accomplishment methods and action items formatted in either style can be performed by 1 person (text read silently or aloud) or by 2 or more people (text typically read aloud).

Back to Top | Article Outline

Conception

The first phase of checklist development is conception, which requires recognizing and analyzing a clinical concern to determine what causes and contributes to the problem. This analysis is ideally conducted by a multidisciplinary team that includes clinicians, human factors and patient safety experts, and relevant administrators.53 The team should consider various options rather than simply deciding to construct a checklist,18,54 which might not be the best solution.28–30,38,55,56 This important phase is given scant attention in the literature and possibly also in practice. Only 7 of the 91 articles reviewed mentioned issues associated with conception. If a checklist is the best solution, the development team should not immediately begin writing down checklist items, but should start instead by thoroughly exploring the checklist’s purpose1 and desired outcomes (see Figure 1).

Figure 1.

Figure 1.

Although diverse checklist taxonomies have been proposed,2,15,16 medical checklists can be most simply organized by the 3 overarching circumstances in which they are used51: (a) normal situations (eg, time-out checklists,50 equipment setup,18,57 shift handover,58 procedure completion,52 treatment guidance1); (b) emergencies (eg, critical event checklists)4,59; or (c) atypical situations that are not a part of usual daily practice but that also are not critical events (eg, a broken surgical instrument). In our review, 89 articles pertained to normal checklists, 2 involved critical event/emergency checklists, and none addressed atypical situations.

During the conception phase, developers must also consider how the planned checklist will be used (see Table 1), how it will fit into and affect existing workflow (minimal disruption is essential),11,12,25 and how it will relate to other existing checklists or safety checks.10,27 In what physical environment will it be used and who will participate in its accomplishment18 and why? What presentation mode (eg, paper, electronic)14,60 will best allow the purposes and goals of the checklist to be fulfilled?

Table 1.

Table 1.

After these questions have been answered, the development team should revisit the question of whether a checklist is the best tool to address the problem. Consistent and proper use of checklists changes how a task is accomplished by imposing standardization that did not previously exist. If standardization or a memory or decision-making aid is not needed or is already in place, then a checklist is unlikely to be a good solution.25,61

Back to Top | Article Outline

Determining Content and Design

Literature reviews, focus groups, Delphi consensus,62 task analyses,18 subject matter interviews, and personal experience are well-established methods for identifying appropriate content. Seventy-five of 91 articles reviewed reported using 2 or more of these approaches. The amount of information and level of detail to include in checklists are among the most difficult issues to resolve during development; choices are driven by competing considerations.19,31 Highly experienced professionals may need fewer prompts and less information than more junior colleagues; under stress or high workload, however, even the most seasoned professionals experience a decrease in performance1,25,63–65 and may forget an important step. We believe that a good rule of thumb is to gear the level of detail toward the most junior clinician expected to practice independently, and thus use the checklist without supervision—critical event checklists, for example, should be geared toward a new attending physician.

Inappropriate or unnecessary technical content undercut a checklist’s effectiveness.17–19,33,65 Medical checklists designed to ensure that every step has been executed correctly sometimes include inconsequential items or actions that can no longer be performed after the fact. For example, “Prepare the location with towel or padding under patient…” should not be included on a Lumbar Puncture checklist that is intended to be reviewed after completion of the procedure.16 Since checklist use may interrupt workflow,66 content should “earn” its way into the checklist; cost (disrupted workflow) must be compared to benefit (reduced chance of human error).18

Aviation checklists, after which many medical checklists have been patterned,67 may appear simple in their design and use, falsely suggesting that they should be easy to construct and implement in medicine. In fact, they are quite complex and can be exceedingly difficult to design well.17,19,25,51,68 Additionally, medical settings are quite different than cockpits, necessitating medical checklist designs and accomplishment methods that are also quite different (see discussion of “Sampling” in the following section).

We have also observed misapplication of findings from basic cognitive psychology research,69 such as guidance that checklists should contain no more than 5 to 7 items because of limited short-term memory capacity.35,36 Information does not need to be retained in short-term memory when it can be referenced on a checklist, so this restriction is unnecessary. However, checklists should be no longer than absolutely needed.

A checklist’s usability and effectiveness are directly dependent on a multifaceted set of interrelated development and design issues17,19,68; an introduction to these is presented in Table 1 (B. K. Burian, PhD, unpublished document, 2016)17 and Figure 2. For example, the physical properties of a checklist are, in part, determined by how the information is presented (eg, paper size or type of electronic device and size of display). This influences the ease with which the checklist is accessed, its usability under various environmental conditions (eg, in a darkened procedure suite), the options available for content presentation (eg, layout, font size), and how a user navigates through the checklist, progressing from item to item.19,70 Although the design of a medical checklist must be tailored to the setting, its purpose, and intended use, some design fundamentals can be applied to all types of medical checklists. Examples include the use of a san serif font, a clean layout (ie, enough white space to enhance readability), and a logical flow of items and information.35,36

Figure 2.

Figure 2.

The intended purpose and use of the checklist (identified during the conception phase), as well as other considerations such as cost, will guide decisions about the best presentation modality, which, in turn, will affect decisions about formatting and layout of content and the overall design and functionality of the checklist. We suggest that rapid-cycle prototyping and simulation be conducted to evaluate the utility of checklists during this stage, before significant resources are spent in later stages of development.33,53

It is no wonder if teams struggle with checklist design; more research is needed to identify the most effective designs for different checklist purposes and uses.24,71 Of the 75 articles reviewed that described methods used to identify content, only 17 reported addressing design issues.

Back to Top | Article Outline

Crisis Checklists and Sampling.

A checklist’s design must reflect and support its use. For example, physicians may need immediate access to information in crisis checklists to ensure a successful outcome. Some crisis checklists4,59,72 contain design elements implying that users should start at the top and proceed step-by-step,73 such as by placing the word “Start” at the top of a list of items and numbering each step in order.4 As part of an ongoing study, we observed that physicians often do not respond to critical events in such a linear manner. Instead, many first use existing knowledge and refer to a checklist for additional ideas or specific information (eg, drug dosages) only after starting treatment.74 We label this type of checklist use “sampling,” and it represents a distinct departure from the way that critical event checklists are currently used in aviation and other domains. Checklists designed for use in a stepwise fashion generally do not support this kind of specific information-seeking behavior. Checklists for medicine that too closely follow an aviation-oriented design may therefore be ineffective5,75,76 or even disruptive.77

Table 2.

Table 2.

Figure 3.

Figure 3.

We believe that it is possible to design checklists to support both sampling and a step-by-step guided response, including responding to an event already in progress. We have created a novel checklist design to support these multiple uses (see Figure 3 and Table 2). In our design, items associated with different response “phases” are grouped into color-coded blocks. This allows users to jump directly to the needed block when accessing the checklist or to jump to Crisis Management actions at any time, even when in the middle of a different block (eg, Treatment). Drugs and dosages, which are often sampled in these checklists, are repeated in their own block for easy reference. This design is optimized for paper critical event checklists; a different design approach is needed for electronic checklists.14,18,57,60,67,78,79 Research is underway to evaluate the effectiveness of our design.

Back to Top | Article Outline

Testing and Validation

After iterative usability testing and final design completion, a new checklist should ideally be subjected to thorough validation before implementation.33 A sufficient number of teams should use the checklist in either simulation or actual operations to allow statistical analysis of how well the checklist achieves its purpose (eg, reduces the number of errors) and whether it is used by the teams as intended, fits into the flow of work, and is easy to use. Some types of checklists (eg, those used for performance evaluations or as screening instruments) will require other types of psychometric appraisals, such as interrater reliability and criterion and construct validity.80,c

In our review, 54 of 91 articles described psychometric analyses, and 25 reported that pilot testing had been conducted, although often the extent of the pilot testing appeared to be gathering user feedback through surveys. Authors of only 1 of the 10 checklists that were implemented reported having conducted psychometric and usability testing before implementation.81

Back to Top | Article Outline

Training and Implementation

A checklist that simply appears with no introduction, training, or rationale as to why it is needed is unlikely to be accepted.74 Therefore, successful implementation of a checklist8,20–23,82,83 requires an intensive educational process that should include an introduction that informs pertinent staff that a checklist is being developed and explains why and when it will be introduced. This should preferably occur well in advance of training.

Even the best-designed checklist requires training in its use, especially because checklists have not traditionally been part of medical work settings. During training, it is necessary to counteract common myths (eg, checklist use signifies lack of expertise)14 and address user concerns. Training should include the checklist’s purpose, who will use it, how, and when.84 Training should also cover potential barriers to use and how to address these roadblocks when they occur.

Ideally, training would involve using the checklist as intended (eg, by a team) in simulation before it is used in actual practice. Training using standard, predictable, or “textbook” scenarios is not sufficient to ensure effective use across all situations.85 More complex, yet realistic perturbations and combinations of conditions should be added to include judgment and decision-making practice.

If high-fidelity simulation is available, realistic scenarios can allow the participants to see firsthand how checklist use can enhance event management. High-fidelity medical simulation, however, is not an absolute requirement, particularly for noncritical event checklists. Advanced, in situ training in the clinical setting can use a nonfunctional or advanced cardiovascular life support training mannequin or an actor (eg, a training team member) who plays the role of the patient.74,86 In low-fidelity or “tabletop” simulations, participants in a conference room are given a scenario and are asked to verbalize their actions, using the checklist as intended. Regardless of the fidelity level, a debriefing to explore participants’ reactions and checklist effects should follow simulations and practice sessions.

Of the 91 articles we reviewed, only 8 discussed having conducted some form of training and 10 discussed actual implementation.

Back to Top | Article Outline

Ongoing Evaluation, Revision, and Possible Retirement

Even well-designed and validated checklists should continue to be evaluated after implementation.11,18,24 Ideally, a system for gathering suggested revisions should be in place so that ideas for improvement can be captured. Additionally, changes to workflow, standards of practice, and new technologies may require modification of checklist content, design, and use to maintain relevance and effectiveness. Some checklists may outlive their usefulness and should be retired, rather than revised. In our literature review, no authors mentioned conducting ongoing evaluations or possible retirement following checklist implementation.

Back to Top | Article Outline

CONCLUSIONS

In 2009, Peter Pronovost, a champion of medical checklists, said “My vision is that the science of how to do checklists is in its infancy.”87 We suggest that 9 years later, we have not moved much beyond toddlerhood. We still lack a good understanding of why clinical checklists are or are not effective in different settings.68 As many have pointed out, checklists do not work in isolation; they are part of a complicated, dynamic, sociotechnical system,37 and many cultural and systemic changes are also necessary for checklists to achieve their potential.25,38,51,76,88,89 Acceptance by clinicians and organizations will depend in large part on the effectiveness and usability of checklists that are well matched to the local operational and social environment.43,78 This requires a systematic and comprehensive development process.

Back to Top | Article Outline

ACKNOWLEDGMENTS

The authors thank Mary Connors, PhD (Associate Division Chief, NASA Ames Research Center, Moffett Field, CA), Michael O’Connor, MD (Professor of Anesthesia and Critical Care, University of Chicago, Chicago, Il), and Sajid Shahul, MD (Associate Professor of Anesthesia and Critical Care, University of Chicago, Chicago, Il), who provided insightful suggestions and comments on earlier drafts of this article.

Back to Top | Article Outline

DISCLOSURES

Name: Barbara K. Burian, PhD.

Contribution: This author helped contribute to the organization and write and prepare the manuscript.

Name: Anna Clebone, MD.

Contribution: This author helped contribute to the organization and write and prepare the manuscript.

Name: Key Dismukes, PhD.

Contribution: This author helped contribute to the organization and write and prepare the manuscript.

Name: Keith J. Ruskin, MD.

Contribution: This author helped contribute to the organization and write and prepare the manuscript.

This manuscript was handled by: Nancy Borkowski, DBA, CPA, FACHE, FHFMA.

Back to Top | Article Outline

FOOTNOTES

aSeveral sources40–42 also provide information that may be useful to developers of evaluation checklists, such as those used to record or evaluate the performance of students during simulation training or ensure that treatments have been completely implemented. Similarly, some useful guidance may also be found in other resources43–46 though this guidance is specific to aviation checklists so is not completely relevant to the development, design, and use of clinical medical checklists.

bEmbodied checklists involve equipment, tools, or materials and it is the organization of the equipment/materials/tools itself, sometimes with additional ancillary textual or graphical information, which guides or structures its use. An example of an embodied checklist is the Central Line Dressing Change Procedure developed using the principles of adherence engineering by Drews.52

cDeVon et al80 provide a nice introduction to psychometric appraisal for readers wanting a refresher.

Back to Top | Article Outline

REFERENCES

1. Hales BM, Pronovost PJ. The checklist—a tool for error management and performance improvement. J Crit Care. 2006;21:231–235.
2. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost PJ. Clinical review: checklists—translating evidence into practice. Crit Care. 2009;13:210.
3. Shillito J, Arfanis K, Smith A. Checking in healthcare safety: theoretical basis and practical application. Int J Health Care Qual Assur. 2010;23:699–707.
4. Ziewacz JE, Arriaga AF, Bader AM, et al. Crisis checklists for the operating room: development and pilot testing. J Am Coll Surg. 2011;213:212–217.e10.
5. Weingessel B, Haas M, Vécsei C, Vécsei-Marlovits PV. Clinical risk management—a 3-year experience of team timeout in 18 081 ophthalmic patients. Acta Ophthalmol. 2017;95:e89–e94.
6. Thomassen Ø, Storesund A, Søfteland E, Brattebø G. The effects of safety checklists in medicine: a systematic review. Acta Anaesthesiol Scand. 2014;58:5–18.
7. Treadwell JR, Lucas S, Tsou AY. Surgical checklists: a systematic review of impacts and implementation. BMJ Qual Saf. 2014;23:299–318.
8. Bergs J, Lambrechts F, Simons P, et al. Barriers and facilitators related to the implementation of surgical safety checklists: a systematic review of the qualitative evidence. BMJ Qual Saf. 2015;24:776–786.
9. Neuhaus C, Hofer S, Hofmann G, Wächter C, Weigand MA, Lichtenstern C. Perioperative safety: learning, not taking, from aviation. Anesth Analg. 2016;122:2059–2063.
10. Anthes E. The trouble with checklists. Nature. 2015;523:516–518.
11. Thomassen O, Brattebø G, Heltne JK, Søfteland E, Espeland A. Checklists in the operating room: Help or hurdle? A qualitative study on health workers’ experiences. BMC Health Serv Res. 2010;10:342.
12. Pysyk CL, Davies JM, Neil Armstrong J. Application of a modified surgical safety checklist: user beware! Can J Anesth. 2013;60:513–518.
13. Prielipp RC, Coursin DB. All that glitters is not a golden recommendation. Anesth Analg. 2015;121:727–733.
14. Grigg E. Smarter clinical checklists: how to minimize checklist fatigue and maximize clinician performance. Anesth Analg. 2015;121:570–573.
15. Marshall S. The use of cognitive aids during emergencies in anesthesia: a review of the literature. Anesth Analg. 2013;117:1162–1171.
16. Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. Int J Qual Health Care. 2008;20:22–30.
17. Burian B. Design guidance for emergency and abnormal checklists in aviation. Human Factors and Ergonomics Society 50th Annual Meeting. 2006.San Francisco: Human Factors and Ergonomics Society.
18. Verdaasdonk EG, Stassen LP, Widhiasmara PP, Dankelman J. Requirements for the design and implementation of checklists for surgical processes. Surg Endosc. 2009;23:715–726.
19. Burian BK. Factors affecting the use of emergency and abnormal checklists: Implications for current and NextGen operations. NASA Technical Memorandum. 2014.NASA/TM-2014-218382.
20. Thomassen Ø, Espeland A, Søfteland E, Lossius HM, Heltne JK, Brattebø G. Implementation of checklists in health care; learning from high-reliability organisations. Scand J Trauma Resusc Emerg Med. 2011;19:53.
21. Leape LL. The checklist conundrum. N Engl J Med. 2014;370:1063–1064.
22. Cullati S, Le Du S, Raë AC, et al. Is the Surgical Safety Checklist successfully conducted? An observational study of social interactions in the operating rooms of a tertiary hospital. BMJ Qual Saf. 2013;22:639–646.
23. Levy SM, Senter CE, Hawkins RB, et al. Implementing a surgical checklist: more than checking a box. Surgery. 2012;152:331–336.
24. Rydenfält C, Ek Å, Larsson PA. Safety checklist compliance and a false sense of safety: new directions for research. BMJ Qual Saf. 2014;23:183–186.
25. Prielipp RC, Birnbach DJ. Pilots use checklists, why don’t anesthesiologists? The future lies in resilience. Anesth Analg. 2016;122:1772–1775.
26. Raghunathan K. Checklists, safety, my culture and me. BMJ Qual Saf. 2012;21:617–620.
27. Fourcade A, Blache JL, Grenier C, Bourgain JL, Minvielle E. Barriers to staff adoption of a surgical safety checklist. BMJ Qual Saf. 2012;21:191–197.
28. Bowie P, Ferguson J, MacLeod M, et al. Participatory design of a preliminary safety checklist for general practice. Br J Gen Pract. 2015;65:e330–e343.
29. Davidoff F. Checklists and guidelines: imaging techniques for visualizing what to do. JAMA. 2010;304:206–207.
30. Gawande A. The Checklist Manifesto: How to Get Things Right. 2009.New York: Metropolitan Books.
31. McLaughlin A. What makes a good checklist. 2010. AHRQ Patient Safety Network Available at: https://psnet.ahrq.gov/perspectives/perspective/92/what-makes-a-good-checklist. Accessed July 2016.
32. Weiser TG, Haynes AB, Lashoher A, et al. Perspectives in quality: designing the WHO Surgical Safety Checklist. Int J Qual Health Care. 2010;22:365–370.
33. Weiser TG, Berry WR. Review article: perioperative checklist methodologies. Can J Anesth. 2013;60:136–142.
34. PSNet. Checklists. 2016. AHRQ Patient Safety Network Available at: https://psnet.ahrq.gov/primers/primer/14/checklists. Accessed August 2016.
35. Leadership E-Bulletin. A checklist for creating checklists. Available at: http://www.hfma.org/WorkArea/DownloadAsset.aspx?id=4845. Accessed August 2016.
36. Gawande A, Boorman D. The Brigham and Women’s Hospital Center for Surgery and Public Health Dissemination Team. A checklist for checklists. Available at: http://www.projectcheck.org/checklist-for-checklists.html. Accessed August 2016.
37. Reason J. Combating omission errors through task analysis and good reminders. Qual Saf Health Care. 2002;11:40–44.
38. Ziewacz JE, Berven SH, Mummaneni VP, et al. The design, development, and implementation of a checklist for intraoperative neuromonitoring changes. Neurosurg Focus. 2012;33:E11.
39. Fletcher KA, Bedwell WL. Cognitive aids: design suggestions for the medical field. Proceedings of the Intl Symp HFES-HC. 2014;3:148–152.
40. Scriven M. The Logic and Methodology of Checklists Western Michigan University. 2005. http://preval.org/documentos/2075.pdf. Accessed September 25, 2016.
41. Stufflebeam D. Guidelines for Developing Evaluation Checklists: The Checklists Development Checklist (CDC). 2000.Available at: https://www.wmich.edu/evaluation/checklists. Accessed September 25, 2016.
42. Schmutz J, Eppich WJ, Hoffmann F, Heimberg E, Manser T. Five steps to develop checklists for evaluating clinical performance: an integrative approach. Acad Med. 2014;89:996–1005.
43. Degani A, Wiener EL. Human factors of flight-deck checklists: the normal checklist. 1990: NASA Contractor Report 177549.
44. Degani A. On the Typography of flight-deck documentation. NASA Technical Memorandum #177605. 1992.Moffett Field, CA: NASA Ames Research Center.
45. Civil Aviation Authority (CAA). Guidance on the Design, Presentation, and Use of Electronic Checklists. CAP 708. Safety Regulation Group. 2000. Available at: https://publicapps.caa.co.uk/docs/33/CAP708.PDF. Accessed September 25, 2016.
46. Civil Aviation Authority (CAA). Guidance on the Design Presentation and Use of Emergency and Abnormal Checklists. CAP 676. Safety Regulation Group. 2006. Available at: http://publicapps.caa.co.uk/docs/33/CAP676.PDF. Accessed September 25, 2016.
47. DeMuth DL, Achorn EH. The physician’s income tax checklist. Pa Med. 1978;81:94–97.
48. Duchesne S, Jannin P. Proposing a manuscript peer-review checklist. Neuroimage. 2008;39:1783–1787.
49. Kattwinkel J, Perlman JM, Aziz K, et al. Part 15: neonatal resuscitation: 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation. 2010;122:S909–S919.
50. The World Health Organization: Surgical Safety Checklist. Available at: http://whqlibdoc.who.int/publications/2009/9789241598590_eng_Checklist.pdf. Accessed July 1, 2016.
51. Clay-Williams R, Colligan L. Back to basics: checklists in aviation and healthcare. BMJ Qual Saf. 2015;24:428–431.
52. Drews F. Adherence engineering: a new approach to increasing adherence to protocols. EID. 2013;21:19–25.
53. Henriksen K, Brady J. The pursuit of better diagnostic performance: a human factors perspective. BMJ Qual Saf. 2013;22suppl 2ii1–ii5.
54. Nolan B, Zakirova R, Bridge J, Nathens AB. Barriers to implementing the World Health Organization’s Trauma Care Checklist: a Canadian single-center experience. J Trauma Acute Care Surg. 2014;77:679–683.
55. Hilligoss B, Moffatt-Bruce SD. The limits of checklists: handoff and narrative thinking. BMJ Qual Saf. 2014;23:528–533.
56. Rosen MA, Pronovost PJ. Advancing the use of checklists for evaluating performance in health care. Acad Med. 2014;89:963–965.
57. Wetmore D, Goldberg A, Gandhi N, Spivack J, McCormick P, DeMaria S Jr.. An embedded checklist in the Anesthesia Information Management System improves pre-anaesthetic induction setup: a randomised controlled trial in a simulation setting. BMJ Qual Saf. 2016;25:739–746.
58. Salzwedel C, Mai V, Punke MA, Kluge S, Reuter DA. The effect of a checklist on the quality of patient handover from the operating room to the intensive care unit: a randomized controlled trial. J Crit Care. 2016;32:170–174.
59. Stanford Anesthesia Cognitive Aid Group. Emergency Manual: Cognitive Aids for Perioperative Clinical Events. 2013. Stanford, CA: Stanford School of Medicine; Available at: http://emergencymanual.stanford.edu. Accessed November 2015.
60. Watkins SC, Anders S, Clebone A, et al. Paper or plastic? Simulation based evaluation of two versions of a cognitive aid for managing pediatric peri-operative critical events by anesthesia trainees: evaluation of the society for pediatric anesthesia emergency checklist. J Clin Monit Comput. 2016;30:275–283.
61. Rogers J. Have we gone too far in translating ideas from aviation to patient safety? Yes. BMJ. 2011;342:c7309.
62. Elwyn G, O’Connor A, Stacey D, et al. Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ. 2006;333:417.
63. Bloomstone J. Humans fail, checklists don’t. J Clin Anesth Manag. 2015;1:1–3.
64. Wheelock A, Suliman A, Wharton R. The impact of operating room distractions on stress, workload, and teamwork. Ann Surg. 2015;261:1079–1084.
65. Dismukes R, Berman B, Loukopoulos L. The Limits of Expertise: Rethinking Pilot Error and the Causes of Airline Accidents. 2007.Aldershot, England: Ashgate.
66. Verdaasdonk EG, Stassen LP, van der Elst M, Karsten TM, Dankelman J. Problems with technical equipment during laparoscopic surgery. An observational study. Surg Endosc. 2007;21:275–279.
67. Hart EM, Owen H. Errors and omissions in anesthesia: a pilot study using a pilot’s checklist. Anesth Analg. 2005;101:246–250.
68. Catchpole K, Russ S. The problem with checklists. BMJ Qual Saf. 2015;24:545–549.
69. Miller GA. The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol Rev. 1956;63:81–97.
70. Marshall SD, Sanderson P, McIntosh CA, Kolawole H. The effect of two cognitive aid designs on team functioning during intra-operative anaphylaxis emergencies: a multi-centre simulation study. Anaesthesia. 2016;71:389–404.
71. Marshall SD. Helping experts and expert teams perform under duress: an agenda for cognitive aid research. Anaesthesia. 2017;72:289–295.
72. The Society for Pediatric Anesthesia Quality and Safety Committee. Critical Events Checklists. Available at: http://www.pedsanesthesia.org/critical-events-checklists/. Accessed July 2016.
73. White RE, Trbovich PL, Easty AC, Savage P, Trip K, Hyland S. Checking it twice: an evaluation of checklists for detecting medication errors at the bedside using a chemotherapy model. Qual Saf Health Care. 2010;19:562–567.
74. Goldhaber-Fiebert SN, Pollock J, Howard SK, Bereknyei Merrell S. Emergency manual uses during actual critical events and changes in safety culture from the perspective of anesthesia residents: a pilot study. Anesth Analg. 2016;123:641–649.
75. Whyte S, Lingard L, Espin S, et al. Paradoxical effects of interprofessional briefings on OR team performance. Cogn Technol Work. 2008;10:287–294.
76. Raman J, Leveson N, Samost AL, et al. When a checklist is not enough: how to improve them and what else is needed. J Thorac Cardiovasc Surg. 2016;152:585–592.
77. Russ S, Rout S, Sevdalis N, Moorthy K, Darzi A, Vincent C. Do safety checklists improve teamwork and communication in the operating room? A systematic review. Ann Surg. 2013;258:856–871.
78. Burian BK. Aeronautical emergency and abnormal checklists: Expectations and realities. Human Factors and Ergonomics Society 50th Annual Meeting. 2006.San Francisco, CA: Human Factors and Ergonomics Society.
79. McEvoy MD, Hand WR, Stoll WD, Furse CM, Nietert PJ. Adherence to guidelines for the management of local anesthetic systemic toxicity is improved by an electronic decision support tool and designated “Reader.” Reg Anesth Pain Med. 2014;39:299–305.
80. DeVon HA, Block ME, Moyle-Wright P, et al. A psychometric toolbox for testing validity and reliability. J Nurs Scholarsh. 2007;39:155–164.
81. McCarroll ML, Zullo MD, Dante Roulette G, et al. Development and implementation results of an interactive computerized surgical checklist for robotic-assisted gynecologic surgery. J Robot Surg. 2015;9:11–18.
82. Korkiakangas T. Mobilising a team for the WHO Surgical Safety Checklist: a qualitative video study. BMJ Qual Saf. 2017;26:177–188.
83. Goldhaber-Fiebert SN, Howard SK. Implementing emergency manuals: can cognitive aids help translate best practices for patient care during acute events? Anesth Analg. 2013;117:1149–1161.
84. World Health Organization. Implementation Manual Surgical Safety Checklist.. Available at: http://www.who.int/patientsafety/safesurgery/tools_resources/SSSL_Manual_finalJun08.pdf?ua=1. Accessed December 2016.
85. Burian BK. The Challenge of Aviation Emergency and Abnormal Situations. NASA Technical Memorandum TM-2005–213462. 2005.Moffett Field, CA: NASA Ames Research Center.
86. Clebone A, Burian BK, Watkins SC, Gálvez JA, Lockman JL, Heitmiller ES; Members of the Society for Pediatric Anesthesia Quality and Safety Committee (see Acknowledgments). The development and implementation of cognitive aids for critical events in pediatric anesthesia: The Society for Pediatric Anesthesia Critical Events Checklists. Anesth Analg. 2017;124:900–907.
87. Laurance J. Peter Pronovost: champion of checklists in critical care. Lancet. 2009;374:443.
88. Bosk CL, Dixon-Woods M, Goeschel CA, Pronovost PJ. Reality check for checklists. Lancet. 2009;374:444–445.
89. Merry AF, Mitchell SJ. Advancing patient safety through the use of cognitive aids. BMJ Qual Saf. 2016;25:733–735.
Copyright © 2017 International Anesthesia Research Society