Cognitive aids such as checklists have been commonly used in aviation since the 1930s when it was felt that aircraft were becoming too complex to fly safely without written standardized procedures.1 Cognitive aids are tools created to guide users while they are performing a task, or group of tasks, with the goal of reducing errors and omissions and increasing the speed and fluidity of performance.2,3 The main difference from guidelines, protocols or standard operating procedures is that they are to be used while the task is being performed (Table 1).4 Many cognitive aids are derived from documents describing extensive and detailed sequences of actions that would be difficult to examine while undertaking a task,1,5,6 particularly during an emergency. Secondarily, their presentation as simplified instructions also makes them useful for training.6 A recent editorial suggested that “health care providers should not be expected to manage rare events well without the use of a cognitive aid”7 since safety checklists have been successfully developed, tested, and used in other high-risk industries. The aim of undertaking this literature review was to critically examine the evidence to support this view that cognitive aids should be developed and used for all crises in anesthetic practice. Evidence was sought to establish (1) whether cognitive aids improve performance of individuals and teams and (2) whether recommendations can be made for future cognitive aid design, testing, and implementation.
;)
Table 1: A Terminology of Standardized Practices
4THE PROPERTIES OF THE IDEAL COGNITIVE AID
To understand how a cognitive aid may assist someone in a crisis, it is useful to consider the properties of an “ideal” cognitive aid. During an emergency, time and cognitive resources are limited. When under stress, clinicians are less able to recall remembered lists and are more likely to become fixated.8,9 A cognitive aid would theoretically guide stressed clinicians through a sequence of complex steps and prevent them from omitting key actions. It would also ensure that the sequence was the most effective for the situation and that unnecessary or extraneous measures were not taken. For this to happen, the cognitive aid must have the following properties: (1) Its content must be derived from “best practice” guidelines or protocols; (2) Its design should be appropriate for use in the context of the emergency situation; (3) It should be familiar, in a format that has been used in practice and training; (4) It should also assist other team members to perform their tasks in a coordinated manner. Researchers studying cognitive aids in aviation10,11 and health care12,13 have suggested that cognitive aids that are deficient in these areas of content, design, training, and team alignment may promote the wrong sequence of actions and potentially cause harmful effects.
If an inaccurate or poorly designed cognitive aid can cause harm, we must consider to what degree it should be assessed before it is used in the clinical environment. Currently, there are no formal standards for development and evaluation; perhaps a useful analogy is to conceptualize a cognitive aid as a medical device or tool that helps the user perform a task. The International Electrotechnical Commission (IEC) defines a medical device as “…any instrument, apparatus, implement, machine … software or similar or related article … for one or more of the specific purposes of—diagnosis, prevention, monitoring, treatment or alleviation of disease…”14 A cognitive aid might arguably be within this definition, but a rigid legal application of the IEC standards to cognitive aids would be inappropriate. However, the IEC standard and other similar standards15,16 might be used to evaluate current cognitive aids and guide future development. Standards such as the IEC 62366 and American National Standards Institute/Association for the Advancement of Medical Instrumentation HE75:2009 describe the design, development, and testing of medical devices in terms of content, format, and training, and these same principles could be applied to cognitive aids. Additional criteria for assessing the effects of the cognitive aid on team performance would also be appropriate.
Content
A medical device must possess the qualities that allow it to be applied to the task for which it is intended, termed its “Primary Operating Function.” For a cognitive aid, these qualities are the content, or information presented within the aid that should represent the most suitable way that the task is performed, i.e., the information should represent best clinical practice.
Design
In the same way that a poorly designed clinical monitor may display all of the relevant physiological information, merely having the correct content on a cognitive aid is not sufficient. The important information needs to be ordered and made salient with appropriate alarms, color, position, and font type and size. A device should be designed to make the job as efficient and effective as possible, termed its “Usability Specification.” The possibilities of harm from improper or unintended use should also be considered (“Hazard Identification”). The design should be evaluated against testable requirements; refining the device’s characteristics is usually an iterative process. Poor design of cognitive aids in aviation have caused unintended effects, such as the shutdown of the wrong engine during an emergency.10 There is no reason to assume that poorly designed cognitive aids in health care would not also lead to errors such as following the wrong arm of an overly complex flowchart.12
Training
The appropriate training for the users should also be considered both in general terms of who will use the device, termed “User profiles,” and the training that is specific to the device. It could be argued that training in the use of a simple device is merely compensating for an inadequate design. But, training may improve the familiarity with a cognitive aid and enhance the chance of it being used effectively, even if it is an apparently simple presentation like a poster or paper checklist.
Individual Versus Team Performance
Given the nature of anesthetic emergencies, in which multiple tasks are undertaken concurrently, a cognitive aid must support both the individual and the team in managing the emergency. A cognitive aid that distracts and interrupts the team from performing their tasks may increase the risk of errors and have the opposite of the intended effect. A partial solution to this problem of distraction can be found in aviation where, in times of high workload such as takeoff and landing, only information directly related to that specific phase of flight is communicated.17 Nevertheless, even an apparently “perfect” cognitive aid may still distract clinicians when used in context and some degree of education about how to use the aids seems unavoidable.
In this review, evidence will be examined to determine:
- Whether written cognitive aids are of benefit to both individuals and teams during emergencies in anesthesia.
- Whether recommendations can be made for improving the development, testing, and acceptance of future cognitive aids.
Specifically, this review is aimed toward the use of cognitive aids during crises, as this is one circumstance where they have been suggested to have a substantial role in improving safety.
METHODS
A literature search was performed for articles describing the use of cognitive aids in anesthetic emergencies. The MEDLINE (full database from 1950), EMBASE, Cochrane library, and PsycInfo databases were searched in all fields and MeSH for the terms “anesthesia/anaesthesia,” and for each of the terms: “algorithm,” “checklist,” “cognitive aid,” “standard operating procedure,” and “guideline.” No restrictions on language were made, and the search criteria were as broad as possible to ensure the literature was comprehensively covered. The reference lists of the articles selected for review were also scanned to identify additional studies not found using the original search terms. Articles were included if they met the following criteria:
- A cognitive aid was used or tested (papers merely referencing the development or existence of cognitive aids were excluded).
- The paper was specifically relevant to emergencies encountered in anesthetic practice such as airway emergencies or cardiac events.
- The cognitive aid related to anesthetic emergencies rather than the routine care to prevent emergencies such as machine checkouts.
- The paper described a cognitive aid that was not only for an educational or assessment purpose.
Papers were also included if they involved the use of a cognitive aid by a group other than anesthesia providers and if the aim of the aid was to assist in emergencies that might also occur during anesthetic care (e.g., the management of cardiac arrest). For each of the papers included, 6 aspects were used to examine the content, design, and training:
- How the content of the cognitive aid was derived
- How the aid was designed
- How the aid was evaluated, such as surveys of perceived usefulness, performance data in real or simulated settings, or patient outcomes
- Whether education had been provided on the use of the aid
- Effects on team functioning
- Evidence of successful implementation as demonstrated by accepted use in the clinical environment or improved clinical outcomes.
RESULTS
A flowchart of the search strategy and results are given in Figure 1. The Cochrane, EMBASE, MEDLINE, and PsycInfo database searches produced 18, 4222, 4472, and 55 results, respectively.
Figure 1: Literature search process.
After a manual review of their abstracts, most articles were rejected because they did not refer to the use of a specific cognitive aid. In total, 22 cognitive aids were found described in 23 evaluation papers. One cognitive aid was evaluated in 3 separate studies,18–20 and 1 study evaluated 2 different cognitive aids.18Table 2 presents the studies reporting the cognitive aids, focusing on their content and how their content was developed. Table 3 presents the same studies, but focuses on how the cognitive aids were evaluated.
Table 2: Summary of Cognitive Aids (Grouped by Type of Incident That the Aids Were Designed to Address) and How the Aid Was Developed and Designed with Related Publications
Table 3: Methods of Evaluation and Outcomes for Each of the Cognitive Aids (CAs) Tested
Content of the Cognitive Aid
Seven of the 22 cognitive aids were developed to help in a range of anesthetic crises, whereas the rest helped with specific crises or related groups of crises such as cardiac arrest or airway emergencies. The Abbreviated Stanford Manual was developed from one of the earliest cognitive aids found for a variety of crises in anesthesia, the crisis management handbook by Gaba et al.,22 now nearly 20 years old.
The content of 13 cognitive aids was derived from existing national guidelines18,19,26,29,30,32–37,40,42 including all of the aids relating to management of cardiac arrest of adults32–37 and neonates.40 This may reflect the existence of established national and international guidelines for cardiopulmonary resuscitation.
Most of the aids with content derived from local expert consensus did not explain how the content was decided upon. The exception to this was the Australian Patient Safety Foundation (APSF) crisis management handbook,24 which has been extensively documented.23 A series of 7 national meetings of up to 100 anesthesiologists per meeting was held, and a consensus for ideal management was reached for each crisis. The authors subsequently condensed this content into a 74-page manual that used a common mnemonic for many emergency situations in anesthesia.
Design of the Aid
The 22 cognitive aids discovered by this search were presented in forms as diverse as handbooks, cards, posters, or desktop or handheld computer-based help systems (Table 2). Two of the studies did not describe how the aid was presented.28,29
For 11 of the 22 cognitive aids, the design process was not described. Mills et al.34 described a subjective method of design by a working group, with the addition of an index and ring for attachment to the code cart. Participants reported that the additions improved the ease of use, although none of the questions in a later survey specifically addressed aspects of design. Only 1 of the 22 cognitive aids was developed using an iterative method that is the standard for other medical devices.27
Evaluation of the Aid
The cognitive aids were evaluated by surveys, simulation studies, and analyses of case reports (Table 3). The 3 studies that collected survey data21,34,41 concentrated mainly on how often the aid was used and how aware clinicians were of it in the clinical setting. These survey data will be dealt with in the section on implementation. Where the cognitive aids were tested, it appears they were not measured against any criteria, and in only 1 was the design altered to improve the usability as suggested by the international standards for medical devices (IEC 62366 Sections 5.6 and 5.9).14
Simulation-based studies were used to evaluate whether the cognitive aids could be useful in emergencies. In 13 studies, researchers assessed technical performance such as speed of task completion and number of errors and omissions.18,19,25–27,31–33,36–38,40,42 All but one of the studies used a part task trainer or mannequin-based simulators. The exception was the study by Berkenstadt et al.25 that used a screen-based computer simulator to determine whether an electronic help system reduced the number of knowledge-based errors.
Ten of the 13 studies suggested an improvement in technical performance in diverse activities ranging from the management of an airway to managing malignant hyperthermia crises. Two of the remaining 3 studies showed no improvement in neonatal (Bould et al.40) and adult (Schneider et al.32) resuscitation. Coopmans and Biddle26 showed a longer delay to diagnose and intervene during emergencies when a personal digital assistant device was provided.
Observational studies and case reports have also been used to determine the effect of introducing cognitive aids. Both Heidegger et al.28 and Combes et al.29 observed fewer failed intubations when a standardized approach was used. Marshall and Flanagan39 reported that treating anesthesiologists believed that successful management of malignant hyperthermia was due to the use of a cognitive aid improving their task management, reducing omissions, and improving team performance.
The Australian Patient Safety Foundation took a larger scale approach by retrospectively analyzing thousands of incident reports from a national database. The first 2000 reports were used to determine whether a standardized approach (the “COVERABCD” algorithm) could have prevented or mitigated the incident using a “walkthrough” of the algorithm described by the cognitive aid.23 They found that 60% of incidents could have been rectified in <1 minute had the aid been used.
Training in the Use of the Cognitive Aid
Reports of formal orientation to the users about the cognitive aid were found in 8 of the 23 evaluation studies.18,25–27,32,33,36,37 In 3 of the remaining 15 studies, the cognitive aid was deliberately not shown to participants before they used it, but participants were informed that a cognitive aid would be present.31,38,40 Presumably, this strategy was taken to determine whether the aid could be used effectively without prior knowledge of its contents or structure. In 2 studies that found no difference in performance while using a cognitive aid, participants were allowed to familiarize themselves with the aid before using it.26,32
Effects on Team Functioning
Only 4 aids have been evaluated with regard to their effects on team functioning beyond the ability to merely perform tasks accurately and efficiently.18,20,40,42 Manser et al.20 observed the ability of teams to manage a malignant hyperthermia crisis when they referred to cognitive aids on personal digital assistants or cards that they carried. Teams that scored highly on a clinical performance score were more likely to discuss their assessment of the situation than the division of tasks. It was suggested that the cognitive aid might have affected how the teams coordinated their activities. Three studies failed to find an improvement in team functioning. Studies by Neal et al.42 and Bould et al.40 observed individual anesthesia trainees working with “actors” performing the supporting clinical roles. No differences in nontechnical skills were found between participants using versus not using a cognitive aid in either study, although Neal et al.42 suggested a slight improvement in the Decision Making category in the participants with the cognitive aid. Burden et al.18 noted a reduction in the volume of communication within the team when a cognitive aid was used.
Evidence of Implementation
Evidence of implementation in the actual clinical environment was identified in 3 studies.21,34,41 Two of these papers reported surveys of the use of cognitive aids in clinical settings. In each of these surveys only half of the respondents reported that they either would use or had used cognitive aids in emergencies.
DISCUSSION
The literature on cognitive aids in anesthesia emergencies appears to be expanding rapidly; 9 of the 23 articles found published in the last 3 years. Given the broad nature of the search and manual review of the papers, all of the cognitive aids currently available for anesthetic emergencies are likely to have been included.43 Some cognitive aids that may improve patient safety, such as the World Health Organization Surgical Safety Checklist,44 have not been included as they are not generally used during an emergency. There is some evidence that cognitive aids improve technical performance during emergencies, but there is much to be learned about when and why they fail.
Drawing the analogy between cognitive aids and medical devices may help us understand when and in what form cognitive aids may be useful. In common with medical devices, cognitive aids need to have the correct content, be well designed, and be accompanied by appropriate training to assist task performance. The content or knowledge contained in cognitive aids should be developed from national or international guidelines or from broad consensus. To ensure continuing usefulness, their content needs to be reviewed and adapted as knowledge changes.
The content seems to have been the main focus during the development of most cognitive aids, with more than half being derived from established guidelines. In contrast, the design processes, presentation, and the resultant usability of cognitive aids seem to have been less thoroughly considered. Only one of the cognitive aids was developed with a systematic design process and adapted as a result of evaluation data.27 Evidence from the human factors literature suggests that poorly designed cognitive aids may lead to unintended consequences.10,13 The use of a human factors design process, as mandated for medical devices,45 may help to reduce the likelihood of poor design but will not eliminate it. It is also important to select appropriate measurements and testing scenarios, and then revise the design appropriately. There is little evidence that the majority of cognitive aids discovered in this literature review have been designed for the context in which they are to be used. Physical constraints of using aids, such as the ability to read, see, hear, or interact with the paper or computer-based aid, potential distraction caused by the aid, and the ability to perform concurrent tasks have not been addressed. These contextual issues could make aids very difficult to use in the actual working environment and may reflect the apparent negative effects on team communication that have been reported.18 Research is currently lacking in this area not only in anesthesia and health care but also in other industries, despite the acknowledgment that checklist design has been a contributing factor in several airline accidents.11 There are functional differences between a long, nonsequential checklist and dynamic decision tree, and the design chosen depends on the intended context (Table 1).
Despite the perceived benefits to the team, there is minimal evidence to support an improvement in team function with the use of cognitive aids. Data suggest that cognitive aids may change team coordination and improve task completion but their effects on team processes are not clear. In particular, the study by Burden et al.18 into the effects of a designated reader demonstrated decreased communication by the team. Reduction in team communication is generally considered deleterious to team coordination,46 and as such the aid may have distracted the team from communicating with each other. Conversely, it may be that the remaining communication was more efficient and targeted. Further studies are needed to determine how cognitive aids affect interpersonal communication during anesthetic emergencies.
Training in the use of a cognitive aid appears to be often cursory or absent. It is reasonable to assume that familiarization to a cognitive aid before its use would mean that the participants would be more likely to use it and use it more effectively. This increased likelihood of use after education is supported by responses to Mills et al.’s survey34 about a cognitive aid for cardiac arrest management: those who had learned about the aid from a formal orientation session were more likely to use it in an emergency than those who were not oriented to the aid. Research shows that education raises awareness of the presence of the aid and helps identify how and when the aid should be used, both stimulating its use and making its use more effective.47 The counter argument is that a cognitive aid should be intuitive to use in an emergency situation, and that educating people to use a cognitive aid merely compensates for a poor design. Ideally, as with any medical device, for the best results the cognitive aid should be intuitive to use and should also be used only by individuals trained in its use.
Reports of implementation into clinical practice demonstrate that even when clinicians are aware of the existence of cognitive aids they often do not use them. Part of the problem of the underuse of cognitive aids may be the existence of a professional culture that does not support their use. For example, when observing the use of cognitive aids in malignant hyperthermia crises, one of the participants in Harrison et al.’s study19 reported the view that using a cognitive aid reflected a lack of confidence or knowledge. Several other studies also report participants’ comments that they did not need the cognitive aid to manage the emergency effectively.19,21,34,40 In contrast, the results of Low et al.’s study37 with junior doctors suggest a willingness to use cognitive aids, and their feeling that using them does not reflect a lack of competence of health professionals. A further issue that is rarely addressed is the positioning of the cognitive aid. A reminder of the aid and therefore prompting to use it can be improved by associating the aid with the task in the same way that any prospective memory task can be prompted.48 For example, the cognitive aid for management of local anesthetic toxicity may be positioned with the Intralipid® on an emergency cart.
A recent editorial suggested that cognitive aids should be available for all rare emergencies in anesthesia7 but this poses more questions, such as who creates, updates, and designs the cognitive aids. A regular formal review of laboratory and clinical evidence by a reputable body would be required for each emergency. The process currently used for the management of cardiac arrest by the International Liaison Committee on Resuscitation49 would seem to be the ideal template. Levels of evidence should ideally be included and should lead to a consensus about the actions required during the emergency. This information would then be passed to a human factors design team for design, testing, and modification based on heuristic and simulation evaluation. Only then would the guidelines be available for distribution.
Cognitive aids are not commonly used during emergencies in anesthesia and at present appear not to be supported by the culture. For cognitive aids to be more broadly accepted, further evidence may be needed that they confer benefits in emergency situations. Although we know that practitioners’ coordination patterns change when a cognitive aid is used, there is a need for larger prospective trials of the effect of aids on task completion, practitioners’ team behaviors, and overall team functioning. Such trials will allow a deeper understanding of how teams may use an aid to best allocate roles so that they can rapidly process tasks in parallel. The trials should use robustly researched and designed aids and should assess outcome measures of accurate and rapid task completion as well as process measures of team behaviors. Testing would best occur in simulation-based settings, where the physiological variables and environment can be more tightly controlled and replicated, and there are no direct risks to patients.
Cognitive aids should be integrated into anesthesiologists’ vocational and continuing education so that practitioners are aware of when and how to use them appropriately. Continuing education programs have started to advocate the use of cognitive aids in emergencies through simulation-based education50 but such education is not readily accessible to all anesthesia providers. Any future testing of cognitive aids in simulation-based settings should first let practitioners become oriented to the aids. The maximum benefit is likely only if practitioners are familiar with the structure of each aid, and how it should be used.
As this literature review has shown, the current evidence for the efficacy of cognitive aids in emergencies is inconclusive. However, although the evidence to support the use of cognitive aids in emergencies is currently weak, the success in other settings is compelling. The lack of evidence is due to both the limited research that has been performed and the deficiencies in design and evaluation of current cognitive aids. By conceptualizing cognitive aids as medical devices, we may be able to address these deficiencies and improve the outcomes of patients experiencing anesthetic emergencies.
DISCLOSURES
Name: Stuart Marshall, MB, ChB, M.HumanFact.
Contribution: Study design, conduct, data collection, and manuscript preparation were all undertaken by the author.
Attestation: Stuart Marshall approved the final manuscript and attests to the integrity of the original data and analysis reported in this manuscript.
This manuscript was handled by: Sorin J. Brull, MD, FCARCSI (Hon).
ACKNOWLEDGMENTS
I would like to acknowledge the invaluable assistance and guidance of Professor Penelope Sanderson, Dr. Cate McIntosh, and Dr. Brendan Flanagan in the preparation of this manuscript.
REFERENCES
1. Meilinger PS. When the Fortress went down. Airforce Mag. 2004:78–82
2. Reason J. Cognitive aids in process environments: prostheses or tools? Int J Man Mach Stud. 1987;27:463–70
3. Roth EM, Mumaw RJ, Lewis PM An Empirical Investigation of Operator Performance in Cognitively Demanding Simulated Emergencies. 1994 Washington, DC US Nuclear Regulatory Commission Contract No.: NUREG/CR-6208
4. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost PJ. Clinical review: checklists—translating evidence into practice. Crit Care. 2009;13:210
5. Morris AH. Decision support and safety of clinical environments. Qual Saf Health Care. 2002;11:69–75
6. Wieringa D, Moore C, Barnes V Procedure Writing Principles and Practice. 1993 Piscataway, NJ IEEE Press
7. Nanji KC, Cooper JB. It is time to use checklists for anesthesia emergencies: simulation is the vehicle for testing and learning. Reg Anesth Pain Med. 2012;37:1–2
8. Kuhlmann S, Piel M, Wolf OT. Impaired memory retrieval after psychosocial stress in healthy young men. J Neurosci. 2005;25:2977–82
9. Xiao Y, Mackenzie CF. Decision making in dynamic environments: fixation errors and their causes. Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. 1995 San Diego, CA Human Factors and Ergonomics Society
10. Mosier KL, Palmer EA, Degani A. Electronic checklists: implications for decision making. Proceedings of the Human Factors Society 36th Annual Meeting. 1992 Atlanta, GA Human Factors and Ergonomics Society
11. Degani A, Wiener EL. Cockpit checklists: concepts, design and use. Hum Factors. 1993;35:345–59
12. Nelson K, Shilkofski N, Haggerty J, Vera K, Saliski M, Hunt E. Cognitive aids do not prompt initiation of cardiopulmonary resuscitation in simulated pediatric cardiopulmonary arrest. Simul Healthc. 2007;2:54
13. Carthey J, Clarke J The ‘How to guide’ for implementing human factors in healthcare Patient Safety First, UK. 2009 Available at:
http://www.patientsafetyfirst.nhs.uk/ashx/Asset.ashx?path=/Intervention-support/Human%20Factors%20How-to%20Guide%20v1.2.pdf. Accessed June 12, 2013.
14. International Electrotechnical Commission.IEC 62366, Ed.1: Medical Devices—Application of Usability Engineering to Medical Devices. 2007 Boston, MA International Electrotechnical Commission
15. American National Standards Institute/Association for the Advancement of Medical Instrumentation.HE75:2009 Human Factors Engineering—Design of Medical Devices. 2009 Arlington, VA Association for the Advancement of Medical Instrumentation
16. Weinger MB, Wiklund ME, Gardner-Bonneau DJ Handbook of Human Factors in Medical Device Design. 2011 Boca Raton, FL CRC Press
17. . Flight crewmember duties. Code of Federal Regulations (Aeronautics and Space), Part 121 (Operating Requirements) Subpart T Flight Operations. Section 121.542
18. Burden AR, Carr ZJ, Staman GW, Littman JJ, Torjman MC. Does every code need a “reader?” improvement of rare event management with a cognitive aid “reader” during a simulated emergency: a pilot study. Simul Healthc. 2012;7:1–9
19. Harrison TK, Manser T, Howard SK, Gaba DM. Use of cognitive aids in a simulated anesthetic crisis. Anesth Analg. 2006;103:551–6
20. Manser T, Harrison TK, Gaba DM, Howard SK. Coordination patterns related to high clinical performance in a simulated anesthetic crisis. Anesth Analg. 2009;108:1606–15
21. Neily J, DeRosier JM, Mills PD, Bishop MJ, Weeks WB, Bagian JP. Awareness and use of a cognitive aid for anesthesiology. Jt Comm J Qual Patient Saf. 2007;33:502–11
22. Gaba DM, Fish KJ, Howard SK Crisis Management in Anesthesiology. 1994 New York, NY Churchill Livingstone
23. Runciman WB, Kluger MT, Morris RW, Paix AD, Watterson LM, Webb RK. Crisis management during anaesthesia: the development of an anaesthetic crisis management manual. Qual Saf Health Care. 2005;14:e1
24. Australian Patient Safety Foundation. Crisis Management Manual: ‘COVER ABCD A SWIFT CHECK. 1996 Adelaide, Australia Australian Patient Safety Foundation
25. Berkenstadt H, Yusim Y, Katznelson R, Ziv A, Livingstone D, Perel A. A novel point-of-care information system reduces anaesthesiologists’ errors while managing case scenarios. Eur J Anaesthesiol. 2006;23:239–50
26. Coopmans VC, Biddle C. CRNA performance using a handheld, computerized, decision-making aid during critical events in a simulated environment: a methodologic inquiry. AANA J. 2008;76:29–35
27. Ziewacz JE, Arriaga AF, Bader AM, Berry WR, Edmondson L, Wong JM, Lipsitz SR, Hepner DL, Peyre S, Nelson S, Boorman DJ, Smink DS, Ashley SW, Gawande AA. Crisis checklists for the operating room: development and pilot testing. J Am Coll Surg. 2011;213:212–217.e10
28. Heidegger T, Gerig HJ, Ulrich B, Kreienbühl G. Validation of a simple algorithm for tracheal intubation: daily practice is the key to success in emergencies–an analysis of 13,248 intubations. Anesth Analg. 2001;92:517–22
29. Combes X, Le Roux B, Suen P, Dumerat M, Motamed C, Sauvat S, Duvaldestin P, Dhonneur G. Unanticipated difficult airway in anesthetized patients. Anesthesiology. 2004;100:1146–50
30. Berkow LC, Greenberg RS, Kan KH, Colantuoni E, Mark LJ, Flint PW, Corridore M, Bhatti N, Heitmiller ES. Need for emergency surgical airway reduced by a comprehensive difficult airway program. Anesth Analg. 2009;109:1860–9
31. Seagull FJ, Ho D, Radcliffe J, Xiao Y, Hu P, Mackenzie CF. Just-in-time training for medical emergencies: Computer versus paper checklists for a tracheal intubation task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2007 Baltimore, MD Human Factors and Ergonomics Society
32. Schneider AJ, Murray WB, Mentzer SC, Miranda F, Vaduva S. “Helper:” A critical events prompter for unexpected emergencies. J Clin Monit. 1995;11:358–64
33. Ward P, Johnson LA, Mulligan NW, Ward MC, Jones DL. Improving cardiopulmonary resuscitation skills retention: effect of two checklists designed to prompt correct performance. Resuscitation. 1997;34:221–5
34. Mills PD, DeRosier JM, Neily J, McKnight SD, Weeks WB, Bagian JP. A cognitive aid for cardiac arrest: you can’t use it if you don’t know about it. Jt Comm J Qual Saf. 2004;30:488–96
35. Dyson E, Voisey S, Hughes S, Higgins B, McQuillan PJ. Educational psychology in medical learning: a randomised controlled trial of two aide memoires for the recall of causes of electromechanical dissociation. Emerg Med J. 2004;21:457–60
36. Merchant RM, Abella BS, Abotsi EJ, Smith TM, Long JA, Trudeau ME, Leary M, Groeneveld PW, Becker LB, Asch DA. Cell phone cardiopulmonary resuscitation: audio instructions when needed by lay rescuers: a randomized, controlled trial. Ann Emerg Med. 2010;55:538–543.e1
37. Low D, Clark N, Soar J, Padkin A, Stoneham A, Perkins GD, Nolan J. A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency. Anaesthesia. 2011;66:255–62
38. Hart EM, Owen H. Errors and omissions in anesthesia: a pilot study using a pilot’s checklist. Anesth Analg. 2005;101:246–50
39. Marshall S, Flanagan B. Cognitive AIDS in a simulated anesthetic crisis. Anesth Analg. 2007;104:1292–3 author reply 1293
40. Bould MD, Hayter MA, Campbell DM, Chandra DB, Joo HS, Naik VN. Cognitive aid for neonatal resuscitation: a randomized controlled trial. Br J Anaesth. 2009;103:570–5
41. Picard J, Ward SC, Zumpe R, Meek T, Barlow J, Harrop-Griffiths W. Guidelines and the adoption of ‘lipid rescue’ therapy for local anaesthetic toxicity. Anaesthesia. 2009;64:122–5
42. Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity. Reg Anesth Pain Med. 2012;37:8–15
43. Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. BMJ. 2004;328:1040
44. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AASafe Surgery Saves Lives Study Group. . A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–9
45. American National Standards Institute/Association for the Advancement of Medical Instrumentation.HE74:2001 Human Factors Design Process for Medical Devices. 2001 Arlington, VA Association for the Advancement of Medical Instrumentation,
46. Klein G, Feltovich PJ, Bradshaw JM, Woods DDRouse WB, Boff KR. Common ground and coordination in joint activity. Organizational Simulation. 2005 New York, NY John Wiley and Sons Inc:139–84
47. Carvalho PVR, dosSantos IL, Vidal MCR. Nuclear power plant shift supervisor’s decision making during microincidents. Int J Ind Ergon. 2005;35:619–44
48. McDaniel MA, Einstein GO. Strategic and automatic processes in prospective memory retrieval: A multiprocess framework. Appl Cogn Psychol. 2000;14:S127–44
49. International Liaison Committee on Resuscitation. ILCOR CoSTR consensus timeline. 2011 Available at:
http://www.ilcor.org/consensus-2010/timeline/. Accessed on June 12,2013
50. Rall ME, Gaba DM, Dieckmann P, Eich CMiller RD. Patient simulation. Anesthesia. 20097th ed Philadelphia, PA Churchill Livingstone:151–92
