Content of the Cognitive Aid
Seven of the 22 cognitive aids were developed to help in a range of anesthetic crises, whereas the rest helped with specific crises or related groups of crises such as cardiac arrest or airway emergencies. The Abbreviated Stanford Manual was developed from one of the earliest cognitive aids found for a variety of crises in anesthesia, the crisis management handbook by Gaba et al.,22 now nearly 20 years old.
The content of 13 cognitive aids was derived from existing national guidelines18,19,26,29,30,32–37,40,42 including all of the aids relating to management of cardiac arrest of adults32–37 and neonates.40 This may reflect the existence of established national and international guidelines for cardiopulmonary resuscitation.
Most of the aids with content derived from local expert consensus did not explain how the content was decided upon. The exception to this was the Australian Patient Safety Foundation (APSF) crisis management handbook,24 which has been extensively documented.23 A series of 7 national meetings of up to 100 anesthesiologists per meeting was held, and a consensus for ideal management was reached for each crisis. The authors subsequently condensed this content into a 74-page manual that used a common mnemonic for many emergency situations in anesthesia.
Design of the Aid
The 22 cognitive aids discovered by this search were presented in forms as diverse as handbooks, cards, posters, or desktop or handheld computer-based help systems (Table 2). Two of the studies did not describe how the aid was presented.28,29
For 11 of the 22 cognitive aids, the design process was not described. Mills et al.34 described a subjective method of design by a working group, with the addition of an index and ring for attachment to the code cart. Participants reported that the additions improved the ease of use, although none of the questions in a later survey specifically addressed aspects of design. Only 1 of the 22 cognitive aids was developed using an iterative method that is the standard for other medical devices.27
Evaluation of the Aid
The cognitive aids were evaluated by surveys, simulation studies, and analyses of case reports (Table 3). The 3 studies that collected survey data21,34,41 concentrated mainly on how often the aid was used and how aware clinicians were of it in the clinical setting. These survey data will be dealt with in the section on implementation. Where the cognitive aids were tested, it appears they were not measured against any criteria, and in only 1 was the design altered to improve the usability as suggested by the international standards for medical devices (IEC 62366 Sections 5.6 and 5.9).14
Simulation-based studies were used to evaluate whether the cognitive aids could be useful in emergencies. In 13 studies, researchers assessed technical performance such as speed of task completion and number of errors and omissions.18,19,25–27,31–33,36–38,40,42 All but one of the studies used a part task trainer or mannequin-based simulators. The exception was the study by Berkenstadt et al.25 that used a screen-based computer simulator to determine whether an electronic help system reduced the number of knowledge-based errors.
Ten of the 13 studies suggested an improvement in technical performance in diverse activities ranging from the management of an airway to managing malignant hyperthermia crises. Two of the remaining 3 studies showed no improvement in neonatal (Bould et al.40) and adult (Schneider et al.32) resuscitation. Coopmans and Biddle26 showed a longer delay to diagnose and intervene during emergencies when a personal digital assistant device was provided.
Observational studies and case reports have also been used to determine the effect of introducing cognitive aids. Both Heidegger et al.28 and Combes et al.29 observed fewer failed intubations when a standardized approach was used. Marshall and Flanagan39 reported that treating anesthesiologists believed that successful management of malignant hyperthermia was due to the use of a cognitive aid improving their task management, reducing omissions, and improving team performance.
The Australian Patient Safety Foundation took a larger scale approach by retrospectively analyzing thousands of incident reports from a national database. The first 2000 reports were used to determine whether a standardized approach (the “COVERABCD” algorithm) could have prevented or mitigated the incident using a “walkthrough” of the algorithm described by the cognitive aid.23 They found that 60% of incidents could have been rectified in <1 minute had the aid been used.
Training in the Use of the Cognitive Aid
Reports of formal orientation to the users about the cognitive aid were found in 8 of the 23 evaluation studies.18,25–27,32,33,36,37 In 3 of the remaining 15 studies, the cognitive aid was deliberately not shown to participants before they used it, but participants were informed that a cognitive aid would be present.31,38,40 Presumably, this strategy was taken to determine whether the aid could be used effectively without prior knowledge of its contents or structure. In 2 studies that found no difference in performance while using a cognitive aid, participants were allowed to familiarize themselves with the aid before using it.26,32
Effects on Team Functioning
Only 4 aids have been evaluated with regard to their effects on team functioning beyond the ability to merely perform tasks accurately and efficiently.18,20,40,42 Manser et al.20 observed the ability of teams to manage a malignant hyperthermia crisis when they referred to cognitive aids on personal digital assistants or cards that they carried. Teams that scored highly on a clinical performance score were more likely to discuss their assessment of the situation than the division of tasks. It was suggested that the cognitive aid might have affected how the teams coordinated their activities. Three studies failed to find an improvement in team functioning. Studies by Neal et al.42 and Bould et al.40 observed individual anesthesia trainees working with “actors” performing the supporting clinical roles. No differences in nontechnical skills were found between participants using versus not using a cognitive aid in either study, although Neal et al.42 suggested a slight improvement in the Decision Making category in the participants with the cognitive aid. Burden et al.18 noted a reduction in the volume of communication within the team when a cognitive aid was used.
Evidence of Implementation
Evidence of implementation in the actual clinical environment was identified in 3 studies.21,34,41 Two of these papers reported surveys of the use of cognitive aids in clinical settings. In each of these surveys only half of the respondents reported that they either would use or had used cognitive aids in emergencies.
The literature on cognitive aids in anesthesia emergencies appears to be expanding rapidly; 9 of the 23 articles found published in the last 3 years. Given the broad nature of the search and manual review of the papers, all of the cognitive aids currently available for anesthetic emergencies are likely to have been included.43 Some cognitive aids that may improve patient safety, such as the World Health Organization Surgical Safety Checklist,44 have not been included as they are not generally used during an emergency. There is some evidence that cognitive aids improve technical performance during emergencies, but there is much to be learned about when and why they fail.
Drawing the analogy between cognitive aids and medical devices may help us understand when and in what form cognitive aids may be useful. In common with medical devices, cognitive aids need to have the correct content, be well designed, and be accompanied by appropriate training to assist task performance. The content or knowledge contained in cognitive aids should be developed from national or international guidelines or from broad consensus. To ensure continuing usefulness, their content needs to be reviewed and adapted as knowledge changes.
The content seems to have been the main focus during the development of most cognitive aids, with more than half being derived from established guidelines. In contrast, the design processes, presentation, and the resultant usability of cognitive aids seem to have been less thoroughly considered. Only one of the cognitive aids was developed with a systematic design process and adapted as a result of evaluation data.27 Evidence from the human factors literature suggests that poorly designed cognitive aids may lead to unintended consequences.10,13 The use of a human factors design process, as mandated for medical devices,45 may help to reduce the likelihood of poor design but will not eliminate it. It is also important to select appropriate measurements and testing scenarios, and then revise the design appropriately. There is little evidence that the majority of cognitive aids discovered in this literature review have been designed for the context in which they are to be used. Physical constraints of using aids, such as the ability to read, see, hear, or interact with the paper or computer-based aid, potential distraction caused by the aid, and the ability to perform concurrent tasks have not been addressed. These contextual issues could make aids very difficult to use in the actual working environment and may reflect the apparent negative effects on team communication that have been reported.18 Research is currently lacking in this area not only in anesthesia and health care but also in other industries, despite the acknowledgment that checklist design has been a contributing factor in several airline accidents.11 There are functional differences between a long, nonsequential checklist and dynamic decision tree, and the design chosen depends on the intended context (Table 1).
Despite the perceived benefits to the team, there is minimal evidence to support an improvement in team function with the use of cognitive aids. Data suggest that cognitive aids may change team coordination and improve task completion but their effects on team processes are not clear. In particular, the study by Burden et al.18 into the effects of a designated reader demonstrated decreased communication by the team. Reduction in team communication is generally considered deleterious to team coordination,46 and as such the aid may have distracted the team from communicating with each other. Conversely, it may be that the remaining communication was more efficient and targeted. Further studies are needed to determine how cognitive aids affect interpersonal communication during anesthetic emergencies.
Training in the use of a cognitive aid appears to be often cursory or absent. It is reasonable to assume that familiarization to a cognitive aid before its use would mean that the participants would be more likely to use it and use it more effectively. This increased likelihood of use after education is supported by responses to Mills et al.’s survey34 about a cognitive aid for cardiac arrest management: those who had learned about the aid from a formal orientation session were more likely to use it in an emergency than those who were not oriented to the aid. Research shows that education raises awareness of the presence of the aid and helps identify how and when the aid should be used, both stimulating its use and making its use more effective.47 The counter argument is that a cognitive aid should be intuitive to use in an emergency situation, and that educating people to use a cognitive aid merely compensates for a poor design. Ideally, as with any medical device, for the best results the cognitive aid should be intuitive to use and should also be used only by individuals trained in its use.
Reports of implementation into clinical practice demonstrate that even when clinicians are aware of the existence of cognitive aids they often do not use them. Part of the problem of the underuse of cognitive aids may be the existence of a professional culture that does not support their use. For example, when observing the use of cognitive aids in malignant hyperthermia crises, one of the participants in Harrison et al.’s study19 reported the view that using a cognitive aid reflected a lack of confidence or knowledge. Several other studies also report participants’ comments that they did not need the cognitive aid to manage the emergency effectively.19,21,34,40 In contrast, the results of Low et al.’s study37 with junior doctors suggest a willingness to use cognitive aids, and their feeling that using them does not reflect a lack of competence of health professionals. A further issue that is rarely addressed is the positioning of the cognitive aid. A reminder of the aid and therefore prompting to use it can be improved by associating the aid with the task in the same way that any prospective memory task can be prompted.48 For example, the cognitive aid for management of local anesthetic toxicity may be positioned with the Intralipid® on an emergency cart.
A recent editorial suggested that cognitive aids should be available for all rare emergencies in anesthesia7 but this poses more questions, such as who creates, updates, and designs the cognitive aids. A regular formal review of laboratory and clinical evidence by a reputable body would be required for each emergency. The process currently used for the management of cardiac arrest by the International Liaison Committee on Resuscitation49 would seem to be the ideal template. Levels of evidence should ideally be included and should lead to a consensus about the actions required during the emergency. This information would then be passed to a human factors design team for design, testing, and modification based on heuristic and simulation evaluation. Only then would the guidelines be available for distribution.
Cognitive aids are not commonly used during emergencies in anesthesia and at present appear not to be supported by the culture. For cognitive aids to be more broadly accepted, further evidence may be needed that they confer benefits in emergency situations. Although we know that practitioners’ coordination patterns change when a cognitive aid is used, there is a need for larger prospective trials of the effect of aids on task completion, practitioners’ team behaviors, and overall team functioning. Such trials will allow a deeper understanding of how teams may use an aid to best allocate roles so that they can rapidly process tasks in parallel. The trials should use robustly researched and designed aids and should assess outcome measures of accurate and rapid task completion as well as process measures of team behaviors. Testing would best occur in simulation-based settings, where the physiological variables and environment can be more tightly controlled and replicated, and there are no direct risks to patients.
Cognitive aids should be integrated into anesthesiologists’ vocational and continuing education so that practitioners are aware of when and how to use them appropriately. Continuing education programs have started to advocate the use of cognitive aids in emergencies through simulation-based education50 but such education is not readily accessible to all anesthesia providers. Any future testing of cognitive aids in simulation-based settings should first let practitioners become oriented to the aids. The maximum benefit is likely only if practitioners are familiar with the structure of each aid, and how it should be used.
As this literature review has shown, the current evidence for the efficacy of cognitive aids in emergencies is inconclusive. However, although the evidence to support the use of cognitive aids in emergencies is currently weak, the success in other settings is compelling. The lack of evidence is due to both the limited research that has been performed and the deficiencies in design and evaluation of current cognitive aids. By conceptualizing cognitive aids as medical devices, we may be able to address these deficiencies and improve the outcomes of patients experiencing anesthetic emergencies.
Name: Stuart Marshall, MB, ChB, M.HumanFact.
Contribution: Study design, conduct, data collection, and manuscript preparation were all undertaken by the author.
Attestation: Stuart Marshall approved the final manuscript and attests to the integrity of the original data and analysis reported in this manuscript.
This manuscript was handled by: Sorin J. Brull, MD, FCARCSI (Hon).
I would like to acknowledge the invaluable assistance and guidance of Professor Penelope Sanderson, Dr. Cate McIntosh, and Dr. Brendan Flanagan in the preparation of this manuscript.
1. Meilinger PS. When the Fortress went down. Airforce Mag. 2004:78–82
2. Reason J. Cognitive aids in process environments: prostheses or tools? Int J Man Mach Stud. 1987;27:463–70
3. Roth EM, Mumaw RJ, Lewis PM An Empirical Investigation of Operator Performance in Cognitively Demanding Simulated Emergencies. 1994 Washington, DC US Nuclear Regulatory Commission Contract No.: NUREG/CR-6208
4. Winters BD, Gurses AP, Lehmann H, Sexton JB, Rampersad CJ, Pronovost PJ. Clinical review: checklists—translating evidence into practice. Crit Care. 2009;13:210
5. Morris AH. Decision support and safety of clinical environments. Qual Saf Health Care. 2002;11:69–75
6. Wieringa D, Moore C, Barnes V Procedure Writing Principles and Practice. 1993 Piscataway, NJ IEEE Press
7. Nanji KC, Cooper JB. It is time to use checklists for anesthesia emergencies: simulation is the vehicle for testing and learning. Reg Anesth Pain Med. 2012;37:1–2
8. Kuhlmann S, Piel M, Wolf OT. Impaired memory retrieval after psychosocial stress in healthy young men. J Neurosci. 2005;25:2977–82
9. Xiao Y, Mackenzie CF. Decision making in dynamic environments: fixation errors and their causes. Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. 1995 San Diego, CA Human Factors and Ergonomics Society
10. Mosier KL, Palmer EA, Degani A. Electronic checklists: implications for decision making. Proceedings of the Human Factors Society 36th Annual Meeting. 1992 Atlanta, GA Human Factors and Ergonomics Society
11. Degani A, Wiener EL. Cockpit checklists: concepts, design and use. Hum Factors. 1993;35:345–59
12. Nelson K, Shilkofski N, Haggerty J, Vera K, Saliski M, Hunt E. Cognitive aids do not prompt initiation of cardiopulmonary resuscitation in simulated pediatric cardiopulmonary arrest. Simul Healthc. 2007;2:54
14. International Electrotechnical Commission.IEC 62366, Ed.1: Medical Devices—Application of Usability Engineering to Medical Devices. 2007 Boston, MA International Electrotechnical Commission
15. American National Standards Institute/Association for the Advancement of Medical Instrumentation.HE75:2009 Human Factors Engineering—Design of Medical Devices. 2009 Arlington, VA Association for the Advancement of Medical Instrumentation
16. Weinger MB, Wiklund ME, Gardner-Bonneau DJ Handbook of Human Factors in Medical Device Design. 2011 Boca Raton, FL CRC Press
17. . Flight crewmember duties. Code of Federal Regulations (Aeronautics and Space), Part 121 (Operating Requirements) Subpart T Flight Operations. Section 121.542
18. Burden AR, Carr ZJ, Staman GW, Littman JJ, Torjman MC. Does every code need a “reader?” improvement of rare event management with a cognitive aid “reader” during a simulated emergency: a pilot study. Simul Healthc. 2012;7:1–9
19. Harrison TK, Manser T, Howard SK, Gaba DM. Use of cognitive aids in a simulated anesthetic crisis. Anesth Analg. 2006;103:551–6
20. Manser T, Harrison TK, Gaba DM, Howard SK. Coordination patterns related to high clinical performance in a simulated anesthetic crisis. Anesth Analg. 2009;108:1606–15
21. Neily J, DeRosier JM, Mills PD, Bishop MJ, Weeks WB, Bagian JP. Awareness and use of a cognitive aid for anesthesiology. Jt Comm J Qual Patient Saf. 2007;33:502–11
22. Gaba DM, Fish KJ, Howard SK Crisis Management in Anesthesiology. 1994 New York, NY Churchill Livingstone
23. Runciman WB, Kluger MT, Morris RW, Paix AD, Watterson LM, Webb RK. Crisis management during anaesthesia: the development of an anaesthetic crisis management manual. Qual Saf Health Care. 2005;14:e1
24. Australian Patient Safety Foundation. Crisis Management Manual: ‘COVER ABCD A SWIFT CHECK. 1996 Adelaide, Australia Australian Patient Safety Foundation
25. Berkenstadt H, Yusim Y, Katznelson R, Ziv A, Livingstone D, Perel A. A novel point-of-care information system reduces anaesthesiologists’ errors while managing case scenarios. Eur J Anaesthesiol. 2006;23:239–50
26. Coopmans VC, Biddle C. CRNA performance using a handheld, computerized, decision-making aid during critical events in a simulated environment: a methodologic inquiry. AANA J. 2008;76:29–35
27. Ziewacz JE, Arriaga AF, Bader AM, Berry WR, Edmondson L, Wong JM, Lipsitz SR, Hepner DL, Peyre S, Nelson S, Boorman DJ, Smink DS, Ashley SW, Gawande AA. Crisis checklists for the operating room: development and pilot testing. J Am Coll Surg. 2011;213:212–217.e10
28. Heidegger T, Gerig HJ, Ulrich B, Kreienbühl G. Validation of a simple algorithm for tracheal intubation: daily practice is the key to success in emergencies–an analysis of 13,248 intubations. Anesth Analg. 2001;92:517–22
29. Combes X, Le Roux B, Suen P, Dumerat M, Motamed C, Sauvat S, Duvaldestin P, Dhonneur G. Unanticipated difficult airway in anesthetized patients. Anesthesiology. 2004;100:1146–50
30. Berkow LC, Greenberg RS, Kan KH, Colantuoni E, Mark LJ, Flint PW, Corridore M, Bhatti N, Heitmiller ES. Need for emergency surgical airway reduced by a comprehensive difficult airway program. Anesth Analg. 2009;109:1860–9
31. Seagull FJ, Ho D, Radcliffe J, Xiao Y, Hu P, Mackenzie CF. Just-in-time training for medical emergencies: Computer versus paper checklists for a tracheal intubation task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2007 Baltimore, MD Human Factors and Ergonomics Society
32. Schneider AJ, Murray WB, Mentzer SC, Miranda F, Vaduva S. “Helper:” A critical events prompter for unexpected emergencies. J Clin Monit. 1995;11:358–64
33. Ward P, Johnson LA, Mulligan NW, Ward MC, Jones DL. Improving cardiopulmonary resuscitation skills retention: effect of two checklists designed to prompt correct performance. Resuscitation. 1997;34:221–5
34. Mills PD, DeRosier JM, Neily J, McKnight SD, Weeks WB, Bagian JP. A cognitive aid for cardiac arrest: you can’t use it if you don’t know about it. Jt Comm J Qual Saf. 2004;30:488–96
35. Dyson E, Voisey S, Hughes S, Higgins B, McQuillan PJ. Educational psychology in medical learning: a randomised controlled trial of two aide memoires for the recall of causes of electromechanical dissociation. Emerg Med J. 2004;21:457–60
36. Merchant RM, Abella BS, Abotsi EJ, Smith TM, Long JA, Trudeau ME, Leary M, Groeneveld PW, Becker LB, Asch DA. Cell phone cardiopulmonary resuscitation: audio instructions when needed by lay rescuers: a randomized, controlled trial. Ann Emerg Med. 2010;55:538–543.e1
37. Low D, Clark N, Soar J, Padkin A, Stoneham A, Perkins GD, Nolan J. A randomised control trial to determine if use of the iResus© application on a smart phone improves the performance of an advanced life support provider in a simulated medical emergency. Anaesthesia. 2011;66:255–62
38. Hart EM, Owen H. Errors and omissions in anesthesia: a pilot study using a pilot’s checklist. Anesth Analg. 2005;101:246–50
39. Marshall S, Flanagan B. Cognitive AIDS in a simulated anesthetic crisis. Anesth Analg. 2007;104:1292–3 author reply 1293
40. Bould MD, Hayter MA, Campbell DM, Chandra DB, Joo HS, Naik VN. Cognitive aid for neonatal resuscitation: a randomized controlled trial. Br J Anaesth. 2009;103:570–5
41. Picard J, Ward SC, Zumpe R, Meek T, Barlow J, Harrop-Griffiths W. Guidelines and the adoption of ‘lipid rescue’ therapy for local anaesthetic toxicity. Anaesthesia. 2009;64:122–5
42. Neal JM, Hsiung RL, Mulroy MF, Halpern BB, Dragnich AD, Slee AE. ASRA checklist improves trainee performance during a simulated episode of local anesthetic systemic toxicity. Reg Anesth Pain Med. 2012;37:8–15
43. Haynes RB, Wilczynski NL. Optimal search strategies for retrieving scientifically strong studies of diagnosis from Medline: analytical survey. BMJ. 2004;328:1040
44. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, Herbosa T, Joseph S, Kibatala PL, Lapitan MC, Merry AF, Moorthy K, Reznick RK, Taylor B, Gawande AASafe Surgery Saves Lives Study Group. . A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360:491–9
45. American National Standards Institute/Association for the Advancement of Medical Instrumentation.HE74:2001 Human Factors Design Process for Medical Devices. 2001 Arlington, VA Association for the Advancement of Medical Instrumentation,
46. Klein G, Feltovich PJ, Bradshaw JM, Woods DDRouse WB, Boff KR. Common ground and coordination in joint activity. Organizational Simulation. 2005 New York, NY John Wiley and Sons Inc:139–84
47. Carvalho PVR, dosSantos IL, Vidal MCR. Nuclear power plant shift supervisor’s decision making during microincidents. Int J Ind Ergon. 2005;35:619–44
48. McDaniel MA, Einstein GO. Strategic and automatic processes in prospective memory retrieval: A multiprocess framework. Appl Cogn Psychol. 2000;14:S127–44
© 2013 International Anesthesia Research Society
50. Rall ME, Gaba DM, Dieckmann P, Eich CMiller RD. Patient simulation. Anesthesia. 20097th ed Philadelphia, PA Churchill Livingstone:151–92