Harrison, T Kyle MD; Manser, Tanja PhD; Howard, Steven K. MD; Gaba, David M. MD
Malignant hyperthermia (MH) is a rare genetic disorder of metabolism that, when triggered by drugs used in anesthesia, creates a major crisis. The incidence of MH ranges from 1 in 62,000 to 1 in 84,000 for general anesthetics in which succinylcholine and inhaled anesthetics are used (1). With this incidence, most anesthesia professionals will never treat a case of MH during their career. Once the diagnosis of MH has been made, a complex and specific treatment plan needs to be implemented quickly and efficiently to prevent a fatal outcome. A treatment scheme has been produced by the Malignant Hyperthermia Society of the United States (MHAUS; www.mhaus.org) to guide practitioners in the treatment of MH.
As a general rule, medicine has relied heavily on clinicians' recall of information as the primary basis for treatment decisions even in emergent settings. Relying on memory may work well when the situation is familiar and there is little stress, but during a stressful situation, memory is likely to be error-prone, resulting in omissions or treatment missteps (2). Acute stress can also have a negative effect on other cognitive functions with a decrease in both selective and divided attention (3).
Written or computerized presentation of important information can be referred to generically as “cognitive aids.” Cognitive aids are not just for novices, nor are they simply “learning aids.” Rather, they provide prompts for and reviews of important diagnostic or corrective actions that can be used during case management by all levels of personnel. A few written cognitive aids are available for routine and emergency procedures in health care. In 1993, the United States Food and Drug Administration developed a pre-use anesthesia apparatus checkout recommendation (4). Despite the importance of the anesthesia machine check, the Food and Drug Administration's checklist has been found to be poorly understood and difficult to use (5,6). Recently, a checklist was found to be useful before general anesthesia for simulated Cesarean delivery (7). Advanced Cardiac Life Support treatment guidelines have been available for a number of years, but their use is variable and the full Advanced Cardiac Life Support algorithms are difficult to navigate during a crisis (8,9). Despite these and other examples, the use of cognitive aids in routine or emergent patient care settings is not common in health care (10), in contrast to other high-risk industries such as aviation and nuclear power.
We hypothesized that cognitive aids would be useful in the treatment of a rare event like MH. We empirically evaluated the extent to which the use of cognitive aids during a simulated MH event facilitated correct and prompt treatment of the event.
We reviewed audiovisual recordings of 48 scenarios of simulated MH; 24 involving residents in their first Clinical Anesthesia year (CA 1) and 24 involving CA 2 residents. All subjects gave informed consent to participate in this IRB-approved study.
The simulation scenarios were part of the Anesthesia Crisis Resource Management (ACRM) course conducted at Stanford/Veterans Administration Health Care System Palo Alto from 1998–2003 (11,12). The simulations were performed in a simulation center using a high capability patient simulator (MedSim Eagle, Sarasota, FL). During the scenarios an experienced anesthesiologist instructor played the surgeon's role. The circulating nurse was played by a retired operating room nurse, and the “scrub tech” was played by one of the participants. In the scenario, anesthetic care was handed off to the primary participant by a confederate anesthesiologist after induction of anesthesia was complete. Another course participant of equal level of training was sequestered in a sound-proof room and could be summoned for help by the primary anesthesiologist if requested.
The case scenario for the CA 1 participants involved a general anesthetic on a healthy adult female undergoing a knee operation. Succinylcholine had been used and anesthesia was maintained with isoflurane and nitrous oxide. The case scenario for the CA 2 participants was more complex and involved a laparoscopic appendectomy on a female patient with a history of Graves' disease. In this case, succinylcholine and isoflurane were also used. For both scenarios, the patient became progressively unstable over 15–20 min with signs of hypermetabolism (increasing heart rate, arterial blood pressure, temperature, and increased production of carbon dioxide) and hyperkalemia (peaked T waves and increasingly frequent premature ventricular contractions). Although participants had to diagnose the presence of MH, in this experiment we studied only the treatment of MH but not the timing or pathway to the diagnosis. Performance was scored only from the point when the participant(s) articulated a diagnosis of MH.
Subjects were instructed to bring to the simulated operating room any materials that they routinely carried into a regular day of work, including any books or personal data assistants. If participants requested the MH cart, they were brought a large box containing all critical items needed to treat MH. Easily accessible in the MH box (placed on top of the other supplies) was the MHAUS poster detailing appropriate treatment of MH. We did not provide formal training concerning the treatment of MH, the contents of the MH box, or the use of cognitive aids before the simulation.
We developed a scoring system to assess the clinical performance of the team, based on a system we described in a previous publication (13) and similar to scoring systems used by other investigators (14) (15). Points were assigned for the various treatment steps needed to successfully treat an episode of MH as outlined by MHAUS (Table 1). The videotaped scenarios were analyzed by two of the authors (TM and KH). For each treatment step, we recorded the time from articulating the diagnosis of MH (time 0) until successful completion of that step by any member of the team. A MH treatment score was calculated from the point values of steps accomplished.
We attributed an action to the use of a cognitive aid if a) an individual's action immediately followed reading a cognitive aid or b) any member of the team stated that the action was in response to an item on a cognitive aid. In addition to the MH treatment score, the overall frequency of cognitive aid use was scored independently by the authors (TM and KH) using a 5-point scale with 0 = no use, 1 = minimal use, 2 = occasional use, 3 = intermittent use, 4 = frequent use, and 5 = extensive use. Extensive use required the participants to use an aid throughout the entire treatment of the MH crisis.
Aggregate data are shown as mean ± sd unless otherwise specified. MH treatment scores and ratings for frequency of cognitive aid use were correlated separately for the CA 1 and the CA 2 group using nonparametric (Spearman) correlation (because the frequency scale was ordinal). In addition we calculated for each treatment step the percentage of CA 1 and CA 2 teams that had used a cognitive aid. Both observers rated the frequency of cognitive aid use independently. A Cohen's kappa of 0.94 for their ratings of the same teams indicates excellent interobserver agreement (16, 17).
In every scenario the primary anesthesiologist requested and received assistance from the sequestered anesthesiologist. All teams of anesthesiologists made the diagnosis of MH except for one set of CA 2 participants who attributed all symptoms to hyperthyroidism. Because the scoring system is based on the time from articulation of the diagnosis of MH to the occurrence of treatment steps, we excluded this case from the analysis. The average time from triggering the MH event on the simulator to the participants declaring an MH emergency was 19 (±4.5) min for the CA 1 group and 26 (±9.4) min for the CA 2 group (likely related to the confounding variable of Graves' disease and the possibility of thyroid storm). The average treatment score for the CA 1 group was 19.8 (±3.9) and for the CA 2 Group it was 21.4 (±3.3).
The most frequently used cognitive aid was the MHAUS poster that was provided in the MH box. Some teams used an alternative aid, such as a textbook or personal digital assistant reference, but none of the teams used an alterative aid frequently or extensively. Of the top scoring teams that used an aid, all used the MHAUS poster. In the CA 1 group, 12 teams used the poster and 7 teams used an alternative aid. In the CA 2 group, 16 teams used the MHAUS poster and 2 teams used an alternative aid. In the CA 1 group, 19 of the 24 teams (79%) used some form of a cognitive aid, but only 8 of the 19 teams used it frequently or extensively. In the CA 2 group, 18 of the 23 teams (78%) used some form of a cognitive aid at some point, but only 6 of them used it frequently or extensively. The frequency of cognitive aid use correlated significantly with the MH treatment score for the CA 1 group (Spearman r = 0.59, P < 0.01) as well as for the CA 2 group (Spearman r = 0.68, P < 0.001).
In the CA 1 group, the 5 highest scoring teams (average MH treatment score of 24.2) had a mean score for cognitive aid use frequency of 4.2 and the 5 lowest scoring teams (average treatment score of 14.4) had a mean frequency of use score of 1. In the CA 2 group, the 5 highest scoring teams (average MH treatment score of 24.4) had a cognitive aid use frequency score of 4.4 and the 5 lowest scoring teams (treatment score of 16.8) and a frequency of use score of 0.4 (Fig. 1).
As summarized in Table 2, 41 teams determined the correct dose of dantrolene, with 26 teams (55%) using a cognitive aid to determine that dose (23 teams using the MHAUS poster and 3 teams using an alternative aid). The average time from diagnosis until the injection of the first dose of dantrolene was 7.2 (±3.8) min for the CA 1 group and 6.5 (±3.2) min for the CA 2. Only 10 of 24 teams (42%) mixed the dantrolene correctly in the CA 1 group and 18 of 23 (78%) mixed the dantrolene correctly in the CA 2 group. Of the teams that mixed the dantrolene correctly, 70% used a cognitive aid in the CA 1 group and 72% did in the CA 2 group. For the CA 1 teams that mixed the dantrolene incorrectly, 8 used both the wrong volume and the wrong diluent (typically lactated Ringer's solution instead of sterile water). Five teams used the wrong diluent but the correct volume, and one team used the wrong volume but correct diluent. Of the 5 CA 2 teams that mixed the dantrolene incorrectly, 3 teams used the wrong volume, 1 team the wrong diluent, and 1 team both. Of the 19 teams that mixed the dantrolene incorrectly, only 1 team used a cognitive aid.
The treatment step that was most often associated with the use of a cognitive aid was determining the correct dose of dantrolene, with 41% of the CA 1s using a cognitive aid and 69% of CA 2s using an aid. The most frequently missed treatment step was treating hyperkalemia (typically >6.0 mEq/L), which was also manifested by peaked T-waves and frequent premature ventricular contractions. More than half of teams (58% of CA 1s and 69% of CA 2s) failed to treat hyperkalemia.
A simulated MH event is an excellent model to test how anesthesia providers respond to a rare but potentially lethal condition. With the widely accepted and defined treatment protocol for MH, we were able to assess how the use of a cognitive aid affected overall performance. We assumed that every duo of anesthesia providers should be able to achieve a perfect score for the MH treatment and that any deficiency could represent a significant problem with the care delivered.
In this study, the average MH treatment score for all participants was 20 (of 25) but ranged from 8 to 25, with most provider teams failing to accomplish at least one major treatment step. We found a statistically significant association between frequency of cognitive aid use and MH treatment performance. The teams that performed the best, as measured by treatment score, used a cognitive aid extensively throughout the simulation and the teams that performed worst failed to use any form of a cognitive aid (Fig. 1). However, use of the cognitive aid explained only 1/3 to 2/3 of the variance in treatment score. There were probably many other factors that affected the adequacy of treatment, including adequacy of teamwork and skill in executing the actions recommended by the cognitive aid.
We observed several “patterns” of cognitive aid use, suggesting that it is not only how frequently an aid is used but also how an aid is integrated in the process of managing an anesthetic crisis that determines performance. Some teams used a cognitive aid early in the MH event and continued to cross-check it throughout the scenario. Several teams read the instructions out loud and other team members responded; this helped establish shared situational awareness. Teams using these two strategies were the highest performing in the study. Other teams did not use a cognitive aid at all or used it only late in the scenario. Some teams that used an aid late often did pick up previously forgotten steps, such as treating hyperkalemia and acidosis. Some individuals consulted an aid but failed to share the information with other team members, resulting in a delay in appropriate management.
Observing participants' use of the MHAUS cognitive aid, it was apparent that the existing cognitive aid failed to clarify relative priorities. In particular, the need to expedite delivery of dantrolene as the highest priority is not transmitted optimally in this cognitive aid. Although the CA 2 teams were faster at administering the first and additional doses of dantrolene as compared to the CA 1 group (Table 3), some teams spent considerable resources on lower priority actions (e.g., attempting to replace the anesthetic machine, placing a nasogastric tube for topical cooling) and thereby delayed administering an appropriate dose of dantrolene. Also, many participants hyperventilated the lungs by hand, occupying one member of the team on a task that could have easily been accomplished by mechanical ventilation. i Gardi et al. (14) also observed this behavior in their study of performance managing simulated MH. Optimal cognitive aids should anticipate common errors and provide prioritized and explicit instructions to prevent them.
A surprising finding of this study was that so many of the anesthesia teams failed to mix the dantrolene correctly. Despite the label on the bottle of dantrolene stating how to properly mix the drug, and the same directions also being printed on the MHAUS treatment poster, 45% of the teams still mixed the drug incorrectly.
The time to diagnosis was longer for the CA 2 than the CA 1 group. In the CA 2 case there were three major confounding factors that might have caused the delay in diagnosis. First, the patient had a history of Graves' disease and it was plausible (at first) that the patient was experiencing “thyroid storm.” In addition, the patient was undergoing laparoscopic surgery for appendicitis and thus the patient's increased CO2 could have been related to insufflation of CO2, and the patient's fever and hyperdynamic hemodynamic state might have resulted from sepsis. In addition, the CA-2 participants in ACRM2 had seen a case of MH in the ACRM1 course the previous year and thus they might have inferred that a repeat MH case was unlikely. From post-scenario think-aloud debriefing, (although we did not formally measure this), it appears that the confounding diagnosis of Graves' disease was by far the most likely cause for the delay in diagnosis. Although a cognitive aid might help some individuals to make the diagnosis of MH in the face of such uncertainty, this was not evaluated in this study. We evaluated the use of a cognitive aid only from the time the diagnosis was made.
This study has a number of limitations. First, we do not know the effect on patient outcome from failing to complete some of the treatment steps outlined in the MHAUS treatment protocol or of executing them incorrectly. It seems likely that major deviations from the protocol would lead to suboptimal outcomes or to greater liability exposure. Another limitation is that the study was retrospective, using scenarios that were designed for teaching rather than for the primary purpose of investigation. The scenarios somewhat compress time relative to the probable time course of MH in real patients; thus, it is not certain that failures or errors seen during simulations would always occur in real patient care. Because of the retrospective character of this study, it was not possible to obtain systematic interview data from the participants regarding their use of the cognitive aid or their reasons for not using an aid. This means that we cannot determine how often a cognitive aid provided information that the team did not already posses, versus reminding them about or confirming their prior knowledge. Further, during a real MH emergency during normal working hours, more personnel might be available to provide their knowledge and physical help. In addition, the participants were anesthesia residents and not fully trained anesthesia professionals. However, in the study by i Gardi et al. (14) looking at the management of MH in experienced teams consisting of nurse-anesthetists (average 8.3 years experience) and anesthesiologists (average 9.8 years experience), only half of the teams administered sodium bicarbonate and only a third administered insulin and glucose for hyperkalemia, findings similar to our own observations of residents.
An additional limitation of this is study is that we cannot know the amount of prior knowledge the participants had in the diagnosis and treatment of MH. The residents received one formal lecture on drug-induced hypermetabolic disorders (including MH) in their residency lecture series. During the ACRM2 course, we scheduled scenarios such that the CA 2 participants who participated in the “hard” MH scenario would not have been either the primary anesthesiologist or first responder anesthesiologist in the “easy” MH scenario that was run during the ACRM1 course in their CA1 year. However, those CA2 residents would have witnessed and participated in the group debriefing of the MH scenario during ACRM1. If they remembered their ACRM1 experience it might have assisted them in managing MH in the ACRM2 course over 1 year later.
Recently, the Veterans Health Care Administration (VHA), in conjunction with the VHA National Center for Patient Safety, has produced a series of cognitive aids for crisis events in anesthesia. These aids are now placed on all anesthesia machines in the VHA system. In a follow-up survey of VHA anesthesia providers, they found that 87% of the respondents had seen the aid and half had used it as a reference but that only 7% had used it in an emergency. However, of the small number of providers who had used it in an emergency, all felt that it had helped (personal communication, J. Neily et al. VA National Center for Patient Safety, 2005).
Hart and Owen (7) have recently shown anesthesia providers on average omitted 13 of 40 steps when preparing to administer general anesthesia for cesarean delivery. When provided with a checklist, 90% of the participants found the checklist useful but only 40% stated that they would use it clinically. Morris (10) has shown that there are several barriers to physicians using guidelines and protocols. These include a lack of appreciation of the limitations of human decision-making, excessive complexity in the protocols, and the concern that the use of protocols could reduce the role for clinicians in medicine. In addition, in a critical time-sensitive event the use of an aid might be perceived as an additional burden, despite its potential benefits.
The use of cognitive aids in medicine differs from other industries such as aviation and nuclear power in many ways. The latter are highly regulated both by the employer and the government, and personnel in these industries are mandated to use checklist-type cognitive aids to provide a more consistent response in standard operating procedures. Medicine involves a much more complex and unpredictable system. A simple checklist may not be adequate to cover the complex environment in health care. However, for many unusual but acute conditions in medicine, a cognitive aid is likely to be helpful in providing additional therapeutic and diagnostic guidance to clinicians. A computerized system of cognitive aids, On-Line Electronic Help, designed for use at the point-of-care for routine and emergent anesthetic situations, has recently been developed by Berkenstadt et al. (18) and endorsed by the European Society of Anesthesiologists. The On-Line Electronic Help system has been shown to improve clinicians' performance for various scenarios on a “screen-based” anesthesia simulation (18).
Although we recommend that health care providers be encouraged to use cognitive aids, we also believe that the design of such aids should be improved. Although the pertinent information for the treatment of MH was provided on the MHAUS poster, the key information did not appear to be transmitted to the participants in all cases. Swain and Guttman (19) found that in nuclear power operations, as the number of items on a checklist increased, there was a greater probability of overlooking a given item. In addition, if a step could not be completed immediately when read on a checklist, the user would often try to store the step in their short-term memory, which is prone to failure under stressful conditions, resulting in omissions (20). Attempts to improve the MHAUS treatment poster could include breaking the various steps into discrete sections or separate aids (diagnosis, treatment, follow-up care) so that there are fewer steps in any one section. The most critical items should be listed first to increase the probability that they will not be missed or omitted. In addition, critically important steps might be repeated throughout the aid to reinforce their impact. For example, information on how to properly mix dantrolene or how to treat hyperkalemia could be duplicated in different places, and with a size and font that would make them stand out. Our findings support that conception and design of cognitive aids is an area for further research especially if they are to be effectively used in health care.
In other high-risk professions, such as aviation or spaceflight, checklists and other cognitive aids are used as part of standard operating procedures. Negative stereotypes about the use of cognitive aids in health care should be confronted. Although we did not question the participants directly about why they did not use a cognitive aid in this study, in informal discussions with residents, many felt that using a cognitive aid would tend to show weakness and give the impression that they did not know what to do in a crisis. In this study, not all of the teams needed a cognitive aid to perform well, but the teams that performed well often did use an aid, and the teams that performed the worst did not use any form of an aid. Because it is impossible to determine prospectively if a team will perform well in a crisis, it might be prudent to encourage all teams to refer to and use a cognitive aid in unfamiliar or life-critical situations to maximize the likelihood of successful resolution.
1. Halliday NJ. Malignant hyperthermia. J Craniofac Surg 2003;14:800–2.
2. Kuhlmann S, Piel M, Wolf OT. Impaired memory retrieval after psychosocial stress in healthy young men. J Neurosci 2005;25:2977–82.
3. Vedhara K, Hyde J, Gilchrist ID, Tytherleigh M, Plummer S. Acute stress, memory, attention and cortisol. Psychoneuroendocrinology 2000;25:535–49.
5. March MG, Crowley JJ. An evaluation of anesthesiologists' present checkout methods and the validity of the FDA checklist. Anesthesiology 1991;75:724–9.
6. Lampotang S, Moon S, Lizdas DE, Feldman JM, Zhang R. Anesthesia machine pre-use check survey- preliminary results. Atlanta: American Society of Anesthesiologists, 2005.
7. Hart EM, Owen H. Errors and omissions in anesthesia: a pilot study using a pilot's checklist. Anesth Analg 2005;101:246–50.
8. Kurrek MM, Devitt JH, Cohen M. Cardiac arrest in the OR: how are our ACLS skills? Can J Anaesth 1998;45:130–2.
9. Ward P, Johnson LA, Mulligan NW, Ward MC, Jones DL. Improving cardiopulmonary resuscitation skills retention: effect of two checklists designed to prompt correct performance. Resuscitation 1997;34:221–5.
10. Morris AH. Decision support and safety of clinical environments. Qual Saf Health Care 2002;11:69–75.
11. Howard SK, Gaba DM, Fish KJ, Yang G, Sarnquist FH. Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 1992;63:763–70.
12. Gaba DM, Howard SK, Fish KJ, Smith BE, Sowb YA. Simulation-based training in anesthesia crisis resource management: a decade of experience. Simul Gaming 2001;32:175–93.
13. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8–18.
14. i Gardi T, Christensen UC, Jacobsen J, Jensen PF, Ording H. How do anaesthesiologists treat malignant hyperthermia in a full-scale anaesthesia simulator? Acta Anaesthesiol Scand 2001;45:1032–5.
15. Chopra V, Gesink BJ, de Jong J, Bovill JG, Spierdijk J, Brand R. Does training on an anaesthesia simulator lead to improvement in performance? Br J Anaesth 1994;73:293–7.
16. Cohen J. Coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46.
17. Fleiss JL. Statistical methods for rates and proportions. New York: Wiley, 1981.
18. Berkenstadt H, Yusim Y, Katznelson R, Ziv A, Livingstone D, Perel A. A novel point-of-care information system reduces anaesthesiologists' errors while managing case scenarios. Eur J Anaesthesiol 2006;23:239–50.
19. Swain AD Guttman HE. Handbook of human reliability analysis with emphasis on nuclear power plant applications. Washington, DC: United States Nuclear Regulatory Commission, 1983.
20. Einstein GO, McDaniel MA, Williford CL, Pagan JL, Dismukes L. Forgetting of intentions in demanding situations is rapid. J Exp Psychology Appl 2003; 9:147–62.