From the *Department of Anesthesia, Sheba Medical Center, Tel Hashomer, Israel; †Israel Board Examination in Anesthesiology, ‡Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel; and §The Israel Center for Medical Simulation, Sheba Medical Center, Tel Hashomer, Israel.
Funding: No funding.
The authors declare no conflict of interest.
Reprints will not be available from the authors.
Address correspondence to Erez Ben-Menachem, MBCHB, MBA, FANZCA, Department of Anesthesia, Sheba Medical Center, Tel Hashomer 52621, Israel. Address e-mail to firstname.lastname@example.org.
Accepted November 24, 2010
Published ahead of print February 2, 2011
BACKGROUND: The potential for catastrophe resulting from anesthetic equipment failure and the failure of training programs to adequately prepare residents to detect and manage equipment failure prompted the Israel Board of Anesthesiologists to include simulation-based testing in the Objective Structured Clinical Evaluation component of the Israeli Board Examination in Anesthesiology.
METHODS: We used simulation-based scenarios to measure the performance of residents while (a) checking the anesthesia machine before the first morning case, (b) checking the anesthesia machine between cases, (c) managing an oxygen pipeline failure, and (d) managing an expiratory valve failure.
RESULTS: During board examination, 3 of 28 examinees failed to correctly check at least 70% of the items on the anesthesia machine checkout list before the first morning case and 3 of 30 failed to correctly check 70% of the items between cases. Although all examinees recognized inadequate oxygen cylinder pressure and a malfunctioning valve, 1 of 31 examinees failed to open the O2 cylinder, 6 of 31 did not disconnect the anesthesia machine from the central oxygen supply, 14 of 31 could not explain how to minimize the use of oxygen, 2 of 30 failed to find the faulty valve, and 15 of 30 could not give the correct differential diagnosis.
CONCLUSIONS: During simulation-based board examination most senior anesthesia residents became aware of equipment failures but many failed to correctly diagnosis and manage the failure.
Mechanical failure of the anesthesia machine is a rare event, but when it does occur it can result in major morbidity or death.1 Managing equipment faults requires an understanding of the anesthesia equipment and recall of response protocols. Simulation-based studies found errors in the way anesthesiologists perform machine checkouts,2 and in their management of oxygen supply failure.3,4 This alarming finding led us to add simulation-based testing to the Objective Structured Clinical Evaluation (OSCE) component of the Israeli Board Examination in Anesthesiology.5,6 We report our simulation-based examination results.
After IRB approval, data on examinees' performance during the simulation-based OSCE component of the Israeli Board Examination in Anesthesiology were analyzed. Because the data were used retrospectively and anonymously, and participants were enrolled as examinees, the IRB did not require us to obtain informed consent.
Anesthesia machine–related tasks were integrated into four 20-minute scenarios performed in a simulated operating room, using an Ohmeda 8000 anesthesia machine (Datex-Ohmeda, Helsinki, Finland), a Datex AS 3 anesthesia monitor (Datex-Ohmeda, Helsinki, Finland), and a METI simulator (Medical Education Technologies Incorporated, Sarasota, FL)., Participants in the examination were informed in advance about the equipment used and were exposed to the equipment during an orientation day before the examination, to overcome the variability in anesthesia equipment and exposure to simulation-based training between residency programs. During the examination, participants were instructed to perform as they would in a real clinical case and to perform a set of actions. Participants were encouraged to verbalize their actions during performance. Two trained examiners assessed each examinee independently. Performance is assessed using a checklist with yes/no scoring and by asking for explanations of relevant dilemmas that may present themselves during the scenario. Scoring is based on the checklist, and in addition a holistic score is given by each examiner for the overall performance of each candidate.
The following tasks were formulated.
1. Checking anesthesia equipment and machine for the first morning case, according to American Society of Anesthesiologists (ASA) guidelines.a Inadequate oxygen cylinder pressure or malfunctioning valves needed to be recognized by the examinees.
2. Checking the anesthesia machine between cases according to the ASA guidelines.
3. Detecting and repairing an oxygen pipeline failure during general anesthesia. The task was incorporated into a rapid sequence induction scenario. After tracheal intubation and connection of the patient to the anesthesia machine, the oxygen pipeline supply failed.
4. Detecting and repairing an expiratory valve failure during anesthesia. The task was incorporated into a rapid sequence induction scenario. After tracheal intubation and connection of the patient to the anesthesia machine, the end inspiratory CO2 levels were found to be higher than zero. In all clinical scenarios the patient remained stable to allow the examinee to concentrate on the anesthesia machine malfunction.
Data were collected retrospectively from the examination committee records, which spanned a 4-year period. Data included the incidence of correct answers for each checklist item, and the holistic scoring given by the examiners (poor/fair/good/very good).
The examination charts of 119 examinees were reviewed, each of whom performed 1 of the 4 scenarios. The total interrater correlations between the 2 raters for checklist items and global scores were 0.92 and 0.79, respectively.
Table 1 shows the results for checking the anesthesia machine before the first morning case. All but 3 of the 28 examinees performed 70% of the checklist items correctly.
Table 2 shows the results for checking the anesthesia machine between cases. All but 3 of the 30 examinees performed 70% of the checklist items correctly.
Tables 3 and 4 show the results for oxygen pipeline failure and expiratory valve malfunction.
All but 1 examinee, not recognizing the O2 supply and pressure alarms, were able to recognize immediate life-threatening equipment failure and act accordingly, as would be expected from residents presenting for final examinations. A detailed understanding of oxygen conservation techniques was lacking in half of our examinees. Similar results of immediate safe patient care but deviations from optimal management—including failure to conserve oxygen supplies and failure to test the composition of the gas supplied from the repaired pipelines—have been found elsewhere.4 This may reflect a predilection of trainees to focus on technical management of critical events without developing a deeper understanding of why such actions are undertaken. Trainees may simply be learning “what to do” and not “why we do what we do.”
A recent study by Mudumbai et al.7 into the response of third-year anesthesiology residents to a simulated pipeline cross-over of oxygen and nitrous oxide provided yet another example of the lack of in-depth understanding of anesthesia delivery systems in trainees. In this study a high proportion of participants chose to use the auxiliary oxygen flowmeter on the anesthesia machine not realizing that it was supplied from the same crossed over wall supply, and none of the participants disconnected the machine from the wall pipeline supply when using back-up oxygen tanks. All participant groups except for 1 were able to eventually provide definitive oxygen supply, but most trialled inappropriate management plans before arriving at a satisfactory option, and the definitive strategy was found only in response to ongoing machine alarms and patient desaturation. Our own findings parallel Mudumbai et al.'s results and suggest that trainees are simply reacting to crises, possibly using a recognition-primed decision approach rather than basing management on either an understanding of the functioning of anesthesia equipment or a systematic checklist approach from the outset.
Given that equipment failures are rare events and that retention of written response protocols8,9 related to these situations is poor,10 it may be a more prudent strategy to focus on teaching trainees a working knowledge of anesthesia delivery systems. Simulation-based training may be an alternative strategy. Simulation may improve knowledge retention over and above that provided by standard lecture series and written examinations, although this hypothesis is yet to be tested. Additionally, simulation-based testing may identify knowledge gaps in trainees and help training coordinators assess the effectiveness of their training programs.
A potential confounder in our data is that a large proportion of candidates presenting for examination are specialists seeking recredentialing after immigration to Israel. These candidates often have many years of experience and may have had exposure to rare clinical events. The high success rate in our examinations may not be replicated in cohorts consisting purely of trainees whose clinical experience is limited to their specialty training. This “demographic” difference may explain the different results found in a Canadian study of trainees3 in which only 40% of the participants recognized the oxygen supply failure and 50% opened the oxygen cylinder during a simulated oxygen supply failure (in comparison with 97% for both actions in our study). In another study assessing specialist anesthesiologists from New Zealand,4 all participants recognized the problem and turned on the oxygen cylinder. Because our study included only 119 examinees, generalizability to larger international programs must be undertaken with caution. Nonetheless, our data provide a basis on which other programs may choose to benchmark their own training, and assess trainees' ability to cope with anesthetic equipment malfunctions.
Name: Erez Ben-Menachem, MBCHB, MBA, FANZCA.
Contribution: Manuscript preparation.
Name: Tiberiu Ezri, MD.
Contribution: Conduct of study.
Name: Amitai Ziv, MD, MHA.
Contribution: Study design.
Name: Avner Sidi, MD.
Contribution: Study design.
Name: Haim Berkenstadt, MD.
Contribution: Study design and statistical analysis.
a Recommendations for Pre-Anesthesia Checkout Procedures (2008). Sub-Committee of ASA Committee on Equipment and Facilities. Available at: http://www.asahq.org/clinical/FINALCheckoutDesignguidelines02-08-2008.pdf. Cited Here...
1. Caplan R, Vistica M, Posner K, Cheney F. Adverse anesthetic outcomes arising from gas delivery equipment: a closed claims analysis. Anesthesiology 1997;87:741–8
2. Merry A, Weller J, Robinson B, Warman G, Davies E, Shaw J, Cheeseman J, Wilson L. A simulation design for research evaluating safety innovations in anaesthesia. Anaesthesia 2008;63:1349–57
3. Lorraway PG, Savoldelli GL, Joo HS, Chandra DB, Chow R, Naik VN. Management of simulated oxygen supply failure: is there a gap in the curriculum? Anesth Analg 2006;102:865–7
4. Weller J, Merry A, Warman G, Robinson B. Anaesthetists' management of oxygen pipeline failure: a room for improvement. Anaesthesia 2007;62:122–6
5. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based objective structured clinical examination (OSCE) into the Israeli National Board Examination in Anesthesiology. Anesth Analg 2006;102:853–8
6. Berkenstadt H, Ziv A, Gafni N, Sidi A. The validation process of incorporating simulation-based accreditation into the anesthesiology Israeli national board exams. Isr Med Assoc J 2006;8:728–33
7. Mudumbai SC, Fanning R, Howard SK, Davies MF, Gaba D. Use of medical simulation to explore equipment failures and human–machine interactions in anesthesia machine pipeline supply crossover. Anesth Analg 2010;110:1292–6
8. Groves J, Edwards N, Carr B. The use of a visual aid to check anaesthetic machines. Is performance improved? Anaesthesia 1994;49:122–5
9. Berkenstadt H, Yusim Y, Ziv A, Ezri T, Perel A. An assessment of a point-of-care information system for the anesthesia provider in simulated malignant hyperthermia crisis. Anesth Analg 2006;102:530–2
10. Goldhaber-Fiebert SN, Goldhaber-Fiebert JD, Roscow CE. Knowledge-based errors in anesthesia: a paired, controlled trial of learning and retention. Can J Anaesth 2009;56:35–45