The course of an anesthetic is largely governed by a carefully orchestrated series of medications given in anticipation of or in response to specific events during a surgical procedure. In the complex and fast-changing environment of the operating room, anesthesiologists are unusual for having complete responsibility for all steps in the medication-handling process without the safety mechanisms of computer decision support or pharmacy input that benefit providers in other settings.1–4 Mental calculations, distractions, task overload, time pressures, and nonstandard medication processes have all been shown to contribute to medication errors in the operating room.5–9 Recent data suggest that medication errors during anesthesia are more prevalent than previously believed; yet, despite mounting evidence, few widely adopted interventions have been developed to improve medication safety during anesthesia and even fewer have been shown to reduce medication errors prospectively.10–17
In an effort to create a more effective and sustainable method for reducing medication errors, the University of Washington Department of Anesthesiology and Pain Medicine and the Division of Design at the University of Washington School of Art + Art History + Design collaborated to create the Anesthesia Medication Template (AMT). Using expertise from visual, interaction, and industrial design, cognitive psychology, and experience designing commercial aircraft cockpits, the team created a three-dimensional device that defines a formal way of organizing and identifying medication syringes in the anesthesia workspace. When providers select a syringe to administer a medication, they identify the correct one by a number of factors including text, shape, size, color, and location. The AMT is a cognitive artifact that optimizes all these factors to ensure that providers select the correct syringe with minimal cognitive processing.18 This study tested the hypothesis that the AMT would reduce syringe swaps, dosing errors, and medication search times during simulated emergencies as well as a series of medication errors in the real operating room.
The AMT was developed over a 3-year period involving multiple test and revision cycles with progressively more sophisticated versions and complex materials. The template was tested by members of the research team in the operating room while new versions were being developed and refined. Figure 1 shows the final version used in this study made from 3 layers of milled type 1 polyvinyl chloride with a manufacturing cost around $200. The overall goal was to create a flexible design for the majority of anesthetics that minimizes medication errors via a number of principles: (1) standardizing medication layout in all anesthetizing locations, (2) minimizing inadvertent syringe swaps, (3) exploiting blank spaces as cues for missing medications, and (4) diminishing search time and overall cognitive load on clinicians. The template was designed to be useful and intuitive for anesthesia providers, thus inspiring universal compliance with minimal training. The layout primarily consists of a series of three-dimensional cells devoted to specific medications or medication classes. The bottom half is flexible space to be used for medication preparation, medications not included in one of the specific cells, or other medication-related paraphernalia.
Medication errors in the operating room are notoriously difficult to capture, which is one reason literature to date is limited. To address this challenge, the study was designed in 2 phases: (1) a simulation phase, to directly observe medication errors in a controlled environment, and (2) a clinical phase, where anesthesia providers in real clinical practice self-reported medication errors prospectively during a 2-year period. This study was conducted at Seattle Children’s Hospital, a freestanding pediatric academic institution affiliated with the University of Washington between August 2013 and August 2015. The protocol was approved by the Seattle Children’s Hospital Institutional Review Board (Seattle, WA, IRB#14654).
Phase 1 of the study was designed to capture errors in a simulated but controlled, reproducible environment. It consisted of a randomized study conducted in an operating room environment with a Laerdal SimBaby mannequin (Laerdal Medical Corp, Wappingers Falls, NY). Volunteers from the Seattle Children’s anesthesia department were asked to participate in 2 emergency clinical scenarios of similar complexity and duration—one involving anaphylaxis and the other laryngospasm—designed for the study. Each scenario required 4 medication administrations of 5 different medications (propofol, cefazolin, succinylcholine, atropine, and epinephrine). For their first session, participants were randomly assigned to which scenario they experienced and whether they used the AMT or a control setup. Each participant then experienced a second session with the opposite set of conditions as shown in Figure 2. Unlike a traditional crossover study, both the medication setup and the scenario changed between sessions to minimize learning bias.
The layout of syringes on the AMT was determined by the specific cells in the AMT itself (Figure 1). The layout of the control setup was a “typical” layout based on 2 dozen photos taken of real carts before the implementation of the AMT. Both setups contained the same number and types of syringes and were arranged in the same way each time by a member of the research team before the scenario began. The only medication difference was the epinephrine preparation: the control setup used a LifeShield ABBOJECT 100 μg/mL syringe (Hospira, Inc, Lake Forest, IL) and the AMT used a 10 μg/mL prefilled syringe (PharMEDium Services, LLC, Lake Forest, IL). Succinylcholine and atropine emergency medications were supplied in standard prefilled PharMEDium syringes, while propofol and cefazolin were prefilled and prelabeled manually, both of which are consistent with clinical practice in our department.
One member of the research team played the part of an anesthesia provider in the middle of a routine case. The study participants entered the room to give a break, but during the handoff an emergency would unfold. The participants were instructed beforehand not to focus on diagnosing or otherwise managing the patient, but only to act as directed. Their singular task was to administer 4 medications in specific doses—relayed in milligrams per kilogram—as instructed by the researcher. The average scenario lasted 5 minutes, and the simulated patients always responded in the same manner to the medications given.
Another member of the research team observed the simulation and recorded: (1) the “search time” to locate the medication, and (2) the “administration time” from locating to delivering the medication. In addition, the observer noted omission or swap errors (whether the correct medication was given) and incorrect dosing errors. Dose rounding was allowed to be clinically realistic because the weights of the simulated patients were intentionally odd numbers (9 kg and 13 kg, respectively), so a dose within 30% above or below the precisely calculated dose was considered “accurate.”
Before the simulation event, participants were asked to fill out a brief demographic questionnaire. After completing both scenarios, participants were asked to fill out the System Usability Scale (SUS)—a validated 10-item Likert scale designed to evaluate the usability of products and services, including hardware, software, mobile devices, websites, and applications.19 The SUS asks questions about ease of use, simplicity, consistency, and training required to master a device, and the specific questions are in Table 1. The scale has a scoring system that converts responses into a single score on a 100-point scale representing the overall usability of the system being studied. Based on validation studies, an average SUS score is 68, and a score above 80.3 is considered excellent.
Phase 2 was an observational, “pre-post” study that included all anesthesia providers at Seattle Children’s Hospital 12 months before and 12 months after implementation of the AMT. Providers were oriented to the medication template through staff meeting presentations explaining the design theory, development process, and proper use. The template was implemented in all anesthetizing locations simultaneously with a combination of strong leadership support and real-time coaching in the operating room. The majority of errors were collected using an anonymous, online, self-reporting tool built specifically for the study, while some errors were captured using preexisting, hospital-wide quality improvement (QI) reporting mechanisms. The new online system was created in-house to collect more specific, structured medication error-related data than existing QI mechanisms and to satisfy the IRB’s request for anonymity to prevent punitive repercussions for providers reporting errors. Although the tool did not collect information specific to who performed or reported the error, it did capture the week the error occurred, the type of case, the type of error, and a narrative about contributing factors. Staff meeting announcements were made at the beginning and halfway point of the study, and biweekly reminder emails were sent to all staff to encourage error reporting during the entire duration of the study. Although the errors collected were from a real clinical setting, participants could not be randomly assigned and they served as their own controls. Around 200 anesthesia providers participated in this phase of the study. There was standard rotation of trainees and minimal nurse anesthetist or attending staff changes during the course of the study.
For phase 1, errors per 100 simulations and Wilson 95% confidence intervals were calculated and plotted for each scenario and medication, with and without the template.20 General estimating equations were used to compare the odds of medication errors with and without template, controlling for scenario, session, training level, and years at training level.
Phase 2 self-reported error events were categorized into 6 types: (1) prescribing and communication, (2) preparation (including dilution and labeling), (3) syringe swap, (4) miscalculation, (5) timing, and (6) infusion pump programming and delivery. Reported errors unrelated to anesthesia medication handling—for example, an issue with chlorhexidine skin preparation—were excluded from the final analysis, as were duplicate error reports. In addition to type, errors were further categorized as “near miss” (not reaching the patient), “precursor events” (reaching the patient but causing no harm or minimal, temporary harm), or “serious safety events” (causing at least moderate, temporary harm) as defined by the Agency for Healthcare Research and Quality Common Formats Harm Scale version 1.2.21
Control charts showing the monthly medication error rates per 1000 anesthetics during the study timeframe (from August 2013 to August 2015) were created based on whether or not the error reached the patient and on error type. A significant change was defined by a run of 6 consecutive data points below the weighted average (center line).22,23 Generalized linear models were used to estimate mean monthly error rates and 95% Poisson confidence intervals for pre- and postimplementation, as well as to compare pre versus post without adjustments. Stata 12 was used for analyses (StataCorp. 2011. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP).
Phase 1 of study had greater than 80% power at a .05 significance level to detect a 30% reduction in medical errors, assuming a baseline error rate of 10% (PASS 14 Power Analysis and Sample Size Software . NCSS, LLC. Kaysville, UT, ncss.com/software/pass).
Forty-one anesthesia providers participated in 82 simulated scenarios for a total of 328 total medication administrations. The participants’ training levels and prior exposure to medication templates is shown in Table 2. Primary outcome measures for the simulation phase of the study included time (to locate and to administer) and medication errors. All observed errors were dosing calculation errors, because the correct medication was given every time. The odds of dosing errors using the AMT was 0.21 times the odds of dosing errors not using the AMT (95% CI, 0.07–0.66), controlling for scenario, session, training level, and years at training level as shown in Table 3. Overall dosing errors decreased from 10.4 (95% CI, 6.6–16.0) to 2.4 (95% CI, 1.0–6.1) errors per 100 medication administrations (values are unadjusted). Figure 3 shows the effect of the medication template on each individual medication with atropine associated with the most errors followed by epinephrine.
The times to locate and administer each of the medications were recorded, but none of the results showed any statistically or clinically significant differences with and without the AMT. Each medication was located and administered within a few seconds. The only outlier was the epinephrine packaged in the LifeShield ABBOJECT box, but the effect was likely more related to the packing and concentration of the medication than the template.
A qualitative overview of medication error reporting before, during, and after the AMT study is shown in Figure 4. The figure shows the raw total of medication errors reported in 6-month epochs to give a sense of the changing quality of reporting over the years. The first 3 years (months from −48 to −13) shows the baseline error reporting before any medication safety efforts began, and only 9 errors were reported during that period. An important medication safety risk analysis was performed during month −12, which kicked off a series of improvement efforts of which the AMT is one. Reporting increased using existing mechanisms so that a total of 21 errors were reported during the year before the AMT study began. The new online reporting system went live when the AMT study started at month 1, and near-miss errors were reported for the first time. The AMT itself was implemented at month 13. After the completion of the AMT study (months 25–30), error reporting continued around the same rate with an increased proportion of near-miss errors—suggesting that any reduction in errors reaching the patient was not the result of reporting fatigue.
A total of 67 medication error reports were submitted during the 2-year study period, while 45,538 anesthetics were performed. After excluding 7 errors unrelated to anesthesia medication handling and 7 duplicate error reports, a total of 53 unique errors remained for further analysis. Table 4 shows the decrease in self-reported medication errors in the operating room after implementation of the AMT. The mean monthly error rate for all medication errors that reached patients decreased from 1.24 (95% CI, 0.85–1.79) to 0.65 (95% CI, 0.39–1.09) errors per 1000 anesthetics. The control chart in Figure 5 shows the temporal change in errors before and after the AMT and the relationship of the monthly error rates to the mean. There are 6 postimplementation data points below the mean versus 4 pre-AMT but not 6 consecutive points. Although the prevailing trend appeared to be a decrease in medication errors postimplementation, there was an unexpected rise in the error rate during months 20 to 23. Figure 5 also shows near miss errors, which were reported throughout the study period with no significant change from 0.27 (95% CI, 0.12–0.59) errors per 1000 anesthetics before implementation to 0.17 (95% CI, 0.07–0.47) errors afterward.
The subset of error types more directly related to the AMT—swap, preparation, miscalculation, and timing errors—showed a more pronounced decrease from 0.97 (95% CI, 0.64–1.48) to 0.35 (95% CI, 0.17–0.70) errors per 1000 anesthetics. Figure 6 shows the monthly error rates in a run chart with 10 of 12 post-AMT data points falling below the mean and 6 consecutive points during months 19 to 24. Prescribing/communication and infusion pump programming errors, which are not directly related to the AMT, did not show a significant change from 0.53 (95% CI, 0.30–0.93) to 0.48 (95% CI 0.27–0.87) errors per 1000 anesthetics. Of note, an increase in infusion pump-related errors during months 20 to 23 helps explain why the overall error rate rose above the mean in Figure 5.
There were a total of 10 reports of errors that resulted in at least moderate, temporary harm during the study with an equal number before and after implementation. There were no events that caused permanent harm, death, or disability. Consequently, the rate of errors resulting in harm did not change from 0.22 (95% CI, 0.09–0.53) to 0.22 (95% CI, 0.09–0.52) errors per 1000 anesthetics.
The System Usability Scale score for the AMT was 90.4 on a 100-point scale. An average usability score is 68 (50th percentile); a score above 80.3 is considered “excellent”; and a score of 90.4 puts the AMT in the 99th percentile for usability.24 Although the SUS is a fairly quick and generic scale for usability designed for a wide range of products, these results suggest that the AMT is exceptionally easy to learn and use. Such high user satisfaction scores help promote acceptance and compliance in clinical practice.
Primary Findings—Phase 1
During a simulated anesthetic, the odds of dosing errors using the AMT was 0.21 times the odds of dosing errors not using the AMT. Even if the conditions of phase 1 were not as realistic as the operating room, the simplicity of the simulated task, the allowance for significant dose rounding, and the lack of explicit time pressure all make the large impact on medication errors all the more significant. The simulated scenarios also highlight the value of a standardized medication setup during vulnerable transitions—emergencies or handoffs—where new providers who are called upon to administer medications know exactly where to locate emergency medications.
Phase 1 also revealed insights into other factors that contribute to medication errors. Although the standard prefilled PharMEDium 0.4 mg/mL atropine syringes were used in all simulations, the significantly higher number of errors with atropine in the non-AMT group suggests that complex calculations may be more challenging without a formally organized setup. The added complexity of assembling and diluting the 100 μg/mL epinephrine ABBOJECT likely also contributed to difficulties with mental math. Both of these findings suggest that simplifying calculations by using prefilled syringes with standardized sizes, concentrations, and labels may reduce error rates, particularly in pediatric anesthesia practice where weight-based calculations are commonplace. Other cognitive aids—like a preprinted sheet with patient-specific weight-based doses for emergency medications—could further reduce the cognitive burden of dose calculation.
One of the most interesting questions raised by this study is how reorganizing medication syringes may help with dose calculation. Syringe swaps could certainly be reduced by standardizing the layout of syringes, but there is not as clear of a connection between the AMT and mental math. One possible explanation is that the AMT reduces cognitive load. According to cognitive load theory, humans have a limited cognitive capacity, and a disorganized anesthesia workspace may lead to avoidable mistakes by increasing the processing demand on the provider.25,26 Even if a workspace is organized, if different providers have different systems, confusion can occur in environments where anesthesia is administered teams. In our study simulations, the act of searching for a medication adds to the high processing demand that includes identifying the correct medication, calculating the correct mass of drug, and converting to a volume to administer.27–30 We theorize that the AMT group may have had a lower processing requirement to locate, calculate, and deliver medications.
Primary Findings—Phase 2
After introduction of the AMT, the mean monthly error rate for all medication errors in the operating room that reached patients at Seattle Children’s decreased from 1.24 to 0.65 errors per 1000 anesthetics. Errors types related to the AMT (syringe swap, preparation, miscalculation, and timing errors) decreased more significantly both in the overall analysis and in the control charts. Errors unrelated to the AMT (prescribing, infusion errors) did not change at all. Although multiple confounding factors existed during the 2-year study, the temporal relationship and the correlation with specific error types both suggest that the AMT contributed to the observed decrease in medication errors.
There was no change in errors resulting in patient harm, which arguably is the most important clinical outcome related to medication errors. However, with only 10 serious safety events recorded during the entire 2-year study period, it is difficult to draw conclusions about the AMT’s effect on significant patient harm. This study was not sufficiently powered to reveal a meaningful impact on harm events as they are a subset of the already infrequent overall error events. Of note, during the 6 months after the study ended, only one harm event was reported.
Although the System Usability Scale is likely unfamiliar to many in the medical field, the importance of usability cannot be overstated. Too often medical devices are either designed without sufficient input from end users or without a basic understanding of design and usability principles. Creating an intuitive system that conforms to existing workflows and is difficult to use improperly is a considerable accomplishment, which is why, for example, the three-dimensional structure of the AMT is so important. Although a slightly more complex version of the AMT has been developed for cardiac and transplant cases, much of the value of the device lies in its minimalism because, much like other cognitive aids—like the World Health Organization’s Surgical Safety Checklist—simplicity is a key component of user acceptance and overall effectiveness.31
Although the importance of this topic is widely recognized, medication errors have proven notoriously difficult to study. They are infrequent events that necessitate long periods of data collection to demonstrate a meaningful change in environments where controlling for confounders is challenging. The longer data are collected, the more potential there is for confounding variables to be introduced.
The simulations in phase 1 were an attempt to control for many the variables found in the real operating room. Unfortunately, phase 1 was instead subject to all the limitations of a simulated clinical environment. Mannequins are not 100% realistic; providers act differently in a scrutinized, simulated environment; and the scenarios were dense and scripted. By distilling the role of the participant down to only 4 medication administrations under the direction of a research team member, we hoped to focus the participants’ attention to medication administration rather than other aspects of patient care. The magnitude of the impact may be exaggerated in the simulation; however, many clinical scenarios are certainly more stressful and complex than these simulations.
During study design, the research team debated whether the control setup would be a consistent “average” setup or whether each participant would be allowed to create their own. In the interest of standardization and reproducibility, we chose a consistent setup of syringes for the control scenarios; however, by not allowing participants to use a familiar setup, this may have decreased their performance during the controls and thereby magnified the effect of the AMT. The other primary confounder during the simulations was the use of the Lifeshield ABBOJECT syringe in the control setup, which differed 10-fold in concentration from the AMT group. The added complexity of assembling and diluting the 100 μg/mL syringe certainly added to the time and cognitive load of delivering epinephrine. This syringe was chosen because it is the standard preparation found in anesthesia and code carts throughout the institution owing to its long-term storage capability and ability to be used in adult-sized patients.
During phase 2, the AMT was not the only medication safety initiative happening at Seattle Children’s, and with hundreds of providers involved, it is impossible to point to a single intervention as the source of all positive change. After a year of increased awareness following the launch of a broad medication safety initiative, the study began with the improved online error-reporting system and weekly reminder emails, all of which appear to have improved error reporting as shown in Figure 4. Reorganized medication trays containing medication vials were implemented during month 1 when the study began. Medication practice guidelines outlining standard syringe sizes and concentrations went live during month 3, and a 2-provider infusion checklist was implemented to help decrease infusion pump-related errors during month 9. There were no confounding interventions after the AMT was implemented on month 13. In addition, without blinding or randomization, users of the device may naturally improve over time either by becoming more facile with the device or for other unknown reasons.
Given the complexity of collecting data from hundreds of anesthesia providers covering dozens of anesthetizing locations over multiple years, there is no feasible way to adjust for all potential confounding variables. The infrequency of medication errors required long periods of time to collect data, which also made implementing the AMT partially (eg, in half of the operating rooms) unrealistic because there would be no way to consistently prevent exposure of certain providers to the tool given the staffing model at the hospital. As a result, a number of known and unknown confounding variables that change over time in addition to the ones specifically mentioned inevitably affect the results of phase 2, which is not unusual with this type of QI work.
The primary limitation of this study is that all errors were self-reported. Inevitably some errors will go unreported, so the absolute incidence of errors cannot be known. Regular reminders via email communications and meeting announcements were included in the study design to encourage reporting. Reporting fatigue can potentially be an issue during a 2-year study period, thus a decrease in reported errors could be a drop in reporting rather than a real improvement. However, an overall steady rate of error reports during the 6 months after the study along with an increase in near-miss errors suggests that overall reporting rates remained consistent throughout the study period.
For the AMT to be truly impactful, it will need to be implemented at other institutions, which raises both cultural and logistical concerns. Regulators increasingly concerned about infection control and medication security must be taken into account when using a template to organize medications. In addition to an intuitive design to support user acceptance, the implementation at Seattle Children’s benefitted from leadership support, a core of enthusiastic designers, and clinicians who could provide coaching. Despite standardization being a key principle, the specific configuration can be customized for other institutions, and we are currently exploring the hurdles involved in employing the device elsewhere.
Medication safety in the operating room is an ongoing and complex problem that will ultimately benefit from a suite of interventions such as color coding, prefilled syringes, bar coding, and other cognitive aids. Although the anesthesia machine has benefitted from a variety of safety engineering innovations over the past 30 years, medication handling is comparatively primitive to date. The AMT has the potential to be another important safety innovation owing to its flexibility, low cost, and ease of use. Some high-tech solutions are complex, expensive, and difficult to incorporate into existing workflows, while the AMT’s carefully crafted and deceptively simple design is based on interaction and visual design principles that provide cognitive support for existing interaction patterns.32 The AMT demonstrates the potential benefits to patient safety of applying interaction and visual design to how providers identify and interact with medications during anesthesia.
Name: Eliot B. Grigg, MD.
Contribution: Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, funding acquisition.
Name: Lizabeth D. Martin, MD.
Contribution: Conceptualization, methodology, investigation, data curation, writing—original draft preparation.
Name: Faith J. Ross, MD, MS.
Contribution: Conceptualization, methodology, investigation, writing—original draft preparation.
Name: Axel Roesler, PhD.
Contribution: Conceptualization, methodology, resources, writing—original draft preparation.
Name: Sally E. Rampersad, MB, FRCA.
Contribution: Conceptualization, methodology, writing—review and editing.
Name: Charles Haberkern, MD, MPH.
Contribution: Conceptualization, methodology, writing—review and editing, supervision.
Name: Daniel K.W. Low, BMedSci, BM, BS, MRCPCH, FRCA.
Contribution: Conceptualization, methodology, writing—review and editing.
Name: Kristen Carlin, MPH.
Contribution: Formal analysis, visualization, writing—original draft preparation.
Name: Lynn D. Martin, MD, MBA.
Contribution: Conceptualization, methodology, writing—review and editing, supervision.
This manuscript was handled by: Nancy Borkowski, DBA, CPA, FACHE, FHFMA.
1. Cooper JB, Newbower RS, Kitz RJ. An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection. Anesthesiology. 1984;60:34–42.
2. Fasting S, Gisvold SE. Adverse drug errors in anesthesia, and the impact of coloured syringe labels. Can J Anesth. 2000;47:1060–1067.
3. Merry AF, Webster CS, Mathew DJ. A new, safety-oriented, integrated drug administration and automated anesthesia record system. Anesth Analg. 2001;93:385–390.
4. Merry AF, Webster CS, Connell H. A new infusion syringe label system designed to reduce task complexity during drug preparation. Anaesthesia. 2007;62:486–491.
5. Stabile M, Webster CS, Merry AF. Medication administration in anesthesia: time for a paradigm shift. APSF Newsletter. 2007;22:344–347.
6. Kothari D, Gupta S, Sharma C, Kothari S. Medication error in anaesthesia and critical care: a cause for concern. Indian J Anaesth. 2010;54:187–192.
7. Mahajah RP. Medication errors: can we prevent them? Br J Anaesth. 2011;107:3–5.
8. Cooper L, Nossaman B. Medication errors in anesthesia: a review. Int Anesthesiol Clin. 2013;51:1–12.
9. Orser BA, Hyland S, U D, Sheppard I, Wilson CR. Review article: improving drug safety for patients undergoing anesthesia and surgery. Can J Anesth. 2013;60:127–135.
10. Orser BA, U D, Cohen MR. Perioperative medication errors: building safer systems. Anesthesiology. 2016;124:1–3.
11. Webster CS, Merry AF, Larsson L, McGrath KA, Weller J. The frequency and nature of drug administration error during anaesthesia. Anaesth Intensive Care. 2001;29:494–500.
12. Orser BA, Chen RJ, Yee DA. Medication errors in anesthetic practice: a survey of 687 practitioners. Can J Anesth. 2001;48:139–146.
13. Llewellyn RL, Gordon PC, Wheatcroft D, et al. Drug administration errors: a prospective survey from three South African teaching hospitals. Anaesth Intensive Care. 2009;37:93–98.
14. Shridhar Iyer U, Fah KK, Chong CK, Macachor J, Chia N. Survey of medication errors among anaesthetists in Singapore. Anaesth Intensive Care. 2011;39:1151–1152.
15. Merry AF, Webster CS, Hannam J, et al. Multimodal system designed to reduce errors in recording and administration of drugs in anaesthesia: prospective randomised clinical evaluation. BMJ. 2011;343:d5543.
16. Jelacic S, Bowdle A, Nair BG, Kusulos D, Bower L, Togashi K. A system for anesthesia drug administration using barcode technology: the codonics safe label system and smart anesthesia manager. Anesth Analg. 2015;121:410–421.
17. Nanji KC, Patel A, Shaikh S, Seger DL, Bates DW. Evaluation of perioperative medication errors and adverse drug events. Anesthesiology. 2016;124:25–34.
18. Norman D. Carroll JM. Cognitive artifacts. In Designing Interaction: Psychology at the Human-Computer Interface (Cambridge Series on Human-Computer Interaction). 1991:New York: Cambridge University Press; 17–38.
19. Brooke J. Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, eds. SUS: a “quick and dirty” usability scale. In Usability Evaluation in Industry. 1996:London: Taylor and Francis; 189–194.
20. Brown LD, Cai TT, Dasgupta A. Interval estimation for a binomial proportion. Stat Sci. 2001;2:101–133.
21. Williams T, Szekendi M, Pavkovic S, Clevenger W, Cerese J. The reliability of AHRQ common format harm scales in rating patient safety events. J Patient Saf. 2015;11:52–59.
22. Perla RJ, Provost LP, Murray SK. The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf. 2011;20:46–51.
23. Provost LP, Murray SK. The Health Care Data Guide. 2011.San Francisco: Jossey-Bass.
24. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum-Comput Int. 2008;24:574–594.
25. Sanches I, Teixeira F, dos Santos JM, Ferreira AJ. Effects of acute sleep deprivation resulting from night shift work on young doctors. Acta Med Port. 2015;28:457–462.
26. Ross V, Jongen EM, Wang W, et al. Investigating the influence of working memory capacity when driving behavior is combined with cognitive load: an LCT study of young novice drivers. Accid Anal Prev. 2014;62:377–387.
27. Naismith LM, Cheung JJ, Ringsted C, Cavalcanti RB. Limitations of subjective cognitive load measures in simulation-based procedural training. Med Educ. 2015;49:805–814.
28. Byrne AJ, Oliver M, Bodger O, et al. Novel method of measuring the mental workload of anaesthetists during clinical practice. Br J Anaesth. 2010;105:767–771.
29. Byrne A. Measurement of mental workload in clinical medicine: a review study. Anesth Pain Med. 2011;1:90–94.
30. Pfeffer S, Maier T, Stricker E, Trick M. Cognitive ergonomics and informatory load in anesthesia. Biomed Tech. 2012;57:947–950.
31. Weiser TG, Haynes AB, Lashoher A, et al. Perspectives in quality: designing the WHO Surgical Safety Checklist. Int J Qual Health Care. 2010;22:365–370.
32. Lovell S, Kemp K. Dieter Rams: As Little Design as Possible. 2011.London: Phaidon Press.