The subset of error types more directly related to the AMT—swap, preparation, miscalculation, and timing errors—showed a more pronounced decrease from 0.97 (95% CI, 0.64–1.48) to 0.35 (95% CI, 0.17–0.70) errors per 1000 anesthetics. Figure 6 shows the monthly error rates in a run chart with 10 of 12 post-AMT data points falling below the mean and 6 consecutive points during months 19 to 24. Prescribing/communication and infusion pump programming errors, which are not directly related to the AMT, did not show a significant change from 0.53 (95% CI, 0.30–0.93) to 0.48 (95% CI 0.27–0.87) errors per 1000 anesthetics. Of note, an increase in infusion pump-related errors during months 20 to 23 helps explain why the overall error rate rose above the mean in Figure 5.
There were a total of 10 reports of errors that resulted in at least moderate, temporary harm during the study with an equal number before and after implementation. There were no events that caused permanent harm, death, or disability. Consequently, the rate of errors resulting in harm did not change from 0.22 (95% CI, 0.09–0.53) to 0.22 (95% CI, 0.09–0.52) errors per 1000 anesthetics.
The System Usability Scale score for the AMT was 90.4 on a 100-point scale. An average usability score is 68 (50th percentile); a score above 80.3 is considered “excellent”; and a score of 90.4 puts the AMT in the 99th percentile for usability.24 Although the SUS is a fairly quick and generic scale for usability designed for a wide range of products, these results suggest that the AMT is exceptionally easy to learn and use. Such high user satisfaction scores help promote acceptance and compliance in clinical practice.
Primary Findings—Phase 1
During a simulated anesthetic, the odds of dosing errors using the AMT was 0.21 times the odds of dosing errors not using the AMT. Even if the conditions of phase 1 were not as realistic as the operating room, the simplicity of the simulated task, the allowance for significant dose rounding, and the lack of explicit time pressure all make the large impact on medication errors all the more significant. The simulated scenarios also highlight the value of a standardized medication setup during vulnerable transitions—emergencies or handoffs—where new providers who are called upon to administer medications know exactly where to locate emergency medications.
Phase 1 also revealed insights into other factors that contribute to medication errors. Although the standard prefilled PharMEDium 0.4 mg/mL atropine syringes were used in all simulations, the significantly higher number of errors with atropine in the non-AMT group suggests that complex calculations may be more challenging without a formally organized setup. The added complexity of assembling and diluting the 100 μg/mL epinephrine ABBOJECT likely also contributed to difficulties with mental math. Both of these findings suggest that simplifying calculations by using prefilled syringes with standardized sizes, concentrations, and labels may reduce error rates, particularly in pediatric anesthesia practice where weight-based calculations are commonplace. Other cognitive aids—like a preprinted sheet with patient-specific weight-based doses for emergency medications—could further reduce the cognitive burden of dose calculation.
One of the most interesting questions raised by this study is how reorganizing medication syringes may help with dose calculation. Syringe swaps could certainly be reduced by standardizing the layout of syringes, but there is not as clear of a connection between the AMT and mental math. One possible explanation is that the AMT reduces cognitive load. According to cognitive load theory, humans have a limited cognitive capacity, and a disorganized anesthesia workspace may lead to avoidable mistakes by increasing the processing demand on the provider.25,26 Even if a workspace is organized, if different providers have different systems, confusion can occur in environments where anesthesia is administered teams. In our study simulations, the act of searching for a medication adds to the high processing demand that includes identifying the correct medication, calculating the correct mass of drug, and converting to a volume to administer.27–30 We theorize that the AMT group may have had a lower processing requirement to locate, calculate, and deliver medications.
Primary Findings—Phase 2
After introduction of the AMT, the mean monthly error rate for all medication errors in the operating room that reached patients at Seattle Children’s decreased from 1.24 to 0.65 errors per 1000 anesthetics. Errors types related to the AMT (syringe swap, preparation, miscalculation, and timing errors) decreased more significantly both in the overall analysis and in the control charts. Errors unrelated to the AMT (prescribing, infusion errors) did not change at all. Although multiple confounding factors existed during the 2-year study, the temporal relationship and the correlation with specific error types both suggest that the AMT contributed to the observed decrease in medication errors.
There was no change in errors resulting in patient harm, which arguably is the most important clinical outcome related to medication errors. However, with only 10 serious safety events recorded during the entire 2-year study period, it is difficult to draw conclusions about the AMT’s effect on significant patient harm. This study was not sufficiently powered to reveal a meaningful impact on harm events as they are a subset of the already infrequent overall error events. Of note, during the 6 months after the study ended, only one harm event was reported.
Although the System Usability Scale is likely unfamiliar to many in the medical field, the importance of usability cannot be overstated. Too often medical devices are either designed without sufficient input from end users or without a basic understanding of design and usability principles. Creating an intuitive system that conforms to existing workflows and is difficult to use improperly is a considerable accomplishment, which is why, for example, the three-dimensional structure of the AMT is so important. Although a slightly more complex version of the AMT has been developed for cardiac and transplant cases, much of the value of the device lies in its minimalism because, much like other cognitive aids—like the World Health Organization’s Surgical Safety Checklist—simplicity is a key component of user acceptance and overall effectiveness.31
Although the importance of this topic is widely recognized, medication errors have proven notoriously difficult to study. They are infrequent events that necessitate long periods of data collection to demonstrate a meaningful change in environments where controlling for confounders is challenging. The longer data are collected, the more potential there is for confounding variables to be introduced.
The simulations in phase 1 were an attempt to control for many the variables found in the real operating room. Unfortunately, phase 1 was instead subject to all the limitations of a simulated clinical environment. Mannequins are not 100% realistic; providers act differently in a scrutinized, simulated environment; and the scenarios were dense and scripted. By distilling the role of the participant down to only 4 medication administrations under the direction of a research team member, we hoped to focus the participants’ attention to medication administration rather than other aspects of patient care. The magnitude of the impact may be exaggerated in the simulation; however, many clinical scenarios are certainly more stressful and complex than these simulations.
During study design, the research team debated whether the control setup would be a consistent “average” setup or whether each participant would be allowed to create their own. In the interest of standardization and reproducibility, we chose a consistent setup of syringes for the control scenarios; however, by not allowing participants to use a familiar setup, this may have decreased their performance during the controls and thereby magnified the effect of the AMT. The other primary confounder during the simulations was the use of the Lifeshield ABBOJECT syringe in the control setup, which differed 10-fold in concentration from the AMT group. The added complexity of assembling and diluting the 100 μg/mL syringe certainly added to the time and cognitive load of delivering epinephrine. This syringe was chosen because it is the standard preparation found in anesthesia and code carts throughout the institution owing to its long-term storage capability and ability to be used in adult-sized patients.
During phase 2, the AMT was not the only medication safety initiative happening at Seattle Children’s, and with hundreds of providers involved, it is impossible to point to a single intervention as the source of all positive change. After a year of increased awareness following the launch of a broad medication safety initiative, the study began with the improved online error-reporting system and weekly reminder emails, all of which appear to have improved error reporting as shown in Figure 4. Reorganized medication trays containing medication vials were implemented during month 1 when the study began. Medication practice guidelines outlining standard syringe sizes and concentrations went live during month 3, and a 2-provider infusion checklist was implemented to help decrease infusion pump-related errors during month 9. There were no confounding interventions after the AMT was implemented on month 13. In addition, without blinding or randomization, users of the device may naturally improve over time either by becoming more facile with the device or for other unknown reasons.
Given the complexity of collecting data from hundreds of anesthesia providers covering dozens of anesthetizing locations over multiple years, there is no feasible way to adjust for all potential confounding variables. The infrequency of medication errors required long periods of time to collect data, which also made implementing the AMT partially (eg, in half of the operating rooms) unrealistic because there would be no way to consistently prevent exposure of certain providers to the tool given the staffing model at the hospital. As a result, a number of known and unknown confounding variables that change over time in addition to the ones specifically mentioned inevitably affect the results of phase 2, which is not unusual with this type of QI work.
The primary limitation of this study is that all errors were self-reported. Inevitably some errors will go unreported, so the absolute incidence of errors cannot be known. Regular reminders via email communications and meeting announcements were included in the study design to encourage reporting. Reporting fatigue can potentially be an issue during a 2-year study period, thus a decrease in reported errors could be a drop in reporting rather than a real improvement. However, an overall steady rate of error reports during the 6 months after the study along with an increase in near-miss errors suggests that overall reporting rates remained consistent throughout the study period.
For the AMT to be truly impactful, it will need to be implemented at other institutions, which raises both cultural and logistical concerns. Regulators increasingly concerned about infection control and medication security must be taken into account when using a template to organize medications. In addition to an intuitive design to support user acceptance, the implementation at Seattle Children’s benefitted from leadership support, a core of enthusiastic designers, and clinicians who could provide coaching. Despite standardization being a key principle, the specific configuration can be customized for other institutions, and we are currently exploring the hurdles involved in employing the device elsewhere.
Medication safety in the operating room is an ongoing and complex problem that will ultimately benefit from a suite of interventions such as color coding, prefilled syringes, bar coding, and other cognitive aids. Although the anesthesia machine has benefitted from a variety of safety engineering innovations over the past 30 years, medication handling is comparatively primitive to date. The AMT has the potential to be another important safety innovation owing to its flexibility, low cost, and ease of use. Some high-tech solutions are complex, expensive, and difficult to incorporate into existing workflows, while the AMT’s carefully crafted and deceptively simple design is based on interaction and visual design principles that provide cognitive support for existing interaction patterns.32 The AMT demonstrates the potential benefits to patient safety of applying interaction and visual design to how providers identify and interact with medications during anesthesia.
Name: Eliot B. Grigg, MD.
Contribution: Conceptualization, methodology, formal analysis, investigation, data curation, visualization, writing—original draft preparation, funding acquisition.
Name: Lizabeth D. Martin, MD.
Contribution: Conceptualization, methodology, investigation, data curation, writing—original draft preparation.
Name: Faith J. Ross, MD, MS.
Contribution: Conceptualization, methodology, investigation, writing—original draft preparation.
Name: Axel Roesler, PhD.
Contribution: Conceptualization, methodology, resources, writing—original draft preparation.
Name: Sally E. Rampersad, MB, FRCA.
Contribution: Conceptualization, methodology, writing—review and editing.
Name: Charles Haberkern, MD, MPH.
Contribution: Conceptualization, methodology, writing—review and editing, supervision.
Name: Daniel K.W. Low, BMedSci, BM, BS, MRCPCH, FRCA.
Contribution: Conceptualization, methodology, writing—review and editing.
Name: Kristen Carlin, MPH.
Contribution: Formal analysis, visualization, writing—original draft preparation.
Name: Lynn D. Martin, MD, MBA.
Contribution: Conceptualization, methodology, writing—review and editing, supervision.
This manuscript was handled by: Nancy Borkowski, DBA, CPA, FACHE, FHFMA.
1. Cooper JB, Newbower RS, Kitz RJ. An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection. Anesthesiology. 1984;60:34–42.
2. Fasting S, Gisvold SE. Adverse drug errors in anesthesia, and the impact of coloured syringe labels. Can J Anesth. 2000;47:1060–1067.
3. Merry AF, Webster CS, Mathew DJ. A new, safety-oriented, integrated drug administration and automated anesthesia record system. Anesth Analg. 2001;93:385–390.
4. Merry AF, Webster CS, Connell H. A new infusion syringe label system designed to reduce task complexity during drug preparation. Anaesthesia. 2007;62:486–491.
5. Stabile M, Webster CS, Merry AF. Medication administration in anesthesia: time for a paradigm shift. APSF Newsletter. 2007;22:344–347.
6. Kothari D, Gupta S, Sharma C, Kothari S. Medication error in anaesthesia and critical care: a cause for concern. Indian J Anaesth. 2010;54:187–192.
7. Mahajah RP. Medication errors: can we prevent them? Br J Anaesth. 2011;107:3–5.
8. Cooper L, Nossaman B. Medication errors in anesthesia: a review. Int Anesthesiol Clin. 2013;51:1–12.
9. Orser BA, Hyland S, U D, Sheppard I, Wilson CR. Review article: improving drug safety for patients undergoing anesthesia and surgery. Can J Anesth. 2013;60:127–135.
10. Orser BA, U D, Cohen MR. Perioperative medication errors: building safer systems. Anesthesiology. 2016;124:1–3.
11. Webster CS, Merry AF, Larsson L, McGrath KA, Weller J. The frequency and nature of drug administration error during anaesthesia. Anaesth Intensive Care. 2001;29:494–500.
12. Orser BA, Chen RJ, Yee DA. Medication errors in anesthetic practice: a survey of 687 practitioners. Can J Anesth. 2001;48:139–146.
13. Llewellyn RL, Gordon PC, Wheatcroft D, et al. Drug administration errors: a prospective survey from three South African teaching hospitals. Anaesth Intensive Care. 2009;37:93–98.
14. Shridhar Iyer U, Fah KK, Chong CK, Macachor J, Chia N. Survey of medication errors among anaesthetists in Singapore. Anaesth Intensive Care. 2011;39:1151–1152.
15. Merry AF, Webster CS, Hannam J, et al. Multimodal system designed to reduce errors in recording and administration of drugs in anaesthesia: prospective randomised clinical evaluation. BMJ. 2011;343:d5543.
16. Jelacic S, Bowdle A, Nair BG, Kusulos D, Bower L, Togashi K. A system for anesthesia drug administration using barcode technology: the codonics safe label system and smart anesthesia manager. Anesth Analg. 2015;121:410–421.
17. Nanji KC, Patel A, Shaikh S, Seger DL, Bates DW. Evaluation of perioperative medication errors and adverse drug events. Anesthesiology. 2016;124:25–34.
18. Norman D. Carroll JM. Cognitive artifacts. In Designing Interaction: Psychology at the Human-Computer Interface (Cambridge Series on Human-Computer Interaction). 1991:New York: Cambridge University Press; 17–38.
19. Brooke J. Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, eds. SUS: a “quick and dirty” usability scale. In Usability Evaluation in Industry. 1996:London: Taylor and Francis; 189–194.
20. Brown LD, Cai TT, Dasgupta A. Interval estimation for a binomial proportion. Stat Sci. 2001;2:101–133.
21. Williams T, Szekendi M, Pavkovic S, Clevenger W, Cerese J. The reliability of AHRQ common format harm scales in rating patient safety events. J Patient Saf. 2015;11:52–59.
22. Perla RJ, Provost LP, Murray SK. The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf. 2011;20:46–51.
23. Provost LP, Murray SK. The Health Care Data Guide. 2011.San Francisco: Jossey-Bass.
24. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum-Comput Int. 2008;24:574–594.
25. Sanches I, Teixeira F, dos Santos JM, Ferreira AJ. Effects of acute sleep deprivation resulting from night shift work on young doctors. Acta Med Port. 2015;28:457–462.
26. Ross V, Jongen EM, Wang W, et al. Investigating the influence of working memory capacity when driving behavior is combined with cognitive load: an LCT study of young novice drivers. Accid Anal Prev. 2014;62:377–387.
27. Naismith LM, Cheung JJ, Ringsted C, Cavalcanti RB. Limitations of subjective cognitive load measures in simulation-based procedural training. Med Educ. 2015;49:805–814.
28. Byrne AJ, Oliver M, Bodger O, et al. Novel method of measuring the mental workload of anaesthetists during clinical practice. Br J Anaesth. 2010;105:767–771.
29. Byrne A. Measurement of mental workload in clinical medicine: a review study. Anesth Pain Med. 2011;1:90–94.
30. Pfeffer S, Maier T, Stricker E, Trick M. Cognitive ergonomics and informatory load in anesthesia. Biomed Tech. 2012;57:947–950.
31. Weiser TG, Haynes AB, Lashoher A, et al. Perspectives in quality: designing the WHO Surgical Safety Checklist. Int J Qual Health Care. 2010;22:365–370.
Copyright © 2017 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the International Anesthesia Research Society.
32. Lovell S, Kemp K. Dieter Rams: As Little Design as Possible. 2011.London: Phaidon Press.