Pharmacy practice in North American health care centers is typically divided into clinical pharmacy, which refers to clinical pharmacists providing pharmaceutical care directly to patients as a part of multidisciplinary teams, and pharmacy services, which refers to drug order validation (thereafter order validation) and drug distribution, including sterile and nonsterile compounding.1 Centralized order validation is an important component of pharmacy services, where pharmacists individually review orders for basic clinical validation. Examples include checking the dose, route, frequency, and verifying drug allergies, drug intolerance, drug interactions, drug duplications, among other things.2 This activity specifically focuses on error detection in an effort to prevent medication errors.1–5
Although order validation is a primary and fundamental act of pharmacy practice, paradoxically, there exists relatively little literature on this process. Previous studies have shown that centralized order validation is effective to prevent medication errors.3 However, health care center pharmacies are busy environments where many activities take place simultaneously. It has been shown that increased workload can be a risk factor for errors during validation.4 In addition, pharmacists are dependent on institutional software to perform this task. Design and workflow issues in pharmacy information systems (PISs) can contribute to errors, similar to those shown with electronic medical records.6
In most settings, pharmacists working at order validation are repeatedly interrupted during their work and must often interrupt their own validation to obtain information or to perform other tasks.7–9 Many activities related to medication, such as administration of medication by nurses, reveal that interruptions represent a risk factor for errors.10,11 Workplace design may also have an effect on medication errors and further influence interruptions.12 However, this relationship has not been specifically studied in the context of order validation by pharmacists.
Simulation in hospital pharmacy practice has mainly focused on pharmaceutical care, with studies showing how its application can improve provided care and optimal communication with patients and interdisciplinary team skills.13–17 Simulation studies on order validation conducted so far have been paper based and focused on validation elements and sequence.2 Studying the occurrence of errors during actual order validation can prove difficult because these events occur infrequently and are heterogeneous. However, a simulation study could be beneficial in studying medication errors occurring in a controlled setting,18,19 allowing for a more homogenous and larger sample of errors.
Oculometry, also called eye tracking, is a technique involving the capture of eye gaze and movement using specialized devices. It is generally used to determine what a person observes on a computer screen over time. It has mostly been studied in research contexts,20 but emerging studies showed a potential in analyzing the practical use of electronic medical records and clinical decision support systems.21 This could potentially be applicable to PISs because these systems share several features with electronic medical records and clinical decision support systems.
We developed a pilot, full-scale simulation study using eye tracking focusing on interruptions and the detection of medication errors by pharmacists and pharmacy residents during order validation.
The primary objective of this study was to determine the rate of medication error detection and the effect of interruptions on error detection during simulated order validation by pharmacists and pharmacy residents.
Because this was a pilot study, secondary objectives were to:
- Determine the number of orders validated during the simulation.
- Measure the time required to perform the simulation.
- Describe the reactions of participants to external interruptions (eg, telephone calls).
- Characterize self-interruptions generated by participants (eg, interrupting their own validation to perform other tasks, such as calling a nurse or speaking to a pharmacy technician).
- Describe participant perceptions toward the simulation.
This was a prospective, observational, pilot simulation study. The study took place at a medical simulation facility located within a 500-bed mother-child university hospital center. A space reproducing an order validation workstation in the central pharmacy was set up with appropriate computers, software, telephones, and paper references. To enhance realism, a pharmacy student was scripted to play the role of a pharmacy technician working at order entry next to the pharmacist in the simulation bay. This study was approved by the institutional research ethics committee (Number 2017-1548) and informed consent was obtained before participation.
A convenience sample of 16 pharmacists and pharmacy residents was constituted. Inclusion criteria were as follows:
- Pharmacist at or pharmacy resident affiliated with our institution.
- Available for 1 hour during the simulation days.
- Trained at order validation on the institutional PIS (GesPharx 8; CGSI Solutions TI, Quebec, Québec, Canada).
- Not having participated in the previous dry run.
- Having provided informed consent.
Participants agreed to not discuss the contents of the simulation scenario outside of their session.
Simulation sessions were video recorded. Screen capture was set up to record the computer screen showing the PIS order validation screen. A REDn oculometer (SensorMotoric Instruments, Teltow, Germany) was mounted on this screen and tracked eye motion using infrared light when the participant looked at the screen. Data captured with this device were processed in the software (SMI REDn Scientific; SensorMotoric Instruments, Teltow, Germany) and a circle showing eye gaze was superimposed to the screen capture videos. Workstations on which validation is performed had two monitors, one showing the PIS and the other showing the scanned handwritten prescription. Because of technical limitations, the screen showing the prescription was not captured by the oculometer.
A scenario script was written by the research team based on real orders and telephone calls. The scenario performed during the simulation had been pilot tested to ensure that there were no major issues. All orders were deidentified and entered into fictional patient records adapted from real records. The scenario consisted of the validation of three handwritten medication order pages for three patients (one page per patient). The pages contained three, five, and four medication orders, respectively. Four, nine, and four errors were present on each page, respectively. A medication order could contain a prescription error or a computer entry error by the pharmacy technician. Errors could be major (eg, seven-fold dose error) or minor (eg, missing instruction or comment in the software entry). Four major errors and 13 minor errors were present among all order pages. The first two pages contained two major errors each and respectively two and seven minor errors. The third page contained only four minor errors. During the simulation, three phone calls were timed to interrupt the participant during the validation of one order for each prescription page. The first two phone calls mandated an intervention by the pharmacist (see Table, Supplemental Digital Content 1, http://links.lww.com/SIH/A396, which provides complete error description and phone call details). Participants were instructed to act as they would in a real situation, for example, speaking to the pharmacy technician to fix an order entry error, or calling a clinical pharmacist or a physician to inquire about a possible ordering error. They were also instructed to manage phone calls as they would in real life. However, to obtain comparable data between participants, we required that participants validate the pages in the order in which they were presented, and move to the following page only after they were confident they were done with the current one.
Each session started by obtaining informed consent, followed by a prebriefing and instructions by a member of the research team. The oculometer was calibrated to the participant, and then the session started. The research team monitored the simulation in a control room with audio and video in real time. The phone calls were initiated according to a scripted procedure at given points in the orders. The session was stopped after 15 minutes or until at least one order page was completed. The session was immediately followed by an individual structured debriefing session with members of the research team. Participants completed a demographic questionnaire. After study completion and data analysis, we conducted a group debriefing session where all participants were invited. Preliminary results were presented, and participants were encouraged to express their thoughts and to share the strategies used to manage interruptions and detect errors.
Participants were divided into two groups according to their experience: “senior” group composed of pharmacists with more than 2 years' experience and the “junior” group composed of residents as well as pharmacists with less than 2 years' experience.
The videos, screen captures, and oculometry data were independently analyzed by two research assistants, one blinded to participants and uninvolved in the study. They compiled data on interruptions, error detection, management strategies for phone calls and errors, as well as the time spent on each activity. This independent coding was compared using κ coefficients for categorical variables and intraclass correlations for continuous variables. Discrepancies were resolved by consensus before analysis.
For each order, we evaluated if an interruption occurred during validation, if each error in the order was detected, and how each error was managed. Errors were only included in the analysis if a participant was exposed to the error during his session. An error was considered detected if participants took action or indicated that an error had been detected in their validation notes, even if no action was taken.
Eye-tracking data were used to break down the validation activities of each participant into time intervals and to confirm that each error considered as detected had actually been seen on the validation screen. It is important to note that eye-tracking data cannot be used alone to confirm detection of an error because information being seen does not ensure that it was perceived and processed by the participant.
For each phone call, time spent on the call, accuracy in answer, and the strategy used to manage the interruption were collected.
Finally, time required to complete the scenario, number of pages completely validated, and time required per page were collected. We also collected the number of times the order validation screen was opened as a measure of the validation process structure.
The primary objective was analyzed using a Cochran-Mantel-Haenzel test taking into account the group as a stratification variable, to control for experience on the effect of interruptions on error detection. Considering the pilot nature of this study and the convenience sample, other objectives were explored descriptively using summary statistics; no additional statistical comparisons were made.
We recruited 16 participants as planned. Ten were senior and six junior. Four participants in the junior group were pharmacy residents. Participant characteristics are presented in Table 1. Three videos were lost because of file corruption. These participants were excluded from further analysis. The final sample was composed of 13 participants: eight seniors and five juniors.
As anticipated, the first page was completed by all participants. Six participants in the senior group completed the second page. None of the participants completed the entire scenario in the allocated time. Consequently, we only included errors from the first page in the analysis to ensure comparability between participants. The mean time to complete the session was 15 minutes 33 seconds in the senior group and 17 minutes 29 seconds in the junior group. The mean time required to validate the first page (excluding time spent on other activities caused by interruptions) was 6 minutes 42 seconds in the senior group and 10 minutes 56 seconds in the junior group. Raters did not differ by more than 4 seconds for this coding; intraclass correlation coefficients for these measures approached a value of 1, thus showing high interrater reliability.
The rate of medication error detection on the first page in the senior group was 69% (22/32) and 95% (19/20) in the junior group. Raters agreed perfectly on this objective (κ = 1). All major errors exposed to participants were detected. For all detected errors, information leading to error detection on the order validation screen was seen.
Table 2 shows the number of errors detected in medication orders with or without interruption. We observed a significant association between interruption and lack of error detection (odds ratio = 0.149, 95% confidence interval = 0.042–0.525, P = 0.005). This association did not vary significantly between groups (P = 0.832). Interrater agreement for this objective was perfect (κ = 1).
Management of External (Phone) Interruptions
Twenty (2.5 per participant) phone interruptions took place in the senior group and eight (1.6 per participant) in the junior group. The most frequent strategy used by participants to mitigate these interruptions was to ask the technician to put the call on hold while they completed a task. Some participants took the call themselves and then put it on hold before responding to it. Six (30%) calls were put on hold by senior participants, whereas two (25%) calls were put on hold by junior participants. The mean time spent on the first phone call was 1 minute 18 seconds in the senior group and 1 minute 08 seconds in the junior group. Intraclass correlation coefficient for this measure was 0.998. All calls that required a pharmacist response were correctly answered.
Management of Self-Interruptions
Self-interruptions occurred most frequently when a participant detected an error or thought that something could be an error and took actions to confirm this hypothesis. Participants spoke with the pharmacy technician 13 times (1.6 per participant) in the senior group and 13 times (2.6 per participant) in the junior group. There were four calls (0.5 per participant) to the simulated patient's nurse in the senior group and two (0.4 per participant) in the junior group. There were eight calls (1 per participant) to the clinical pharmacist in charge of the simulated patient by seniors and six (1.2 per participant) by juniors. The raters agreed strongly (κ = 0.955) on these data. Communications with the technician were mostly to ask for a correction in a software entry, whereas communications with nurses and other pharmacists were mostly to inquire about poorly written prescriptions, to ask for prescription correction, or to confirm weight and allergy status. These actions were directly related to the errors embedded within the scenario.
During validation of the first page, the order validation screen was opened a mean of 3.38 times per participant in the senior group and 4.20 times per participant in the junior group, with an interrater agreement of κ value of 0.913.
Debriefing and Participant Perceptions
During debriefing, participants addressed three major themes. They felt that this exercise was stressful at first and that they experienced higher vigilance than usual because they expected to be presented with errors to detect. However, as the simulation progressed, they were able to work as they usually would. They also expressed positive perceptions toward the debriefing session that allowed them to reflect on their experience. Finally, participants commented that although the level of realism of the scenario was acceptable, albeit with a larger number of errors, the simulation bay lacked some realism because it was quieter than usually. The actual order validation zone in the pharmacy is a busy environment with noise originating from machines and people.
Our study showed that an interruption by a phone call during the validation of an order containing an error significantly decreased the probability of detecting that error. Although this finding is logical considering similar data observed in other situations involving medication,8–11 the large effect size with an odds ratio of 0.149 was surprising. We have little reason to doubt that such an effect would exist. However, the context of the simulation might have magnified the effect leading to overestimation. The scenario was error dense, with several errors introduced in a single order page. Participants commented that the number of errors in the scenario was high compared with what they observe in reality. Although this should be interpreted with caution, this interesting finding warrants further exploration under different conditions to see whether this is generalizable to others settings and contexts.
Junior pharmacists took more time to validate the first order page than seniors, and juniors detected almost all errors, whereas seniors detected 69%. We believe that experience might explain these findings. We hypothesize that as senior pharmacists gain experience at validating orders with time constraints, they might actively manage their time, take risks, and choose to focus on clinically significant errors. Indeed, all major errors were detected by all participants. Differences in detection were exclusively attributed to minor errors. Several minor errors in our scenario were clinically inconsequential, for example, attributing an order to doctor A in the pharmacy software while the order was actually signed by doctor B. Because of the high density of errors in our scenario, experienced pharmacists may have chosen to purposefully ignore these errors to focus their attention on what they considered clinically significant. Pharmacists with less experience and pharmacy residents may have been reluctant to do so because they were less skilled in anticipating repercussions of a minor error slip. Although one should be cautious about extrapolating this finding to the real world, it would be interesting to further explore how professionals balance the risk of ignoring clinically insignificant errors with the time and effort required to correct them.
Although senior pharmacists were exposed to more phone calls than juniors, we did not observe major differences in external interruption management strategies between the two groups. During debriefings, participants seemed very aware that taking phone calls while validating could present a risk and several chose to put the call on hold as a way to mitigate that risk. Self-interruptions to speak to the pharmacy technician were observed more frequently in the junior group. Qualitatively, during video review and debriefing, we observed that this could have been related to inexperience and uncertainty about whether an element was an actual error. This uncertainty might explain the higher number of accesses to the order validation screen in the junior group, showing back-and-forth actions between validating and obtaining information.
This study showed the feasibility and innovation involved in combining video, screen capture, and oculometry into a pharmacy simulation. The results are not ground breaking, but this methodology could be used for future research on the workflow of pharmacists working at order validation. Although this seems to be innovative, readers should keep in mind the workload associated with data entry before the simulation to provide a likely scenario. Further studies are required to identify optimal technology combinations that would provide a data set more easily exploitable. Practical applications of this technique may include real time detection of fields that were not gazed at during validation as a safety feature of PISs.
The limits of our study include a small sample size and a convenience sample. Although this might limit the generalizability of our findings, this pilot study showed that simulation is feasible as a method to study order validation by pharmacists, specifically for events that are difficult to observe prospectively in practice, such as medication errors. Furthermore, as discussed previously, the simulation nature of the study and the error-dense scenario may have affected the primary outcome. Future studies using the same methodology should plan a “calibration exercise,” such as validating a short order before actual data collection would begin, to control for this effect. Our use of eye tracking in this study was not optimal. Unfortunately, the oculometer only recorded the order validation screen and not the screen showing handwritten prescriptions. As stated in the results, we were able to ascertain that errors shown on the validation screen were seen for all errors that were acted upon. However, we were unable to analyze undetected errors with oculometer data, because we could not determine whether an error, which was not acted upon, was seen or not. Indeed, this error could have been seen only on the prescription screen and therefore not captured by oculometer data. Future studies should ensure that the entire computer display is recorded and tracked by the oculometer, because these data could help determine whether there is a correlation between error detection as defined by oculometer data and by actions taken.
The use of simulation in health care is gaining popularity and evidence showing its validity is increasing.22 Two meta-analyses recognized simulation as an effective teaching tool, superior to traditional teaching methods.23,24 Although in practice errors must be prevented and corrected when they occur, errors during simulated scenarios can be embedded on purpose to correct behaviors at their source. They can also be used to explore decision patterns leading to errors in a safe environment.25 A recent systematic review concluded that simulation could be effective in preventing events occurring rarely, such as medication errors, but that the effectiveness of such scenarios depended heavily on their realism.26
Thus, we believe that simulation of order validation has great potential as a teaching tool for pharmacists in training or in practice, especially for competencies such as medication error detection, managing interruptions, communication, and teamwork. Inclusion of these teaching modalities in university curricula specifically targeting order validation, or during initial training in a health care institution, could increase awareness of the impact of interruptions on the quality of work delivered by pharmacists. In addition, this could further contribute to teaching evidence-based management strategies to pharmacists, which are currently acquired informally through experience.
Human factors research shows that simulation is a useful tool to study interruptions, errors, and risk. Recent research on human factors in healthcare focuses on high-risk environments such as intensive care units or emergency departments, studying mainly physicians and nurses in those environments.27–31 This literature shows that interruptions are frequent and contribute to medication errors and that the best strategies for their prevention and management are unknown. Certain interruptions may also be beneficial, depending on context. For example, Myers et al32 found that interruptions resulting in refocusing nurses' attention to the patient because of an emerging need were beneficial to the quality of care provided, whereas those resulting in interrupting attention or treatment were detrimental. Another body of research focuses on human factors in medical informatics, especially in the context of electronic prescribing or clinical decision support.33 Medication order validation by pharmacists has common characteristics with both contexts, notably a high-risk, stimuli-rich environment with many distractors and parallel tasks, as well as interaction with software of variable features and design quality. The environment designed for this study could be used to further explore human factors in pharmacist work by varying workload and measuring error detection and management, for example.
In conclusion, we observed an association between phone interruptions and a decrease in error detection during validation, but the size of this effect may have been overestimated. We observed that pharmacists with more experience tend to work faster but may not act upon minor errors, whereas pharmacy residents and pharmacists with less experience work slower but act upon more errors, albeit with more self-interruptions. We believe that simulation provides an opportunity to study order validation by pharmacists and represents a valuable teaching tool for pharmacists and pharmacy residents learning order validation. The educational impact of simulated-based training for order validations may potentially reduce actual medication errors and ultimately lead to better patient safety.
1. American Society of Hospital Pharmacists. ASHP guidelines: minimum standard for pharmacies in hospitals. Am J Health-Syst Pharm
2. Dubois S, Barbier A, Thibault M, Atkinson S, Bussières JF. Selection and optimal sequence of critical elements for medication review: a simulation with hospital pharmacy
residents. Ann Pharm Fr
3. Ibáñez-Garcia S, Rodriguez-Gonzalez CG, Martin-Barbero ML, Sanjurjo-Saez M, Herranz-Alonso A. Adding value through pharmacy
validation: a safety and cost perspective. J Eval Clin Pract
4. Gorbach C, Blanton L, Lukawski BA, et al. Frequency of and risk factors for medication errors by pharmacists during order verification in a tertiary care medical center. Am J Health-Syst Pharm
5. Charpiat B, Goutelle S, Schoeffler M, et al. Prescriptions analysis by clinical pharmacists in the post-operative period: a 4-year prospective study. Acta Anaesthesiol Scand
6. Amato MG, Salazar A, Hickman TT, et al. Computerized prescriber order entry-related patient safety reports: analysis of 2522 medication errors. J Am Med Inform Assoc
7. Guérin A, Caron É, Lebel D, Bussières JF. Before-and-after study of interruptions in a pharmacy
department. Can J Hosp Pharm
8. Beso A, Franklin BD, Barber N. The frequency and potential causes of dispensing errors in a hospital pharmacy
. Pharm World Sci
9. Flynn EA, Barker KN, Gibson JT, et al. Impact of interruptions and distractions on dispensing errors in an ambulatory care pharmacy
. Am J Health-Syst Pharm
10. Johnson M, Sanchez P, Langdon R, et al. The impact of interruptions on medication errors in hospitals: an observational study of nurses. J Nurs Manag
11. Westbrook JI, Woods A, Rob MI, Dunsmuir WT, Day RO. Association of interruptions with an increased risk and severity of medication administration errors. Arch Intern Med
12. Committee on the Work Environment for Nurses and Patient Safety, Board on Health Care Services: Work and Workspace Design to Prevent and Mitigate Errors, Keeping Patients Safe: Transforming the Work Environment of Nurses. In: Page A. ed. Washington, DC, National Academies Press (US), 2004, pp 226–285. Available at: https://www.ncbi.nlm.nih.gov/books/NBK216190/
. Accessed October 30, 2018.
13. Serag-Bolos ES, Miranda AC, Gelot SR, Dharia SP, Shaeer KM. Assessing students' knowledge regarding the roles and responsibilities of a pharmacist
with focus on care transitions through simulation. Curr Pharm Teach Learn
14. Thompson Bastin ML, Cook AM, Flannery AH. Use of simulation training to prepare pharmacy
residents for medical emergencies. Am J Health-Syst Pharm
15. Collier IA, Baker DM. Creation of an active learning healthcare communications course using simulations relevant to pharmacy
practice. Curr Pharm Teach Learn
16. Fejzic J, Barker M, Hills R, Priddle A. Communication capacity building through pharmacy
practice simulation. Am J Pharm Educ
17. Leon N, Hajjar E, DeSevo Bellottie G. Sequencing of simulation and clinic experiences in an introductory pharmacy
practice experience. Am J Pharm Educ
18. Metzger NL, Chesson MM, Momary KM. Simulated order verification and medication reconciliation during an introductory pharmacy
practice experience. Am J Pharm Educ
19. Daupin J, Atkinson S, Bédard P, et al. Medication errors room: a simulation to assess the medical, nursing and pharmacy
staffs' ability to identify errors related to the medication-use system. J Eval Clin Pract
20. Asan O, Yang Y. Using eye trackers for usability evaluation of health information technology: a systematic literature review. JMIR Hum Factors
21. King AJ, Hochheiser H, Visweswaran S, Clermont G, Cooper GF. Eye-tracking for clinical decision support: a method to capture automatically what physicians are viewing in the EMR. AMIA Jt Summits Transl Sci Proc
22. Orledge J, Phillips WJ, Murray WB, Lerant A. The use of simulation in healthcare: from systems issues, to team building, to task training, to education and high stakes examinations. Curr Opin Crit Care
23. Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a systematic review and meta-analysis. Simul Healthc J
24. McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med J Assoc
25. Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Simul Healthc
26. Sarfati L, Ranchon F, Vantard N, et al. Human-simulation-based learning to prevent medication error
: a systematic review. J Eval Clin Pract
2018. Available at: https://doi.org/10.1111/jep.12883
. Accessed October 30, 2018.
27. Biron AD, Loiselle CG, Lavoie-Tremblay M. Work interruptions and their contribution to medication administration errors: an evidence review. Worldviews Evid Based Nurs
28. Blocker RC, Heaton HA, Forsyth KL, et al. Physician, interrupted: workflow interruptions and patient care in the emergency department. J Emerg Med
29. Bower R, Jackson C, Manning JC. Interruptions and medication administration in critical care. Nurs Crit Care
30. Raban MZ, Westbrook JI. Are interventions to reduce interruptions and errors during medication administration effective?: a systematic review. BMJ Qual Saf
31. Ratwani RM, Fong A, Puthumana JS, Hettinger AZ. Emergency physician use of cognitive strategies to manage interruptions. Ann Emerg Med
32. Myers RA, McCarthy MC, Whitlatch A, Parikh PJ. Differentiating between detrimental and beneficial interruptions: a mixed-methods study. BMJ Qual Saf
33. Phansalkar S, Edworthy J, Hellier E, et al. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc