Journal Logo

FIRES Article

Participation in a System-Thinking Simulation Experience Changes Adverse Event Reporting

Sanko, Jill S. PhD, MS, APRN, CHSE-A, FSSH; Mckay, Mary DNP, APRN, CNE

Author Information
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: June 2020 - Volume 15 - Issue 3 - p 167-171
doi: 10.1097/SIH.0000000000000473
  • Free

Abstract

System failures are significant contributing factors in the hundreds of thousands of adverse events occurring in US healthcare institutions each year. Despite persuasive evidence that changing systems would reduce harm, altering healthcare systems is especially challenging.1 Healthcare systems are some of the most complex organizational structures, involving continually evolving, intricate technology used by a menagerie of highly self-directed individuals1 to care for complicated and often quite infirm patients. Lucian Leap is noted to have said; systems cannot be improved if they are not understood1; thus, system thinking (ST) is needed if there is a desire to better existing systems.2 Richmond, a well-known leader in the field of systems dynamics, is credited with coining the term “system thinking.”3 Richmond recognized that as society becomes more reliant on greater interdependencies, society must learn that paying attention to only their particular “piece of the rock” and only solving problems at the local level will stifle the evolutionary progress humans have enjoyed since the beginning of time. This way of thinking is important in general4,5 but vital in healthcare because of the multiple complex systems that are interdependent upon each other for good outcomes to occur.6,7

System thinking has numerous definitions but at its core is an understanding that outcomes of systems are products of the interrelationships and interactions of system parts.8 System thinking is fundamental to quality improvement efforts,7 can be measured, and has been suggested as a key element in improving patient safety.9 Recent research demonstrated a relationship between ST and safety practices.10 A study by Hwang and Park10 showed that nurses with higher ST measured by the System Thinking Scale (STS)11 scores had a greater tendency to report medical errors.

Under reporting of adverse events occurring in healthcare is a well-known problem.12 The failure to capture the facts associated with errors and near misses significantly reduces the possibility that a complete understanding of the causes of these events can be fully known. An increase in error reporting may lead to a decrease in the number of errors occurrences because of the ability to develop and deploy possible targeted prevention efforts that could come from uncovering the causes. The need for more reporting as well as an increase in the quality of reporting is suggested.13,14 Moreover, there are noteworthy examples of learning and subsequent improvements that have been made after reporting of serious patient safety events.15 One of the major recommendations of the landmark 1999 Institute of Medicine report, To Err is Human, was to focus on incident reporting.16

This study explored the hypothesis that exposure to a simulation experience aimed at improving ST would impact adverse event reporting patterns in undergraduate nursing students. Furthermore, it was hypothesized that differences in reporting patterns of specific types of adverse events based on having or not having exposure to a simulation designed to teach ST would be found. Finally, we hypothesized that the effects of the exposure would be impacted by time.

METHODS

After institutional review board approval, an intervention-control study design using secondary data from 3 cohorts of undergraduate Accelerated Option Bachelor of Science of Nursing (AO-BSN) students were used to explore the impacts of participation in a simulation experience aimed at developing ST on adverse event reporting. Students enrolled in the AO-BSN program are second-degree students, who have completed a prior degree in a field other than nursing but are now seeking a BSN degree through a 1-year accelerated program.

For this study, data from 2 cohorts of students served as the control group and data from a third as an intervention group. Because randomization was not used to select the intervention group, 2 rather than 1 comparison year were selected to increase the probability that the findings represented actual differences in behaviors rather than falsely detecting a treatment bias.17 The data from these 2 groups were combined to form the control group.

Patient Safety Course

Each summer (the second semester of course work 1 of 3), students admitted in our spring semester who are enrolled in the university's AO-BSN program participate in a week-long simulation-based interprofessional patient safety course (SBE-PS-IPE) with students from the school of medicine entering their third year of medical school. This annual course has the objective of providing students foundational knowledge and skills on the topics of patient safety and teamwork in an interprofessional context. The SBE-PS-IPE course is an adaptation of an original course developed at our university for preclinical medical students18 and has been running since 2013. During the summer of 2017, students enrolled in the SBE-PS-IPE course participated in Friday Night in the ER (FNER)19 as one of the course simulation activities. During the course, students from nursing and medicine are grouped into mixed discipline teams in which they encounter most course simulation activities. Year to year, few substantial changes are made to the overall SBE-PS-IPE course objectives or activities. Most of the changes made from year to year include tightening scenario objectives, updating scenarios, and tweaking scheduling of activities to improve course flow. In 2017, however, the opportunity to add FNER as a simulation experience was presented, affording the ability to study differences in adverse event reporting in groups who experienced FNER and those who did not.

Friday Night in the ER was not included subsequent course years because of factors related to limited space and faculty resources needed to be able to include it. Outside of the addition of the FNER activity, the only changes made to the course from 2016 to 2018 included eliminating a scenario in 2017 and beyond that had poor evaluations, which was focused on safety for patients with dementia, and adding a discharge planning case in 2017 as part of a follow-through scenario activity for an already included scenario; this new scenario continues to be included as part of the course. We additionally made a change to the course group-graded assignment. Before 2018, the assignment was a group root cause analysis of a case assigned to each team. In 2018, this assignment was changed to one where the teams were tasked with identifying adverse event found in a literature search and presenting an evidence-based solution that would help prevent a future occurrence of the event. In addition, there were no changes to the core faculty running or facilitating the courses across all 3 years of this study.

Higher ST scores measured using the STS11 have been found in nursing, medical, pharmacy, and physical therapy students after participation in an FNER simulation20,21; thus, it was presumed that ST was different in the control and intervention groups. It was further hypothesized that this difference would also result in an observable difference in adverse event reporting. As part of the SBE-PS-IPE course, students in the intervention group were administered the STS11 before and after exposure to FNER. Administration of the STS was done for the purposes of evaluating the learning impacts of the activity, but students were asked if their data could be used for the purposes of research. Data from students indicating that they did not wish for their data to be included in research were excluded from analysis. The results of this analysis are included in the results section.

Friday Night at the ER

Friday Night at the ER is a commercially available table-top simulation used to teach and developing ST.19,22 Friday Night at the ER has a global following, with more than 1000 licensed users, more than 20 years of use, and has been used in a multitude of disciplines both within and outside healthcare to teach ST.19–23 Despite its healthcare context and name, the ability to play and subsequently learn from FNER does not require learners to have healthcare knowledge. Friday Night at the ER engages teams of 4 players at a board representing a simulated hospital composed of 4 hospital departments (emergency department, surgery, step-down, and critical care).19 Friday Night at the ER challenges teams to manage a busy hospital during a simulated 24-hour period.19,22 Each player handles patient flow and staffing needs of their department, deals with any emergencies that arise, and documents performance based on prescribed metrics that are tracked as a department manager.19 Multiple boards can be played during a session to simulate a multihospital system context. Friday Night at the ER sessions commences with a prebriefing that includes how and why to play and culminates with an in depth debrief. Sessions are carried out by trained facilitators using provided program power point slides that guide the debriefing keeping it aligned with the simulation's objectives but are fluid enough to allow program-specific discussions to unfold and examples to be used. In total, an FNER session takes approximately 2 hours to run. Formal training is not a requirement to facilitate the activity; however, formal training is available. The faculty who facilitated the sessions was a formally trained facilitator. Requirements to run a session include space with appropriately sized tables (20 inches by 20 inches), chairs for each participant, print outs of the tracking paper work used as part of the metrics collected during the game, writing utensils, and a screen and projector to display the power point slides.

Given the large number of students participating in the 2017 SB-PS-IPE, 2 sessions of FNER were run to accommodate all students participating in the course. To facilitate the 2 sessions of large groups, we needed to find adequate space to fit 27 tables of approximately 4 players. Adjustments can be made to accommodate groups that are not divisible by 4 by “overstaffing or understaffing” a simulated hospital. This condition can be incorporated into debriefing to bring about discussions on the impacts of such circumstances on systems.

Although we were able to find a space large enough to run the FNER activity in 2017, we were not as fortunate in subsequent years and therefore had to make the decision to drop this activity from the SB-PS-IPE course. Friday Night at the ER is still used in other courses, however, throughout the year where smaller groups can be arranged.

Measures

The STS

The STS was developed by Moore and Dolansky11 (2010) originally as a 30-item 3-factor tool. Initial psychometric analysis demonstrated low factor loadings for 2 of the 3 factors, therefore items for these factors were not included in their subsequent psychometric analysis of the tool. Secondary analysis of the tool undertaken by Moore et al11 included only the 20 items included as part of the factor titled System Interdependencies. This analysis demonstrated a single factor tool with a Cronbach α value of 0.89 and test-retest reliability of 0.74. As part of a multisite study exploring the impacts of FNER on ST,21 further psychometric scale analysis of the 20-item tool was undertaken. This analysis found good evidence of validity and reliability of the STS. All interitem correlations were greater than 0.410; Cronbach α value was equal to 0.994.21

Adverse Event Reporting System

As part of the usual school of nursing simulation program, all nursing students are encouraged to report adverse events observed or committed during simulation encounters in a simulated adverse event reporting system (AERS)24–26 embedded into the simulation program. The goals for developing and embedding AERS into the simulation program were 2-fold: (1) to provide a place where students could practice adverse event reporting and (2) as a tool to collect data to direct evidence-based improvements in curricula. Validation of this system was completed in a feasibility study conducted before the full-scale launch.24 For this current study, all adverse events reported by students participating in the 2016 (control, n = 68), 2017 (intervention, n = 85), and 2018 (control, n = 78) patient safety courses were extracted from the system and analyzed to explore the impacts of exposure to FNER on adverse event reporting behaviors. Two semesters of adverse event reporting data {the semester in which students participated in the patient safety course (second semester of course work) and the subsequent semester [the last (third, terminal) semester of course work]} were analyzed from 3 cohorts of students enrolled in the AO-BSN program and admitted during the spring semester.

Statistical Analysis

System Thinking Scores

Paired t tests were used to analyze pre-post ST scores in the intervention group using statistical software following extraction of the data from the electronic web-based system used to collect the data.

Adverse Event Reporting Rates

Data from the 3 cohorts of students included in the study were extracted from the adverse event reporting database used as part of the school of nursing simulation program. Descriptive statistics was used to describe the characteristics of study cases and calculate rates of adverse event reporting. To control for differences in the samples sizes among the groups compared, proportions were calculated using raw data for each group and condition. Proportion comparisons were completed using χ2 tests to examine differences in adverse events reported by participants in the groups studied.

Data Analysis Procedures

Multiple tests were performed to be able to detect where and how long impacts of exposure to FNER on reporting patterns would be found. To determine whether there would be washout of the effect of exposure comparisons were made in the semester in which students encountered the simulation as well as the subsequent semester following the intervention. To determine whether one group reported more often than the other, the proportions of the total number of adverse events made in each semester were examined. To determine whether exposure impacted reporting patterns of only certain types and categories of adverse events, stratification of the data was completed and then analyzed.

The stratifications were as follows: (1) comparisons of adverse events reported as error, near miss, sentinel, or other types and (2) comparisons of adverse events reported in the categories (scope of practice events, medication events, confidentiality breach events, fall events, order execution events, failure to rescue events, and airway events). In total, 24 separate χ2 tests were performed. To control for the use of multiple comparisons to uncover where and under what time frame changes were occurring the Benjamini-Hochberg (BH) procedure was used to control the false discovery rate.27 This procedure decreases the probability that an incorrect rejection of the true null hypothesis would occur because of the use of multiple comparisons.27 The BH procedure adjusts the P value. The BH P value is notated in all reported results.

RESULTS

Data Analysis

System Thinking

System thinking was measured in the intervention group before and after the intervention using the STS.11 Analysis showed a time effect (premean = 48.00, postmean = 65.81, P < 0.001) with a large effect size (d = 1.42).

Reporting of an Event Proportions Comparisons

Findings demonstrated differences in the proportions of reported adverse events based on exposure to FNER. In both semesters, the intervention group reported proportionally more adverse events than the control. The findings point to a somewhat longitudinal impact of the intervention on adverse event reporting; however, there is a noted drop-off in the differences from the semester where the exposure occurred in the following one. In semester where the exposure occurred, there was a 17.37% difference in the number of reports made (intervention = 66.27% Adverse event reporting rate (AERR), combined control years = 48.9% AERR, χ2 = 31.03, 95% confidence interval = 11.35 to 23.11), P < 0.001, BH P = 0.002); however, in the terminal semester, this difference although remaining significant shrunk to only a 5.8% difference, intervention group (55.8% AERR) reporting proportionally more adverse events compared with the combined controls [50.0% AERR, χ2 = 3.38, 95% confidence interval = −0.034% to 11.56%, P = 0.052, BH P = 0.143 (significant)].

Results by Adverse Event Type

Stratified comparisons examining reporting patterns across adverse event type (error, near miss, sentinel event) with control groups combined failed to show statistically significant differences in all analyses completed.

Results by Adverse Event Categories

Stratified comparisons examining reporting patterns across adverse event categories (medication events, scope of practice events, failure to rescue events, order execution events, airway events, fall events, and confidentiality breaches) without stratification of these events into the adverse event type (error, near miss, sentinel event) showed interesting patterns of differences (Tables 1–3).

TABLE 1
TABLE 1:
Proportions of Medication Event Reporting by Condition
TABLE 2
TABLE 2:
Proportions of Failure to Rescue Reporting by Condition
TABLE 3
TABLE 3:
Proportions of Airway Event Reporting by Condition

There were differences in medication event reporting patterns. Medication events were found to be reported statistically more often by the intervention group compared with the control group (Table 1) but was solely found to be significant in the semester where the FNER intervention occurred. This finding further supports the notion that there may be a washout effect of the interventional impact.

Failure to rescue and airway events were also found to have statistically significant differences when comparing the groups. The control group reported more failure to rescue events in the semester where FNER occurred; however, this effect was not found in the terminal semester (Table 2). Finally, there were statistically significant differences in reporting of airway events, with the control group reporting statistically more airway events than the intervention group; however, this is only found in the terminal semester (Table 3).

DISCUSSION

Participation in FNER as a simulation experience has been found to increase ST in this study as well as others21 and also seems to alter adverse event reporting frequency in general as well as for certain types of adverse events. In this study, findings demonstrated patterns of adverse event reporting that were different among the groups studied when events were stratified into their constituent categories. These pattern differences seem to be influenced by having had or not had exposure to FNER. This finding seems to suggest that those with presumably higher levels of ST related to exposure to FNER notice and subsequently report different categories of events.

The intervention group reported medication events more often than the control group (Table 1). This finding reflects what might be expected based on studies found in the literature reporting the association of most medication errors occurring related to systems factors.28 Moreover, medication errors have been described as having multidimensional causes.29 Thus, the conclusion that individuals with higher levels of ST might notice events more closely influenced by systems is plausible and also supported by systems theory.30

The control group was found to have statistically significant greater reporting of events categorized as failure to rescue and airway associated events. According to the literature, these types of errors tend to stem from breaches in cognition, and failure to fully monitor the situation, and subsequently missing changes in a patient's status.31–33 Unlike medication events, failure to rescue and airway events have a greater tendency to be caused by singular person process issues transpiring at the sharp end (the patient's bedside) of the care spectrum as opposed to systems related failures. Given the study findings, and supporting literature, it may be feasible to conclude that there is an association between individuals' level of ST and the noticing of certain categories of adverse events, whereas those with lower levels of ST may make or notice more events stemming from causes that do not involve system workings. This study also demonstrated that the effects of learning ST may be time limited; thus, attempts to teach or strengthen these skills should not occur only as a single event. Having repeated opportunities to reinforce learned concepts spanning over time may prove to be important in maintaining the skills and knowledge.

This study had several limitations including the use of a simulated AERS to capture data, the use of a student population, the inclusion of a single site, and using an approach that relied on multiple tests of stratified data even despite having used a technique for correcting this. Despite the significant history of the AERS and the prior work validating it, there remains the possibility that the data may be substantially different than what might be found in data collected from actual clinical adverse event reporting. Furthermore, the use of students and a single site could impact potential generalizability of the study findings. The study did not allow for a determination of differences in the commission versus the observation of adverse events because the data did not capture whether the person reporting had committed or noticed the reported event. Finally, to fully explore the potential changes in adverse event reporting at a detailed level, the use of multiple tests was used to explore stratified data. Although a BH correction procedure with a 25% false discovery rate was performed, it is still possible that one of the significant findings could have been falsely found.

Strengths of this study include the use of simulation as both an intervention and a method capture data challenging to otherwise obtain. This is one of the first attempts to examine the impacts of ST on adverse event reporting, thus providing a possible model for the future to explore the relationship between ST and safety monitoring.

CONCLUSIONS

The findings of this study may begin to support the notion that developing ST could change safety monitoring behaviors like what was found in a 2017 study by Hwang and Park.10 The findings of this research as well as those of the Hwang and Park10 study taken together support the premise that development of ST improves error reporting leading to enhanced error prevention strategies. Based on these findings, it may be of benefit to patient safety efforts to begin including ST content and regular reinforcement of its principles in prelicensure healthcare curricula as well as professional development programming.

REFERENCES

1. Leape L. Patient safety in the era of healthcare reform. Clin Orthopaed Relat Res 2015;473:1568–1573.
2. Leape L, Berwick D, Clancy C, et al. Transforming healthcare: a safety imperative. Qual Saf BMJ 2009;18:424–428.
3. Richmond B. Systems dynamics/systems thinking: let's just get on with it. Int Syst Dyn Rev 2006;10:135–157.
4. Richmond B. Systems Thinking: Four key questions. Watkinsville: High Performance Systems, Inc.; 1991:2–3.
5. Senge PM. The Fifth Discipline Field book: Strategies and Tools for Building a Learning Organization. New York: Currency – Doubleday; 1994:6–9.
6. Dolansky MA, Moore SM. Quality and Safety Education for Nurses (QSEN): the key is systems thinking. OJIN 2013;18(3):1.
7. Cipriano PF. Improving health care with systems thinking. Am Nurse Today 2008;3(9):6.
8. Peters DH. The application of systems thinking in health: why use systems thinking? Health Res Pol Syst 2014;12(1):51–51.
9. Agency for Healthcare Research and Quality: Hospital survey on patient safety culture. Available at: http://www.ahrq.gov/sops/quality-patient-safety/patientsafetyculture/hospital/index.html Web site. http://www.ahrq.gov/sops/quality-patient-safety/patientsafetyculture/hospital/index.html. Updated 2018. Accessed May 8, 2019.
10. Hwang J, Park H. Nurses' systems thinking competency, medical reporting, and the occurrence of adverse events: a cross-sectional study. Contemp Nurse 2017;53(6):622–632.
11. Moore SM, Dolansky MA, Singh M, Palmieri P, Alemi F: The systems thinking scale. Frances Payne Bolton School of Nursing 2010. Accessed at: https://case.edu/nursing/research/research-studies-labs/systems-thinking/systems-thinking-scale-manual. Accessed November 1, 2019.
12. Stergiopoulos S, Brown C, Felix A, Grampp T, Getz G. A survey of adverse event reporting practices among US healthcare professionals. Drug Saf 2016;39(11):1117–1127.
13. Hutchinson A, Young TA, Cooper KL, et al. Trends in healthcare incident reporting and relationship to safety and quality data in acute hospitals: results from the National Reporting and Learning System. Qual Saf Health Care 2009;18:5–10.
14. Mitchell I, Schuster A, Smith K, Pronovost P, Wu A. Patient safety incident reporting: a qualitative study of thoughts and perceptions of experts 15 years after ‘To Err is Human’. BMJ Qual Saf 2016;25(2):92–99.
15. Macrae C. The problem with incident reporting. BMJ Qual Saf 2016;25:71–75.
16. Kohn L, Corrigan J, Donaldson M. To Err Is Human: building a safer health system. Institute of Medicine-Committee Quality of Health Care in America 1999;86–101.
17. Rosenbaum PR. The role of a second control group in an observational study. Stat Sci 1987;2:292–316.
18. Shekhter I, Rosen L, Sanko J, Everett-Thomas R, Fitzpatrick M, Birnbach D. A patient safety course for preclinical medical students. Clin Teach 2012;9:376–381.
19. Breakthrough Learing. About Friday Night at the ER®. Available at: https://fridaynightattheer.com/. 2019 Accessed March 10 2020.
20. Thorton BC, Trent P, McCoy T. Enhanced systems thinking for undergraduate nursing students using Friday Night at the ER. J Nurs Educ 2018;57(11):687–689.
21. Sanko J, Gattamorta K, Young J, Durham C, Sherwood G, Dolansky M. A multi-site study demonstrates positive impacts to systems thinking using a table-top simulation experience. Nurse Educator 2020; PAP. Available at: https://journals.lww.com/nurseeducatoronline/Abstract/publishahead/A_Multisite_Study_Demonstrates_Positive_Impacts_to.99360.aspx. Accessed March 15, 2020.
22. Gardner B. Developing essential leadership skills through simulation. Organ Dev Rev 2019;51(2):58.
23. Young J. Using a role-play simulation game to promote systems thinking. J Contin Educ Nurs 2018;49(1):10–11.
24. Mckay M, Sanko J. Simulation based adverse event reporting system: development and feasibility. Clin Simul Nurs 2014;10(5):e261–e269.
25. Sanko J, Mckay M, DeSantis J, Solle N. Learning from simulation-based medication event reporting: a mixed methods analysis. Clin Simul Nurs 2015;11(6):300–308.
26. Sanko JS, Kim JY, Mckay M. Adverse event reporting following simulation encounters in accelerated and traditional BSN students. Nurse Educ Today 2018;70:34–39.
27. Benjamini Y, Yosef H. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc B Methodol 1995;57(1):289–300.
28. Keers R, Williams N, Cooke S, Ashcroft D. Causes of medication administration errors in hospitals: a systematic review of quantitative and qualitative evidence. Drug Saf 2013;36(11):1045–1067.
29. Gorgich E, Barfroshan S, Ghoreishi G, Yaghoobi M. Investigating the causes of medication errors and strategies to prevention of them from nurses and nursing student viewpoint. Global J Health Sci 2016;8(8):544–548.
30. Bertalanffy L. General System Theory; Foundations, Development, Applications. Revised ed. New York: G. Braziller; 1974:10–17.
31. Cook T, MacDougall-Davis S. Complications and failure of airway management. Br J Anaesth 2012;109(Supp l1):I68–I85.
32. Mohammed AR, Pozzessere NR, Asija A, et al. Failure to rescue: causes and educational opportunities for nursing and medical staff. Crit Care Med 2015;43(12 suppl 1):240.
33. Schulz CM, Burden A, Posner KL, et al. Frequency and type of situational awareness errors contributing to death and brain damage: a closed claims analysis. Anesthesiology 2017;127(2):326–337.
Keywords:

Adverse event reporting; system thinking; simulation; adverse events; reporting

Copyright © 2020 Society for Simulation in Healthcare