Journal Logo

Feature Articles

A Multisite Study Demonstrates Positive Impacts to Systems Thinking Using a Table-top Simulation Experience

Sanko, Jill S. PhD, MS, APRN, CHSE-A, FSSH; Gattamorta, Karina PhD, EdS; Young, Judith DNP, RN, CNE; Durham, Carol F. EdD, RN, ANEF, FAAN, FSSH; Sherwood, Gwen PhD, RN, FAAN, ANEF; Dolansky, Mary PhD, RN, FAAN

Author Information
doi: 10.1097/NNE.0000000000000817
  • Free

Abstract

Systems thinking (ST) has been defined as both a skill and an awareness. Conceptualized as a process, ST is the realization that individuals are integral parts to a system and therefore contribute to the outcomes of it.1 Conceptualized as an idea, ST is the ability to recognize, understand, and synthesize interactions and interdependencies within a set of components designed for a purpose.2,3 Systems thinking has been cited as key in improving both the safety of systems1 and systems as a whole.2 The core concepts of ST include using data for decision-making, innovating to improve systems, and collaborating effectively within a system. An example of ST in a health care context might include having an interdisciplinary team of health care providers work together with the patient to organize and collaborate to ensure that outcomes are optimal, safe, cost effective, and efficient while meeting the needs and goals of the patient and hospital system. This exemplar demonstrates ST where the components of the system include the patient, a mix of health care providers, individual departments, and interactions and interdependencies that occur between each component within the system.

Lucien Leape, a pioneer of the patient safety movement, stressed the importance of ST over a decade ago, underscoring the idea that systems cannot be improved if there is not an understanding of them.4 Systems thinking is widely included in the training and curricula of disciplines working in complex high-reliability organizations (HROs), organizations where errors can cause large-scale life safety risks, such as aviation, chemical processing, and nuclear energy production. This inclusion of ST in the education of individuals working in HROs has resulted in the reduction of errors in these areas and the attainment of a strong record of safety within them.2

With some sources estimating that more than 400 000 deaths occur each year as a result of preventable health care errors,5 most of which are tied to systems failures,6,7 the exclusion of ST in health care curricula for professionals and health care students appears irresponsible. Despite the documented safety benefits of ST identified in the literature,7,8 ST is not yet consistently integrated into all types of health care educational programs.7,9,10-12 Nursing appears to have the most ST content woven throughout their educational programs and curricula7; however, there are inconsistencies in ST content across other health care educational programs.10,13

Evidence supports simulation as an effective technique to teach and practice topics where skills and knowledge meet. Furthermore, simulation allows learners to apply learned skills in a safe, nonthreatening setting such that learners can appreciate how decisions and actions impact outcomes. Friday Night at the ER (FNER) is a commercially available table-top team-learning simulation experience used to introduce the core concepts of ST (innovation, collaboration, and using data in decision making).14,15

A few studies have explored learner impacts of FNER in nursing education16,17; however, there are gaps in knowledge about broader usages and subsequent learner outcomes. This study explored the effects of FNER on ST across a variety of health care students and academic programs using FNER in their curricula. Research questions included the following: (1) What is the overall impact of participating in FNER on ST measured by the Systems Thinking Scale (STS)? (2) Are there differences in the impact of FNER on ST moderated by gender, degree sought (graduate vs undergraduate), length of BSN degree (accelerated or traditional), or discipline studied?

Methods

This pretest-posttest study was conducted in 5 academic institutions and included graduate and prelicensure students enrolled in nursing, medicine, physical therapy, public health, psychology, and pharmacy programs (n = 961 participants). Site 1 (a private research university in the southeast United States) carried out multidiscipline sessions that included students from medicine, nursing, physical therapy, public health, and psychology. Site 2 (a private university in the southeast United States) enrolled a single discipline (pharmacy), but the students completed their sessions with the nursing program from site 1. The remaining 3 sites (a mix of public and private universities located in the mid-Atlantic and Midwest United States) had single discipline (nursing) students participating and carried out single discipline sessions.

Sample and Setting

A convenience sample was recruited from programs that had implemented FNER and expressed interest in participating. Leaders at each site (coauthors of this paper) explained the study. Participants could opt out if they did not want their data included. Each participating institution obtained institutional review board (IRB) approval to administer the STS to learners participating in FNER sessions at its institutions. The IRB at each institution independently reviewed the proposal and deemed the project exempt as an educational quality improvement study.

Instrument

The STS was used to measure the impact of participation in FNER on ST.18 The STS is a tool designed to measure the construct of ST. The STS is a 20-item self-report instrument on which participants rate each item using a 5-point scale ranging from 0 (never) to 4 (most of the time). The ratings for each item are summed to compute a total STS score, ranging from 0 to 80, where higher scores indicate better ST. Moore and Dolansky completed initial testing of the STS to establish preliminary psychometric data.18 After the first round of analysis, changes to the tool were made guided by initial psychometric test results.18 A subsequent analysis was completed. This subsequent analysis demonstrated good discriminant validity and low concurrent criterion-related validity using the Quality Improvement Knowledge Application Tool19 as the criterion measure. Scale analysis during this round of demonstrated good internal consistency (α = .82).

Scale analysis was used to examine the psychometric properties of the single-factor tool before hypothesis testing for this multi-institutional study. Measures of internal consistency reliability were found to be adequate; the Cronbach α equaled .994, and all item-total correlations were greater than 0.410.

FNER Simulation

Friday Night at the ER is a table-top simulation that has been used across the globe in a multitude of disciplines to teach ST.13,14 Despite its moniker, the ability to play and subsequently learn from the game is not dependent on having any health care knowledge. The simulation activity engages teams of 4 players at a board depicting 4 hospital departments (emergency room, surgery, step-down, and critical care) within a simulated hospital.14 Friday Night at the ER challenges teams to manage a busy hospital during a simulated 24-hour period.13,14 Each player handles patient flow and staffing needs of their department, deals with any emergencies that arise, and documents performance based on prescribed game metrics that are tracked.13 Multiple boards can be played during a single session to simulate a multihospital system environment. Sessions including more than 1 board provides an additional challenge of teams needed to collaborate with neighboring simulated hospitals. An entire FNER session begins with a prebriefing that includes how and why to play and concludes with an in-depth debrief of the lessons learned and experiences. Sessions are carried out by trained facilitators and are relatively simple to run.

Variables

The dependent variable was STS scores. Independent variables included gender, discipline studied (nursing, medicine, public health, physical therapy, psychology, pharmacy, other), BSN program length (traditional or accelerated option/second degree), and nursing degree level (BSN, RN-BSN, graduate).

Data Collection

Data were collected from all participating sites and then merged into a single database for analysis. The STS was administered within a week before and subsequent to participation in an FNER session, allowing for the analysis of the impact of the intervention. The presurvey included several demographic questions including gender, program length, and type and degree level. Each institution had trained FNER facilitators conducting the FNER sessions.

Data Analysis

Each institution submitted their data, yielding a total of 395 paired samples available for analysis (Table 1). Of note, although pharmacy students participated, there were no paired data available for analysis from this group. Total scores on the STS were calculated by summing the scores from the items included in the STS measure; these scores were then used to compare outcomes. Paired t tests and analysis of variance (ANOVA) were used to analyze the data.

Table 1 - Participant Demographics
Demographic Group N
Gender Male 84
Female 315
Discipline studied Medicine 51
Nursing 318
Physical therapy 20
Public health 2
Psychology 3
Other 1
Nursing degree level BSN 218
RN-BSN 47
Graduate 1
BSN degree length T-BSN 136
AO-BSN 129
No matched pairs available for analysis for pharmacy. Participants were not required to answer questions; therefore, the n's reported are based on data from participants who answered the demographic questions.

Results

Findings analyzing the impact of the intervention on ST demonstrated a significant increase in STS scores from pre-FNER to post-FNER conditions (t420 = −5.64, P < .001; pre-FNER STS score: mean [SD], 58.02 [9.79]; post-FNER STS score, mean [SD], 60.03 [0.32]). A small effect size (d = 0.20) was found.

Mixed ANOVAs examining the effect of exposure to the intervention (pre to post changes in ST) and between-subject variables including gender, degree sought (graduate vs undergraduate nursing), length of nursing program (traditional [T-BSN], accelerated [AO-BSN], RN-BSN), and discipline studied were used. This analysis demonstrated insignificant interaction (moderating) effects in all comparisons made (Table 2).

Table 2 - Between-Subject Interactions
Demographic Group Pre, Mean (SD) Post, Mean (SD) F df P
Gender Male 56.37 (10.25) 58.33 (10.81) 0.063 1 .802
Female 58.24 (9.62) 59.63 (10.12)
Discipline Studied Medicine 56.12 (10.40) 58.22 (11.73) 1.91 5 .092
Nursing 58.52 (9.66) 59.92 (10.14)
Physical therapy 55.20 (9.34) 59.75 (9.19)
Public health 53.50 (6.36) 66.00 (4.25)
Psychology 52.00 (10.39) 54.33 (12.90)
Othera 49.00 59.00
Nursing degree level BSN 57.76 (9.46) 59.30 (10.19) 0.739 2 .479
RN-BSN 56.17 (10.30) 57.17 (9.64)
Graduatea 66.00 59.00
BSN degree length T-BSN 55.77 (10.04) 57.48 (10.28) 0.432 1 .512
AO-BSN 59.35 (8.84) 60.46 (9.74)
No matched pairs available for analysis for pharmacy.
aSingle subject, no SD reported.

Participation in FNER was found to improve ST scores regardless of discipline studied (F1, 389 = 10.82, P < .001, ŋ2 = 0.027), gender (F1, 397 = 17.15, P < .001, ŋ2 = 0.041), or type of nursing degree (F1, 263 = 9.29, P = .003, ŋ2 = 0.034). Participation in FNER did not significantly change ST scores based on level of degree being obtained (undergraduate vs graduate), P = .557.

Analyses exploring the moderating impacts of length of BSN program demonstrated significant differences between students' scores dependent on enrollment in accelerated versus a traditional BSN program. The AO-BSN students had significantly different STS score means compared with those in T-BSN programs (AO-BSN: mean, 59.91; T-BSN: mean, 56.63; 95% confidence interval, 1.095-5.454; P = .003). Improvements in ST scores were found in both groups (AO-BSN: pre-mean, 59.36; post-mean, 60.45; P = .347; T-BSN: pre-mean, 55.78; post-mean, 57.48; P = .168). However, these were not statistically significant.

Discussion

This study demonstrates the ease of implementing the FNER simulation in a variety of health care curricula. The findings of this study were similar to others found in the literature exploring the impacts of FNER in a nursing student population16,17 but additionally demonstrated the impacts of FNER in improving knowledge of systems in a variety of other health care professions' learners.

The findings of this multisite, multidiscipline study demonstrated improvements in ST as measured by the STS regardless of the participants' degree program, gender, which undergraduate nursing degree pursued (BSN, RN-BSN), and length of nursing degree. Differences noted between the accelerated and traditional BSN students' mean scores may be attributed to differences in age, life experience, and previous degrees resulting in different baseline knowledge. The AO-BSN students in this study had attained prior college degrees, whereas the T-BSN students were more likely to be seeking their first college degree. Most sites had a mix of accelerated and traditional BSN students participating so it is unlikely that curricular differences contributed to the differences.

Limitations

This study had several limitations, including examining the impact of a single activity at a single point in time to improve ST rather than longitudinally. Few data are available to compare the results of this study with other approaches to improve ST. The study did not include a longitudinal view to examine the long-term impact to ST or the need for spaced reinforcement of concepts to assist in retention of the improvement in ST. In addition, the pretest-posttest design poses a risk for a testing effect whereas having taken the STS previously impacts the postintervention scores due to familiarity with the items. Finally, because the impact of FNER on ST was undertaken solely in a student population, it is possible that the findings are not generalizable to all postlicensure professionals; however, given the inclusion of RN-BSN students and the noted impact in this population, there is some confidence that generalizability is feasible but not certain. Strengths of the study included multiple educational institutions and health sciences, which enhanced the diversity of the study sample, the use of trained facilitators, and the use of an established tool to measure the outcome of interest.

Conclusion

The use of FNER improved ST in a wide variety of health-related majors. The demonstration of the effectiveness of participating in these simulations across a variety of health care students assists in demonstrating FNER's potential broad applicability to effect change in ST. The similarity of outcomes found in most of disciplines included in this study may demonstrate potential wide-ranging uses for FNER across many health care disciplines. Findings may additionally suggest a potential benefit of participation in FNER for health care professionals themselves. Inclusion of postlicensed students (RN-BSN) who were already working in nursing demonstrated small improvements in ST scores similar to other studies.16 This study did not include a large enough sample of RN-BSN participants to sufficiently explore the impact of FNER on ST in this population; however, these findings, taken together with the findings of a 2018 study that demonstrated improvements in ST after participation in FNER in an RN-BSN population point to the likely potential impact in licensed nursing staff. Additional studies would need to be carried out to more fully explore this notion and outcomes in other health care professionals.

Overall, the study provided additional evidence that a commercially available, relatively inexpensive, and easy-to-implement table-top simulation can make improvements in ST, a key factor found to be important optimizing in health care systems and improving outcomes. Such knowledge and skill attainment in emerging and early career health care providers can help shift the health care culture toward paying additional attention to the importance of system interactions and being more collaborative, innovative, and curious about what data trends are indicating about the efficiency and effectiveness of a system.

Building a health care workforce equipped with strong ST skills assists to change the long-held belief that safety is situated within individuals, devices, or single units of an organization to an understanding that safety is found within the system and only when system vulnerabilities are discovered.20 Individuals with an ST mindset born from ST knowledge and skills are most likely to discover system vulnerabilities21 and in turn create solutions and work collaboratively to implement them. The existence and population of systems thinkers within health care, however, is dependent on evidence-based interventions aimed at developing requisite ST skills being implemented in curricula.22 With most of health care–related errors caused by systems failures,21 ST must be included as part of the plan.20 It will be up to leaders and educators to be the promoters of this movement.

References

1. Stalter A, Phillips J, Ruggiero J, et al. A concept analysis of systems thinking. Nurs Forum. 2017;52(4):323–330.
2. Swanson RC, Cattaneo A, Bradley E, et al. Rethinking health systems strengthening: key systems thinking tools and strategies for transformational change. Health Policy Plan. 2012;27(suppl 4):Iv54–Iv61.
3. Waldman J, Schargel F. Twins in trouble (II): Systems thinking in health care and education. Total Qual Manag Bus Excellence. 2006;17(1):117–130.
4. Leape LL. Errors in medicine. Clin Chim Acta. 2009;404(1):2–5.
5. Makary MA, Daniel M. Medical error—the third leading cause of death in the US. BMJ. 2016;353.
6. Vincent C, Amalberti R. Safer Health Care: Strategies for the Real World. Cham: Springer Open; 2016. Available at http://www.oapen.org/download?type=document&docid=1001964.
7. Stalter A, Jauch A. Systems thinking education in RN-BSN programs; A regional study. Nurs Educ. 2019;44(2):112–115.
8. Vautier JF, Dechy N, Coye de Brunelis T, et al. Benefits of systems thinking for a human and organizational factors approach to safety management. Environ Systems Decisions. 2018;38(3):353–366.
9. Looman W. Teaching systems thinking for advanced family nursing practice: a theory-based tool [published online ahead of print December 27, 2019]. J Fam Nurs.
10. Clark K, Hoffman A. Educating healthcare students: strategies to teach systems thinking to prepare new healthcare graduates. J Prof Nurs. 2018;35(3):195–200.
11. Dolansky MA, Moore SM. Quality and Safety Education for Nurses (QSEN): the key is systems thinking. Online J Issues Nurs. 2013;18(3):1–1.
12. Plack MM, Goldman EF, Scott AR, et al. Systems thinking and systems-based practice across the health professions: an inquiry into definitions, teaching practices, and assessment. Teach Learn Med. 2018;30(3):242–254.
13. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–1958.
14. Breakthrough Learning. About Friday Night at the ER®. 2019. Available at https://fridaynightattheer.com/. Accessed October 15, 2019.
15. Garner B. Developing essential leadership skills through simulation. Org Dev Rev. 2019;51(2):58–59.
16. ThorNton Bacon C, Trent P, McCoy TP. Enhancing systems thinking for undergraduate nursing students using Friday Night at the ER. J Nurs Educ. 2018;57(11):687–689.
17. Young J. Using a role-play simulation game to promote systems thinking. J Contin Educ Nurs. 2018;49(1):10–11.
18. Moore SM, Dolansky MA, Singh M, et al. The Systems Thinking Scale. 2010. Available at https://case.edu/nursing/sites/case.edu.nursing/files/2018-04/STS_Manual.pdf. Accessed October 15, 2019.
19. Morrison LJ, Headrick LA, Ogrinc G, et al. The Quality Improvement Knowledge Application Tool: an instrument to assess knowledge application in practice-based learning and improvement [abstract]. J Gen Intern Med. 2003;18:S250.
20. Woods DD, Dekker S, Cook R, et al. Behind Human Error. 2nd ed. Boca Raton, FL: CRC Press LLC; 2010.
21. Sanchez JA, Lobdell KW, Moffatt-Bruce SD, et al. Investigating the causes of adverse events. Ann Thorac Surg. 2017;103(6):1693–1699.
22. DeAngelis T. Preventing medical errors. Am Psych Assoc. 2016;47(8):48.
Keywords:

Friday Night at the ER simulation; health care professions education; simulation; systems thinking; table-top simulation

Copyright © 2020 Wolters Kluwer Health, Inc. All rights reserved.