Journal Logo

Empirical Investigations

Prior Participation in Simulation Events Is Associated With Insimulation Team Performance Among Emergency Medical Services Professionals

Gurňáková, Jitka PhD; Gröpel, Peter PhD

Author Information
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: August 2019 - Volume 14 - Issue 4 - p 235-240
doi: 10.1097/SIH.0000000000000371

Abstract

Emergency medical services (EMS) professionals are expected to be competent and efficient in providing a multitude of urgent and clinically invasive interventions.1,2 One important method in EMS education is the use of high-fidelity simulation.3 A high-fidelity simulation refers to simulation experiences that are realistic and provide a high level of interactivity for the learner.4 Evidence clearly shows that simulation use significantly improves knowledge and skill performance in a number of healthcare disciplines.5–7 In prehospital care, for example, paramedic students reduced their error rates when simulation was used to instruct specific program components.8–10 This evidence provides sufficient justification to support the use of simulation in initial EMS education. However, it is less clear whether EMS professionals can also benefit from high-fidelity simulation or their accumulated clinical experience is sufficient for the quality of provided healthcare. Understanding the precursors of competency and proficiency among EMS providers is crucial to improved patient safety and better care in the prehospital settings.

Researchers have demonstrated that the amount of accumulated practice is a strong predictor of performance.11 The accumulated years of clinical practice have often been used to differentiate experts from nonexperts in emergency medicine.12,13 However, several studies indicate that high-fidelity simulation is equal to or even more important for the quality of performance than an equivalent clinical experience.14,15 This may be due to the reality that clinical learning experiences often rely on chance encounters16 and provide delayed or ambiguous feedback17 and that there is a limited number of opportunities to perform advanced care on real patients.18,19 Clinical learning experience is also limited when it comes to the low-frequency/high-difficulty events. Consequently, simulation-based learning may partly substitute for actual field experience and, thus, effectively improve performance of EMS personnel on all levels of expertise. This notion has been consistently supported among paramedic students,8–10 but there is a dearth of research on the association between high-fidelity simulation and performance among EMS professionals.

The aim of the present research is to examine prior participation in simulation events and years of experience as predictors of performance among EMS professionals. We hypothesize that both the prior participation in simulation events and the accumulated years of practice will predict performance. We further expect an interaction; we hypothesize that the relationship between prior participation in simulation events and performance will be stronger among participants with less as compared with more experience.

METHODS

Study Design

This study was a part of cross-sectional data collection during the 19th International EMS Competition “RALLYE REJVÍZ,” which is one of the largest simulation-based competitions for EMS professionals in Europe. We followed teams of EMS providers and tested whether prior participation in simulation events and years of EMS experience played a role in how the teams performed in the simulation-based competition. Hence, insimulation performance was the outcome variable. We conducted the study on May 29 and 30, 2015. Ethical approval for the study was secured in line with the authors' institutional procedures.

Study Setting

The Rallye Rejvíz competition is annually organized in Kouty nad Desnou, Czech Republic. The competition consisted of 11 simulated, near-field tasks that were set in natural environment (eg, in forest, at school, in a pub) in the town and surrounding communities. Table 1 gives a short overview of the tasks (see PDF, Supplemental Digital Content 1, which contains all tasks explained in detail; http://links.lww.com/SIH/A420). The EMS teams used their own ambulances (fully equipped) for transportation between the task locations.

TABLE 1
TABLE 1:
Overview of the Tasks

Study Participants

Participants were Slovak and Czech EMS professionals. Fifty-one paramedic and physician teams comprising 3 or 4 team members each (159 persons in total) competed in the Rallye Rejvíz competition. Paramedic teams only consisted of paramedics, whereas physician teams consisted of a physician (mostly an anesthesiologist) and 2 or 3 paramedics.

Data Collection

In cooperation with the Rallye Rejvíz organizers, the teams were invited to participate in the study after they registered online for the competition and a link to the online survey was sent to each team member. After participants logged into the survey, they were briefed about the study and provided informed consent. Participants were then asked to list their age, sex, years of experience, prior participation in simulation events, and complete a couple of short questionnaires unrelated to the present study. The participants' experience was measured with 2 questions: “How many years have you worked in EMS?” for the years of EMS practice and “How many years have you worked in the healthcare sector (incl. EMS)?” for the years of overall healthcare practice (HC practice). The prior participation in simulation events was measured with the question “How many simulation-based exercises or events have you attended so far?” Participants received a small gift (USB flash drive) in exchange for their participation.

Simulation Cases and Performance Assessment

Emergency medical services insimulation performance was measured at the team level during the competition, which consisted of 11 simulated, near-field tasks with different levels of difficulty (Table 1). The tasks were designed by EMS instructors, performed within predefined time limits, and modeled just as close to actual situations as possible. Trained actors served as patients. Performance in each task was rated by at least 2 expert judges on the quality of examination, treatment, team cooperation, and communication (see PDF, Supplemental Digital Content 2, which describes performance assessment in detail; http://links.lww.com/SIH/A421). The judges did not rate independently but were asked to discuss and score each performance together, because each of them rated different skills within a simulation task. All judges were active physicians or paramedics with extensive experience in rating clinical performance in their field. They had received rater training and were provided with details about the case they would assess along with explicit performance expectations and a standardized rating form. The total performance score across all tasks was used as the dependent variable; higher score represented better insimulation performance.

Data Analysis

To control for sex differences, we conducted separate t tests between men and women on the reported years of HC practice, years of EMS practice, and prior participation in simulation events. To ensure that insimulation team performance would not be affected by the proportion of women within teams, we computed proportional score by averaging participants' sex within teams and then correlated the score with insimulation team performance. To ensure that insimulation team performance would not be affected by the number of team members and the type of team, we compared insimulation team performance of the 3- and 4-member teams and of the physician and paramedic teams with separate t tests. To test whether there was enough variance in the predictor variables at the team level, which would allow for a meaningful testing of the study hypotheses, multilevel linear models were used because of the nested data structure, with individual team member score being nested within each team. In particular, we computed a random intercept-only model separately for each predictor variable and expected significant variance in intercepts across teams.

Because insimulation performance was measured at the team level, team was the unit of analysis for testing the study hypotheses. Therefore, individual simulation and experience scores had to be converted into a measure that represented team composition in terms of prior participation in simulation events and EMS experience. The most common methods of aggregation are the mean score of team members, the lowest member score (minimum method), the highest member score (maximum method), and the spread of team member scores (variance method).20 The mean and minimum methods reflect high levels of a trait across the team, whereas the variance method reflects variability of the trait within the team. We thus aggregated individual-level predictor scores into team-level mean, standard deviation, minimum, and maximum scores, and then correlated the aggregated scores with insimulation team performance.

Finally, to test the interaction hypothesis, hierarchical regression analyses were conducted on insimulation team performance separately for each aggregation method. Team-level experience (years of HC practice or years of EMS practice) and prior participation in simulation events were entered as the first step, and their interaction term was entered as the second. In all analyses, predictor variables were standardized before their interaction term was calculated. Two regression statistics, the change in determination coefficient (ΔR2) and standardized β weights, were used to interpret the results. ΔR2 indicates the incremental increase in the proportion of the variance in the dependent variable that is predicted from predictor variables at each step. A standardized β weight indicates the strength of the effect of each predictor variable to the dependent variable. All statistical analyses were performed using SPSS 24.0 (IBM Corp, Armonk, NY). The level of significance was set at P < 0.05 (two-tailed).

RESULTS

Participants' Characteristics

Completed surveys were received from 120 participants for a response rate of 75% of the total population. Participants' characteristics are presented in Table 2. Nonresponders (n = 39, 72% male, Mage = 33.1 years) did not differ significantly from responders in either age or the proportion of women (sex and age of the nonresponders were available from the online registration system). At the team level, data of all 51 teams were included in the analysis because at least one member of each team completed the survey.

TABLE 2
TABLE 2:
Participants' Characteristics

Preliminary Analyses

The mean insimulation team performance score was 10,133.24 (SD = 987.99, range = 8221–12,145), which was 73% of the possible maximum score (13,830). There were no significant differences between 3-member teams and 4-member teams in insimulation team performance (Table 3). Similarly, physician and paramedic teams did not differ in insimulation team performance. We thus did not control for team composition and the type of team in further analyses. Furthermore, there were no significant differences between men and women either in the years of practice or in the prior participation in simulation events. Higher proportion of women within a team did not correlate with insimulation team performance. Therefore, we do not discuss sex any further.

TABLE 3
TABLE 3:
Team Composition, Type of Team, and Performance

We further analyzed variance in predictor variables. Did teams differ, on average, in the years of HC and EMS practice and in how many simulation events they had participated? Participants' years of HC practice did not show significant variance in intercepts across teams, Var(u0j) = 1.89, Z = 0.45, P = 0.65. In contrast, the years of EMS practice and the prior participation in simulation events showed significant variance in intercepts across teams, Var(u0j) = 7.27, Z = 2.27, P = 0.02, and Var(u0j) = 6.80, Z = 2.71, P = 0.01, respectively.

Participation in Simulation Events, Experience, and Team Performance

Table 4 presents correlations among team-level predictors and insimulation team performance. Prior participation in simulation events significantly and positively correlated with the performance. The more frequently team members had participated in simulation events, the better team performance was in the competition. In contrast, neither the years of HC practice nor the years of EMS practice were significantly correlated with insimulation team performance.

TABLE 4
TABLE 4:
Means, Standard Deviations, and Correlations Between Team-Level Predictor Variables and Insimulation Team Performance

We further tested an interaction between years of practice and prior participation in simulation events. Table 5 summarizes the results of hierarchical regression analyses for each aggregation method. Mirroring the correlational results, the findings showed significant main effects of prior participation in simulation events for each aggregation method, but no significant main effects of either the years of HC practice or the years of EMS practice. Furthermore, there were no interaction effects, as demonstrated by nonsignificant changes in variance accounted for in step 2.

TABLE 5
TABLE 5:
Summary of Hierarchical Regression Analyses Testing the Interaction Effect Between Years of Practice and Simulation Use on Insimulation Team Performance

DISCUSSION

The aim of this study was to test whether prior participation in simulation events and accumulated years of practice predict performance among EMS professionals. We found a significant effect of prior participation in simulation events on insimulation team performance. Teams with EMS professionals who reported having participated in simulation events more frequently performed better in the competition. In contrast, neither the years of EMS practice nor the years of healthcare practice predicted insimulation performance. Teams with more experienced team members (in terms of accumulated years of practice) did not perform significantly better than teams with less experienced team members. Furthermore, there was no interaction between prior participation in simulation events and years of practice, indicating that the participation in simulation events may be helpful regardless of participants' experience. Even teams with highly experienced members benefited from prior participation in simulation events.

The present results extend prior research on simulation use among paramedics by showing that benefits of high-fidelity simulation are not limited to initial EMS education8–10 but spread also to experienced professionals. This is most likely because EMS professionals also need to learn, because the work of EMS is becoming more complex and new procedures are constantly being developed.2,21 Simulation use provides the opportunity to practice important skills and learn new procedures while receiving performance feedback. Moreover, simulation use enables individuals to improve their skills without subjecting actual patients to risk, allows standardization and consistent replication of patient conditions, and overcomes field limitations such as low-frequency/high-difficulty encounters and delayed or completely missing feedback.3 Indeed, being confronted with a situation that challenges one's skill and for which the performer receives clear feedback results in skill improvement and expertise.22 Moreover, a realistic environment and a rapid feedback are necessary conditions for the development of skilled intuition, which in turn helps professionals make appropriate decisions.17

The effect of prior participation in simulation events was consistent in our data considering that all 4 aggregation methods yielded significant results. This suggests that participation in simulation events may have both supplementary and complementary role in insimulation team performance.20 It is generally advantageous for performance when all team members participate in simulation events frequently, as indicated by the mean and minimum methods. However, an infrequent participation in simulation events of a single team member need not be a problem if the team also includes an individual with numerous participations in simulation events able to compensate for the less experienced colleague, as indicated by the variance and maximum methods. This conclusion should be nevertheless taken with caution as the consistently significant results with all aggregation methods may also come from the rather small team size (mostly 3-member teams).

Regarding years of practice, the data did not support our hypothesis. Neither the years of EMS practice nor the years of healthcare practice predicted insimulation performance. This might seem surprising given that researchers repeatedly found performance being dependent on the accumulated years of practice.11,22 However, a closer investigation of the practice-performance relationship revealed that this relationship especially holds for less skilled individuals; the accumulated years of practice explained only 1% of variance in performance among elite performers.23 The implication is that the amount of accumulated practice can help EMS personnel gain expertise especially during their initial education and early career, but the impact of accumulated practice may flatten out for common and typical skills after the personnel become professionals. Specific practice for low-frequency/high-difficulty encounters is not necessary subsumed within usual accumulated practice.

We further hypothesized an interaction between years of practice and prior participation in simulation events. We expected that the relationship between participation in simulation events and insimulation performance will be stronger for those with fewer years of practice, but we did not find evidence for it. This implies, as noted earlier, that the participation in simulation events is beneficial for EMS performance regardless of how long the participating EMS professionals work in the field. The previous results regarding years of practice especially hold for the years of EMS practice. Caution should be used when interpreting the results for the years of healthcare practice because the multilevel analysis yielded a nonsignificant variance in intercepts across team. Thus, at the team level, our participants were similarly experienced in terms of years of healthcare practice, which might weaken the relationship between healthcare practice and insimulation performance.

Although not the focus of the present study, we observed that 3-person teams performed equivalently to 4-person teams. Given that personnel are expensive, this seems like a worthwhile area for future research. In addition, we found no difference in insimulation performance between physician and paramedic teams. This may add to the research on the controversy about whether paramedic or physician teams can provide better medical care in the prehospital setting.24 Our results support the view that both paramedic and physician teams may provide equally good care.

Limitations

Although testing performance in the simulation-based competition has many strengths, such as objectively measured insimulation performance, standardization, and a broad range of skills tested, it also has some limitations. First, the study did not access real-world provider performance. We cannot conclude that simulation use improves the participants' clinical performance. Second, because we used a cross-sectional study design, no causal conclusions can be drawn. Prospective longitudinal studies are necessary in this regard. Third, the judges were not blinded as to who the competitors were, which might have impacted the scoring. Fourth, the Rallye Rejvíz competition is a prestigious event with a limited number of participating teams and thus our sample presumably included highly motivated individuals. We can assume that individuals who are motivated to compete in EMS events also strive to take more advantage from simulation events, which might exaggerate the role of simulation in our study. Fifth, only 75% of all competition participants completed the survey, which limits the reliability of the team-level predictors. However, the study results did not change substantially when only “all-responders” teams (45% of teams) were analyzed. Sixth, we did not collect data on call volumes or patients per year, which might better reflect participants' experience than mere years of practice, and on team practice for the sake of the competition, which might strengthen the effect of simulation use. Finally, it is possible that participants who had participated in simulation events more frequently were more accustomed to simulation setting and thus received higher scores in the simulation-based competition than participants with fewer participations. Although the acclimatization effect cannot be ruled out and may limit our results, it is not necessarily a disadvantage, because the familiarity with simulated environments can reduce stress on the scene. The simulation cases in our study were highly varied and had not been used in prior events, but teams that had been through a greater number of prior simulation events were able to more competently apply current guidelines to these new and varied scenarios. We may propose that this was partly due to the improved stress management on the scene. However, we also propose that high-fidelity simulation has a number of advantages that go far beyond acclimatization: (a) simulation helps develop routines and automatize safety behaviors,3 which has been linked to intuitive, rapid decisional processes25; (b) simulation helps reduce uncertainty in the field26; and (c), as noted earlier, participants of a simulation event typically receive feedback on their performance along with the explanation of how to treat the case according to the most recent guidelines, which help them to stay up-to-date with novel procedures.

CONCLUSIONS

The study results indicate that prior participation in simulation events is associated with further insimulation performance among EMS professionals. The accumulated years of practice seem to play only a weak role and do not interact with prior participation in simulation events in predicting insimulation performance. Hence, our findings support the idea that simulation use is not limited to initial EMS education, but it may be a helpful educational method regardless of participants' years of practice. Future research should examine whether this also translates into better clinical performance.

REFERENCES

1. National Registry of EMTs: 2004 National EMS Practice Analysis. Columbus, OH: NREMT; 2005.
2. Studnek JR, Fernandez AR, Shimberg B, Garifo M, Correll M. The association between emergency medical services field performance assessed by high-fidelity simulation and the cognitive knowledge of practicing paramedics. Acad Emerg Med 2011;18(11):1177–1185.
3. McKenna KD, Carhart E, Bercher D, Spain A, Todaro J, Freel J. Simulation Use in Paramedic Education Research (SUPER): a descriptive study. Prehosp Emerg Care 2015;19(3):432–440.
4. Lopreiato JO, Downing D, Gammon W, Lioce L, Sittner B, Slot V, Spain AE, and the Terminology & Concepts Working Group: Healthcare Simulation Dictionary. SSH, 2016. Available at: http://www.ssih.org/dictionary. Accessed September 2, 2018.
5. Cook DA, Hatala R, Brydges R, Zendejas B, Szosteck JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011;306:978–988.
6. Owei L, Neylan CJ, Rao R, Caskey RC, Morris JB, Sensenig R, et al. In situ operating room–based simulation: a review. J Surg Educ 2017;74(4):579–588.
7. Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg 2008;248(2):166–179.
8. Wyatt A, Fallows B, Archer F. Do clinical simulations using a human patient simulator in the education of paramedics in trauma care reduce error rates in preclinical performance? Prehosp Emerg Care 2004;8:435–436.
9. Gordon DL, Issenberg SB, Gordon MS, LaCombe D, McGaghie WC, Petrusa ER. Stroke training of prehospital providers: an example of simulation-enhanced blended learning and evaluation. Med Teach 2005;27:114–121.
10. Hall RE, Plant JR, Bands CJ, Wall AR, Kang J, Hall CA. Human patient simulation is effective for teaching paramedic students endotracheal intubation. Acad EmergMed 2005;12:850–855.
11. Macnamara BN, Hambrick DZ, Oswald FL. Deliberate practice and performance in music, games, sports, education, and professions: a meta-analysis. Psychol Sci 2014;25(8):1608–1618.
12. McCormack C, Wiggins MW, Loveday T, Festa M. Expert and competent non-expert visual cues during simulated diagnosis in intensive care. Front Psychol 2014;5:949.
13. Smith MW, Bentley MA, Fernandez AR, Gibson G, Schweikhart SB, Woods DD. Performance of experienced versus less experienced paramedics in managing challenging scenarios: a cognitive task analysis study. Ann Emerg Med 2013;62(4):367–379.
14. Hayden JK, Smiley RA, Alexander M, Kardong-Edgren S, Jeffries MR. The NCSBN National Simulation Study: a longitudinal, randomized, controlled study replacing clinical hours with simulation in prelicensure nursing education. J Nurs Regul 2014;5(Suppl 2):S1–S64.
15. Watson K, Wright A, Morris N, McMeeken J, Rivett D, Blackstock F, et al. Can simulation replace part of clinical time? Two parallel randomised controlled trials. Med Educ 2012;46:657–667.
16. Motola I, Devine LA, Chung HS, Sullivan JE, Issenberg SB. Simulation in healthcare education: a best evidence practical guide. AMEE Guide No. 82. Med Teach 2013;35:e1511–e1530.
17. Kahneman D, Klein G. Conditions for intuitive expertise: a failure to disagree. Am Psychol 2009;64(6):515–526.
18. Westergard BC, Peterson BK, Salzman JG, Anderson R, Buldra M, Burnett AM. Longitudinal and regional trends in paramedic student exposure to advanced airway placement: 2001-2011. Prehosp Emerg Care 2013;17:379–385.
19. Johnston B, Seitz SR, Wang HE. Limited opportunities for paramedic student endotracheal intubation training in the operating room. Acad Emerg Med 2006;13:1051–1055.
20. Prewett MS, Walvoord AA, Stilson FR, Rossi ME, Brannick MT. The team personality–team performance relationship revisited: the impact of criterion choice, pattern of workflow, and method of aggregation. Hum Perform 2009;22(4):273–296.
21. Ball L. Setting the scene for the paramedic in primary care: a review of the literature. Emerg Med J 2005;22:896–900.
22. Ericsson KA, Krampe RT, Tesch-Römer C. The role of deliberate practice in the acquisition of expert performance. Psychol Rev 1993;100(3):363–406.
23. Macnamara BN, Moreau D, Hambrick DZ. The relationship between deliberate practice and performance in sports: a meta-analysis. Perspect Psychol Sci 2016;11(3):333–350.
24. Timmermann A, Russo SG, Hollmann MW. Paramedic versus emergency physician emergency medical service: role of the anaesthesiologist and the European versus the Anglo-American concept. Curr Opin Anaesthesiol 2008;21(2):222–227.
25. Sedlár M. Situation assessment and decision making strategies of emergency medical services physicians in routine and non-routine situations. Studia Psychologica 2017;59(2):127–138.
26. Harenčárová H. Managing uncertainty in paramedics' decision making. J Cogn Eng Decis Mak 2017;11(1):42–62.
Keywords:

Simulation; emergency medical services; paramedic; education; practice

Supplemental Digital Content

Copyright © 2019 Society for Simulation in Healthcare