In critically ill or injured patients, early identification and treatment of life-threatening conditions improves survival.1,2 By recreating common and rare events in a simulated environment, health care professionals can learn how to manage many of these critical events in a controlled setting.3,4 In simulation studies designed to improve teamwork skills, participants are expected to follow Advanced Cardiac Life Support, Pediatric Advanced Life Support, or Advanced Trauma Life Support (ATLS) algorithms to rapidly manage a life-threatening crisis.5–10 Simulation is also used to help physicians acquire a variety of inferential and deductive skills needed to recognize and manage complex clinical conditions.11–16 Health care professionals typically learn to recognize abnormal patient trajectories and rapidly diagnose conditions using heuristics.17 By learning to recognize these patterns in a simulation setting, physicians are trained to diagnose and treat potentially adverse events before irreversible organ damage or cardiopulmonary arrest occur.11–16
Trauma teams frequently use heuristics to diagnose injuries and then follow a set of well-developed algorithms to manage the injured patient. For example, using a trauma heuristic, an injured patient with hypotension is assumed to be hypovolemic as a result of blood loss. In trauma settings, this approach has effectively reduced the morbidity and mortality from trauma, the leading cause of death in individuals younger than 45 years18; however, in some patients, these diagnostic shortcuts can lead to an incorrect diagnosis and inappropriate treatment. The resulting delay or missed diagnosis can result in an adverse patient outcome. Simulation is advocated as a method to provide physicians with prospective experiences to recognize when these rapid diagnoses need to be revisited, and a more analytic approach to diagnosis is needed.19,20 This approach could help physicians to develop a more analytic approach to diagnosis, more rapidly develop expertise, and learn how to avoid some of the overconfidence frequently associated with missed and delayed diagnosis.21–28 Few, if any, studies provide a methodology to provide physicians with simulation experiences in managing scenarios that require an analytic approach to diagnosis. By recreating clinical settings where a heuristic approach had led to a delayed or missed diagnosis in a trauma setting, simulation could be used to provide teams with prospective experiences in how to recognize the error. These experiences could be helpful in improving diagnostic skills and encouraging teams to recognize the limitations of heuristic approaches and when to rely on a more analytic approach to trauma diagnosis and management. Ultimately, this methodology simulation could be used to improve decision-making skills, develop strategies to improve diagnostic reasoning, as well as study the cognitive process that teams use to recognize and manage settings associated with a missed or delayed diagnosis. In high-acuity settings, the consequences of diagnostic errors that result in a delay or missed diagnosis are more likely to impact survival.
The objectives of this study were (a) to construct a multiple-scenario assessment that included trauma scenarios that required teams to apply an analytic approach to diagnosis because of a preexisting condition or additional injury, (b) to determine whether simulation can be used to provide teams with a broader experience in trauma management that required heuristic and analytic skills to diagnosis, and (c) to evaluate how teams and team leaders recognize and manage scenarios that require both heuristic and more analytic diagnostic and management approaches.
Scenario Content and Construct
Ten trauma events were selected for the design and development of simulation scenarios. We selected scenario content based on a range of traumatic events encountered in patients presenting at our level 1 trauma center. The scenarios were designed to include constructs that would require teams to accomplish specific actions and skills and demonstrate competencies that clinicians are expected to possess in caring for critically ill and injured patients. Seven of the 10 scenarios were designed to follow well-recognized patterns of injury; the patient could be effectively managed when teams followed ATLS-recommended algorithms. To successfully manage these 7 scenarios, teams were expected to have skills in immediate stabilization and knowledge about the type of injury (eg, penetrating and nonpenetrating, anatomic location, mechanism of injury). Teams were supposed to assess the injury, select and interpret diagnostic studies, interpret imaging and x-ray findings, and provide the appropriate therapeutic intervention. The 3 additional scenarios required similar knowledge of ATLS algorithms but were constructed to include injuries that did not respond to the initial ATLS-recommended therapy. The scenarios included (1) a motor vehicle accident patient who had a splenic injury and an associated spinal cord injury, (2) preinjury myocardial ischemia and associated chest trauma following a single vehicle motor accident, and (3) chest trauma with rib fractures and tracheobronchial injury (Table 1). The failure to respond to therapy occurred primarily because of the magnitude of the injury (chest trauma with tracheobronchial rupture) or because of a coexisting disease or injury (myocardial infarction and spinal cord injury). These scenarios were constructed based on the findings of trauma patients who had presented with similar conditions, and their diagnosis was not recognized until later in their hospital care. Trauma resuscitation teams were required to demonstrate an awareness that the patient was not responding to therapy and to consider additional diagnoses and associated treatments.
The institutional review board at Washington University approved the study, and written informed consent was obtained from all the participants. The study cohort included emergency medicine, surgery, and anesthesia residents enrolled at the Washington University/BJC consortium residency programs. The 21 trauma teams included a total of 83 individuals. There were 24 postgraduate year 1 (PGY-1) (30.0%), 16 PGY-2 (20.0%), 22 PGY-3 (27.5%), 14 PGY-4 (17.5%), and 4 PGY-5 (5.0%) participants. Based on specialty, most of the participants were surgeons (n = 42, 52.5%), followed by emergency physicians (n = 33, 41.3%), and anesthesiologists (n = 5, 6.3%). Each team included between 3 and 5 practitioners who participated in a 2-hour simulation session scheduled during a 9-month period. All of the participants had previous experiences with simulation during their residency training program, and all had participated as members of a trauma resuscitation team.
Scenario Protocol and Process
After an introduction to the purpose of the research project, the residents formed a trauma team to manage 7 scenarios including an introductory scenario (femur fracture) followed by 4 heuristic and 2 analytic scenarios that were randomly presented to the teams. The analytic scenarios did not respond to the standard ATLS trauma algorithm and were associated with a secondary diagnosis that required teams to reconsider their initial diagnosis and implement a different therapeutic plan. The team leader role was rotated among the emergency medicine and surgery residents. The majority of residents served as team leader for 2 scenarios. The role of the anesthesia residents was to assess and manage the patient’s airway. The team leader assigned the other team members trauma responsibilities (assessment, airway management, procedures) for each scenario. After completion of the introductory scenario, a physician member of the investigative team debriefed the teams.
For each of the scenarios, a “hand off” occurred between the trauma team and a scripted confederate serving as a paramedic who provided the relevant history and changes in status that occurred en route to the trauma bay. The scenarios frequently included progressive deterioration in the patient’s (mannequin’s) vital signs that would continue unless the team actively managed the simulated patient’s condition. In addition to hemodynamic, respiratory, and neurologic signs and symptoms that the trauma teams could assess during the scenario, the simulated patient (mannequin), if conscious, could provide verbal responses via a microphone in the control room. The responses to common questions as well as general inquiries that the team members were likely to ask were scripted to consistently provide the teams with similar information. For each scenario, various imaging as well as laboratory and electrocardiographic data, when requested, were made available to the team. The availability of this information was based on “typical” response times when these diagnostic studies are ordered in our level 1 trauma unit.
For the majority of the scenarios, the administration ended when the trauma team stabilized the patient or reached a decision about surgery, additional studies or treatment (ie, angiography, embolization, magnetic resonance imaging), or transfer to a surgical intensive care unit for additional therapy. Teams were able to initiate the resuscitation and determine a next step following the trauma bay management in approximately 6 or 7 minutes. Frequently, in scenarios associated with continued deterioration, the trauma team decided to operate. This signaled the end of the scenario.
The evaluation was conducted in a 2-hour training session at the Howard & Joyce Wood Simulation Center. The scenarios were presented to the participants in 1 of our 4 simulation suites (20 × 20); a high-fidelity mannequin was used to simulate the trauma patient (METI HPS or METI Man). The mannequin’s vital signs were presented on a Phillips monitor with additional trauma equipment including Alaris medfusion pumps including a defibrillator with “crash” cart. Each performance was videotaped for scoring purposes. Two or more of the investigators were present for each of the trauma sessions.
Once the scenarios were selected, faculty from trauma surgery, critical care, and emergency medicine reviewed the content and described performance expectations for trauma teams who would manage these types of patients. Based on this input, a key action item checklist of 8 to 12 scoring items was constructed for each scenario and used, independently, by 2 raters to score each team. A global score was also assigned to the trauma team’s performance for each scenario with 1 to 3 indicating unsatisfactory, 4 to 6 indicating satisfactory, and 7 to 9 indicating a superior performance. The global score was assigned, independently, by each of the 2 raters after the key action checklist was completed. Based on the scenario-specific checklists, a “final” scoring item was selected for each scenario. This final action was used to indicate that the team had successfully diagnosed and would implement therapy that would resolve the crisis in the trauma bay. For example, the final action for the initial scenario, a previously healthy patient with a fractured femur was to reassess the distal extremity pulse and assure effective perfusion following closed reduction of the femur (Table 1). For each scenario, all checklist items including the final item scores were scored as 0 (did not get credit for the action) or 1 (received credit for the action).
The raters included board-certified surgeons as well as an emergency department nurse. All of the raters participated in an orientation session that provided information regarding the purpose of the assessment and the scoring criteria. The criteria expected for a team to receive credit for a checklist item was reviewed for each of the scenarios. Likewise, for the global score, the performance criteria to assign unsatisfactory, satisfactory, or superior ratings were discussed. Two raters independently scored each of the scenarios. An analytic total score, by rater and scenario, was calculated as a percentage of checklist items credited. Individual scenario team scores were the average of the 2 raters’ scores (for both the checklist, including the definitive action, and global rating). An overall score for each team, both for the checklist and global metrics, was calculated as the average of the scenario scores.
Generalizability studies were conducted to estimate the reliability of the trauma team technical performance measures (checklist, global rating). In addition, interrater reliability of the technical measures was calculated at the scenario level. For the definitive actions, the number of rater disagreements was tallied.
Descriptive statistics (mean, SD) were used to summarize performance at the scenario level. Based on missing demographic data for 3 participants, the team scores for 7 encounters were not included in the analysis. In addition, the performance measures for the advanced and complex injury scenarios (checklist, global rating) were summarized by the training level of the team leader (junior, PGY-1 and PGY-2; senior, PGY-3, PGY-4, and PGY-5). To investigate if performance varied as a function of the training level of the leader (junior, senior) and the type of scenario (heuristic vs. analytic), a 2-way analysis of variance was conducted. This was performed separately for the checklist and global rating scores. For the definitive actions, a χ2 test was used to test whether performance varied as a function of scenario type (advanced vs. complex).
Reliability of Scenario Scoring
The 21 teams participated in 7 simulation scenarios, yielding 147 total encounters. For the 140 scored scenarios with complete data sets, the reliability of the team checklist scores based on a 7-scenario assessment was 0.75. The reliability of the global team performance ratings based on a 7-scenario assessment was 0.80. The average interrater reliability of the checklist scores was 0.80; the average interrater reliability of the global scores was 0.46. The 2 raters disagreed on whether the decisive action was done for 6 (4.3%) of the 140 scenarios (Fig. 1).
Overall, the complex scenarios were more challenging (ie, teams obtained lower scores) relative to scenarios that had only a single injury (Table 2). In the scenario that simulated tracheobronchial disruption, for example, teams rarely recognized the need for an alternative method of ventilation despite worsening hypoxia. For the final actions, across all scenarios, there was a significant association between achievement (obtaining credit or not) and scenario type (heuristic injury/analytic injury) (P < 0.01). The teams were less likely to accomplish the decisive action items on the complex scenarios (53.0% vs. 83.2%, P < 0.05). The teams required 435 (35) seconds (range, 259–679 seconds) to complete the scenario as described in the Methods section.
Performance by Level of Training and Scenario Type
The 2-way analysis of variance (training level, scenario type—single vs. complex injury scenarios) was conducted separately for both the checklist and global scores. The analysis yielded a significant effect based on the type of scenario (F = 16.9, P ≤ 0.01). Averaged over resident groups (experience level of the team leader), performance on the single injury scenarios was significantly better than on the complex trauma scenarios, for both the checklist and global scores, regardless of experience level of the team leader. For the heuristic trauma scenarios, teams who had leaders with more training and experience received higher checklist and global scores. However, teams with less experienced leaders received higher global scores on the complex scenarios (Table 3). For the analytic scenarios, there were no significant differences in checklist performance as a function of the experience level of the team leader.
This study indicates that simulation can be used to model scenarios that include heuristic approaches to diagnosis as well as scenarios that require a more analytic approach by teams. The analytic scenarios required teams to reevaluate a diagnosis and alter their management as additional information was gathered during the trauma resuscitation. Teams applied a heuristic diagnostic approach for all of the scenarios including the scenarios that this approach led to an incorrect treatment (fluid and blood in myocardial infarction and spinal cord injury). As expected, teams were more effective at managing a trauma injury that followed heuristics and responded to a treatment regimen based on ATLS algorithms. For scenarios that required teams to apply a more analytic approach to diagnosis and recognize the potential for an additional injury or preexisting condition that did not respond to an initial resuscitation, the majority of teams recognized the failure to respond to therapy, but some of the teams were not able to determine the additional injury. Many teams were able to diagnose the preexisting or second injury but did not change their therapy, leading to a continuing delay in treatment that in practice would potentially alter patient outcome (eg, cardiology consult for myocardial ischemia, ventilation strategy for ruptured bronchus, inotropic therapy for spinal shock).
Our goal in designing the assessment was to provide trauma teams with experiences in managing scenarios that could be managed using heuristic as well as scenarios that required a more analytic approach. In the majority of studies, including our own, scenarios are designed to follow a recognized heuristic pattern and participants rarely, if ever, need to reevaluate their diagnosis or monitor the effectiveness of treatment. Although individuals and teams developed skill and experience in managing critical events, this previous curriculum design did not provide experiences in how these heuristic approaches can result in a missed or delayed diagnosis. The scenarios that required analytic approaches required teams to reevaluate their initial rapid diagnosis and use an analytic approach to effectively manage the scenario. Many experts believe that the most promising strategies to develop diagnostic skills are to provide experiences and feedback in high-fidelity settings particularly when time pressure often leads to an overreliance on heuristics.20–22 For this investigation, scenarios were designed to provide teams with experience in applying both nonanalytic (heuristic) and analytic diagnostic strategies. Our goal was to provide teams with experience in recognizing the limitations of heuristic approaches and the need to reevaluate and to apply an analytic approach. A combination of scenarios that promote decisions based on heuristics as well as decisions requiring more advanced clinical reasoning could be used to study decision making, develop better strategies to help physicians in settings that require rapid diagnosis and management, and avoid the potential for delays in diagnosis or missed diagnoses. A similar simulation methodology could be applied to a variety of other settings, where practitioners not only frequently apply heuristic decision-making methods to make a rapid diagnosis but also need to be able to recognize and apply an analytic approach when indicated.
We anticipated that training and experience would lead to better management on both types of scenarios. However, teams led by residents with more training and experience only received higher scores on those scenarios with a single injury, ones that would be more easily diagnosed by following a recommended ATLS algorithm. Unexpectedly, the teams led by more junior residents received higher scores in the complex scenarios. There are a variety of potential explanations for this finding, including the fact that more experienced team leaders may be more familiar and potentially more reliant on heuristic diagnostic approaches. The emphasis on rapid diagnosis and prompt treatment may have led these more experienced leaders to demonstrate overconfidence in their diagnostic acumen. An additional explanation may be that the junior team leaders’ lack of confidence might have encouraged other more experienced team members to take a more active role in the analysis of the clinical findings, leading to better management. Finally, perhaps, the portrayal of these more complex, analytic scenarios might be less representative of trauma settings compared with the simpler scenarios that could be diagnosed with heuristics. Our findings indicate a need for additional research, particularly studies that explore, through debriefing or “think aloud” exercises, how teams made their diagnostic decisions.29,30 As outlined in a recent editorial, simulation education, particularly training for the more advanced practitioner, should include more complex scenarios that provide participants the experience in a variety of adverse events, including death.31 Combining these types of scenarios with tools that are specifically tailored to the assessment of clinical decision making should yield valuable data that can guide future educational activities, potentially having a positive impact on patient safety.
In many critical patient care settings such as trauma, the emphasis is on rapid decisions made primarily via heuristic approaches. Simulation can provide practitioners with prospective experiences in managing scenarios that include heuristic as well as analytic diagnostic approaches. Simulation avoids many of the recognized limitations and biases associated with post hoc review of conditions associated with delayed or missed diagnosis. In clinical settings, even if errors could be identified rapidly and the need for retrospective review could be eliminated, the pace of high-acuity care would limit opportunities to provide education about these types of mistakes. The retrospective methods currently used to learn about and study decision making and how a missed or delayed diagnosis may occur are often not able to identify the cognitive processes that the teams used when deciding on therapy.29,30 Simulation could be used to study and better understand the cognitive processes that can either contribute to or help teams avoid or correct errors. In the future, studies would need to include additional assessments that measure the communication and cognitive processes that teams use to manage critical events and determine how these dynamics differ when teams deal with patient conditions that required more advanced diagnostic reasoning.
Our assessment methodology has some limitations. First, we only assessed trauma teams and specifically used scenarios where the protocols guiding resuscitations and the teamwork and communication expectations during early trauma care are well established. Although we believe this approach generalizes to other high-acuity settings, additional studies, in other patient care sites, are needed to determine whether a similar simulation-based educational approach is warranted. Second, although our goal was to allow practitioners to use more advanced decision-making skills, constructing and administering scenarios that require these cognitive skills are more difficult and require input from experts and pilot testing to assure that the scenarios reflect the findings from the clinical setting. To the extent that these types of patient presentations are unique, context specificity may undermine the reliability of the assessment process and, to some extent, confound the interpretation of the findings. Third, in scoring more complex trauma scenarios, the logical, inferential, and sequential steps in managing the patient are more difficult to capture using traditional metrics. For example, in one of our scenarios (ie, blunt trauma in pregnancy), the experts did not agree about the priorities in management, and as a result, interrater reliability was low for the global measure of team performance. Fourth, we did not categorize the types of cognitive errors that occurred during the simulation. A set of metrics that more effectively capture the decision-making process is needed for these types of scenarios. Further studies are needed, perhaps those that include more extensive debriefing, to determine the various causes of error and to determine why some teams, particularly teams led by junior residents, were more successful than others in managing more complex scenarios. Finally, as with most simulation studies of this nature, team performance may differ in the clinical setting. However, there is increasing evidence that educational interventions using high-fidelity simulation do have a positive impact on patient care32
In summary, we describe a multiple-scenario simulation team assessment that provides prospective experiences in decision making. Simulation scenarios can be designed to recreate scenarios managed by trauma algorithms as well as more complex trauma scenarios that in practice settings were associated with diagnostic error. As expected, teams performed better on scenarios when algorithms and heuristic approaches led to the correct diagnosis. By including scenarios that require a more analytic approach, trauma teams experienced how their diagnostic reasoning could lead to a delayed or missed diagnosis. Future investigations are needed to understand the cognitive processes that lead to expert trauma management and determine how to encourage better decision-making skills, particularly in settings that require rapid diagnosis and prompt management.
1. Pronovost PJ, Angus DC, Dorman T, Robinson KA, Dremsizov TT, Young TL. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA
2002; 288: 2151–2162.
2. Stockwell DC, Slonim AD. Quality and safety in the intensive care unit. J Intensive Care Med
2006; 21: 199–210.
3. Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology
1988; 69 (3): 387–394.
4. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care professional skills training and assessment. JAMA
1999; 282 (9): 861–866.
5. McEvoy MD, Smalley JC, Nietert PJ, et al. Validation of a detailed scoring checklist for use during advanced cardiac life support certification. Simul Healthc
2012; 7 (4): 222–235.
6. Wayne DB, Didwania A, Feinglass J, Fudala MJ, Barsuk JH, McGaghie WC. Simulation-based education
improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest
2008; 133 (1): 56–61.
7. Howell JD, Greenwald BM. Breathing new life into pediatric advanced life support training. Crit Care Med
2014; 42 (3): 744–745.
8. Kurosawa H, Ikeyama T, Achuff P, et al. A randomized, controlled trial of in situ pediatric advanced life support training. Crit Care Med
2014; 42 (3): 610–618.
9. Knudson MM, Khaw L, Bullard MK, et al. Trauma training in simulation: translating skills from SIM time to real time. J Trauma
2008; 64 (2): 255–263.
10. Lipman SS, Daniels KI, Carvalho B, et al. Deficits in the provision of cardiopulmonary resuscitation during simulated obstetric crises. Am J Obstet Gynecol
2010; 203: 179.
11. Steadman RH, Coates WC, Huang YM, et al. Simulation-based-training is superior to problem-based learning for the acquisition of critical assessment and management skills. Crit Care Med
2006; 34: 151–157.
12. Boulet JR, Murray DJ, Kras J, Woodhouse J, McAllister JD, Ziv A. Reliability and validity of a simulation-based assessment for medical students and residents. Anesthesiology
2003; 99: 1270–1280.
13. Lighthall GK, Barr J, Howard SK, et al. Use of a fully simulated intensive care unit environment for critical event management training for internal medicine residents. Crit Care Med
2003; 31: 2437–2443.
14. Hamilton N, Freeman BD, Woodhouse J, Ridley C, Murray D, Klingensmith ME. Team function during trauma resuscitation: a simulation-based performance assessment. J Grad Med Educ
2009; 1: 253–259.
15. McBride ME, Waldrop WB, Fehr JJ, Boulet JR, Murray DJ. Simulation in pediatrics: the reliability and validity of a multi-scenario assessment. Pediatrics
2011; 128: 335–343.
16. Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology
2007; 107 (5): 705–713.
17. McLaughlin K, Eva KW, Norman GR. Reexamining our bias against heuristics. Adv Health Sci Educ Theory Pract
2014; 19: 457–464.
18. Centers for Disease Control and Prevention. Injury Prevention & Control: Data & Statistics (WISQARSTM). Available at: http://www.cdc.gov/injury/wisqars
. Accessed on November 21, 2014.
19. Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf
2013; 22: Sii21–Sii27.
20. Newman-Toker DE, Pronovost PJ. Diagnostic errors—the next frontier for patient safety. JAMA
2009; 301 (10): 1060–1062.
21. Kahneman D, Klein G. Conditions for intuitive expertise: a failure to disagree. Am Psychol
2009; 64: 515–526.
22. Kahneman D, Tversky A. Judgment under uncertainty: heuristics and biases. In: Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty: Heuristics and Biases
New York, NY: Cambridge University Press; 1982: 3–21.
23. Klein G. Naturalistic decision making. Hum Factors
2008; 50 (3): 456–460.
24. Croskerry P, Norman G. Overconfidence in clinical decision making. Am J Med
2008; 121: S24–S29.
25. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med
2008; 121 (5 suppl): S2–S23.
26. Norman GR, Eva KW. Diagnostic error and clinical reasoning. Med Educ
2010; 44 (1): 94–100.
27. Elstein AS. Heuristics and biases: selected errors in clinical reasoning. Acad Med
1999; 74: 791–794.
28. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med
2003; 78: 775–780.
29. Henriksen K, Kaplan H. Hindsight bias, outcome knowledge and adaptive learning. Qual Saf Health Care
2003; 12 (suppl 2): ii46–ii50.
30. Wears RL, Nemeth CP. Replacing hindsight with insight: toward better understanding of diagnostic failures. Ann Emerg Med
2007; 49: 206–209.
31. Gaba DM. Simulations that are challenging to the psyche of participants: how much should we worry and about what? Simul Healthc
2013; 8 (1): 4–7.
32. McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-based mastery learning with translational outcomes. Med Educ
2014; 48 (4): 375–385.