Secondary Logo

Journal Logo

A Method for Designing Symmetrical Simulation Scenarios for Evaluation of Behavioral Skills

Bush, Mary Chasko MD; Jankouskas, Tara S. MSN, RN; Sinz, Elizabeth H. MD; Rudy, Sally BSN, RN; Henry, Jody BS; Murray, W Bosseau MD

Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: July 2007 - Volume 2 - Issue 2 - p 102-109
doi: 10.1097/01.SIH.0b013e31802d981c
Empirical Investigations
Free

From the Simulation Development and Cognitive Science Laboratory, Office of Education Affairs, Pennsylvania State University College of Medicine, Hershey, PA.

Reprints: Elizabeth Sinz, MD, Department of Anesthesiology, H187, 500 University Drive, PO Box 850, Hershey, PA 17033-0850 (e-mail: esinz@psu.edu).

The authors have indicated they have no conflict of interest to disclose.

The increasing use of simulation technology in medical education necessitates a method to formulate carefully planned scenarios to assess performance. Much of simulation in healthcare endeavors to teach technical skills such as the drugs and dosages needed during Advanced Cardiac Life Support (ACLS)1 or procedural skills such as those needed for laparoscopic surgery2–8 or bronchoscopy.9–11 In contrast, scenarios can be designed specifically to evaluate behavioral skills such as communication, leadership, and situation awareness. These are skills typically emphasized in crisis resource management (CRM) courses.12–15 This manuscript describes a systematic framework for developing interchangeable, “symmetrical” simulation scenarios that can be randomly ordered to repeatedly test crisis behavioral skills. The mechanism given in this methodologic description is reproducible and can generate a series of scenarios that are similar in flow, basic content, and difficulty. This standardizes the expertise necessary to solve each crisis and avoids the wide range of complexity commonly seen in simulation scenarios. Symmetrical scenarios provide a context for assessment using behavioral skills scoring systems by providing clear opportunities within each scenario for participants to demonstrate elements of each skill. To compare “before and after” performance and behavioral skills retention using such a rating system, it is essential to have comparable simulation scenarios.

Many institutions derive their CRM scenarios from a database of clinical scenarios already in existence. Scenarios developed from actual cases may be the most realistic and can avoid a potential complaint that “this would never happen in real life.” Some experts would argue that all scenarios should be based on actual clinical events for this reason, and it is reasonable to assume that purely formative sessions might best be grounded in specific clinical experiences. However, restricting simulations to known clinical events may lead to widely disparate scenarios, making comparisons of technical and behavioral skills difficult. A recent study by Yee and coauthors assessed and reevaluated crisis behavioral skills in anesthesiology residents using clinical scenarios from an existing repertoire.16 Using a variety of clinical scenarios with a diverse range of complexity, they found a measurable benefit from CRM training but could not show additional benefit from further simulator sessions. When assessing behavioral skills with complex, catastrophic crises, it may be difficult to distinguish poor performance from flawed scenarios.17 Indeed, an early investigation by Gaba et al. in 1998 stated a need for “more tightly controlled scenarios.”17 This proposed method of symmetrical scenario development relies on actual clinical experiences adapted in a realistic but specific manner to elicit particular responses from team members, thus facilitating assessment of behavioral skills for controlled trials.

Evaluating the performances of simulation laboratory participants has prompted the development of many scoring systems. One such tool is the Anesthesia Nontechnical Skills (ANTS) scoring system, whose categories are shown in Table 1.18 ANTS is a validated system created by researchers at the University of Aberdeen.19 It is recognized for its practical ease of use, capability to assess both nurses and physicians, and applicability to multiple clinical settings and specialties.19,20 Other evaluation systems that facilitate the evaluation of crisis behavioral skills include the older NOTECHS system, borrowed from aviation literature,21 the Crisis Management Behaviors rating instrument (Table 2),17 and the Ottawa Crisis Resource Management Global Rating Scale (Table 3).22 The method described in this paper is based on the principles outlined in the ANTS construct; however, scenarios can be developed to use any scoring system that assesses teamwork and interpersonal skills.

Table 1

Table 1

Table 2

Table 2

Table 3

Table 3

Back to Top | Article Outline

DESCRIPTION OF METHODOLOGY (WITH EXAMPLES)

This methodology creates a series of two or more CRM scenarios that can be used to assess crisis behavioral skills (Table 4). Each series of scenarios has common characteristics, such as the type of human simulator and the nature of the “crisis event.” Once these determinations are made, each individual scenario in the series needs unique identifying characteristics, such as event location and the roles of the human actors who interact with study participants. Finally, it is useful to organize the details of the resulting scenarios (Table 5) and match each element to a specific CRM behavior (Table 6).

Table 4

Table 4

Table 5

Table 5

Table 6

Table 6

Table 6

Table 6

Back to Top | Article Outline

STEPS COMMON TO ALL SCENARIOS

1. Assemble a Team of Scenario Developers

Members of the scenario development team should include those with clinical expertise, knowledge of the CRM principles being assessed, and familiarity with the measurement system being used. For example, the researchers in our planning team included a multidisciplinary group of nurses and physicians well-versed in the principles of CRM. Several members were proficient with the ANTS scale.

Back to Top | Article Outline

2. Consider Study Participants When Selecting Patient Characteristics

Next, consider the levels of clinical expertise of the participants as well as their medical disciplines. The selection of patient characteristics can be tailored to the experience of study participants, such as pairing pediatric nurses with pediatric patients. In this example, the physiologic parameters and basic traits of the simulator patient (such as age and weight) were identical for each scenario. This standardization contributed to symmetry among the scenarios in the series, acting as a control to eliminate the differing clinical issues associated with different patients and different clinical signs. Also, participants can focus on CRM principles rather than on technical details given that a standard patient weight simplifies the calculation of drug dosages.

Back to Top | Article Outline

3. Choose a “Crisis Event”

To further deemphasize clinical and technical issues, we avoided highly complex medical problems and presented circumstances where mastery of behavioral skills would promote smooth handling of the crisis. Additionally, based on the clinical expertise of the design team, a straightforward, typical, single pathology was chosen as the triggering event for the crisis. For example, each of the four scenarios involves a medication error leading to a hypoxic event. Medication errors are a daily risk encountered by all health care providers and provide a logical clinical basis for hypoxia, making the resulting scenarios realistic. This crisis event can be recognized by multiple levels of clinicians from medical students to nurses to attending physicians, and therefore was appropriate for participants with different skill sets.

Back to Top | Article Outline

4. Develop Scenario Flow

The goal of this step is to determine how and when the study participants will appear in the scenario and how varying the makeup of the crisis team can elicit CRM behaviors. For example, the scenario begins when pediatric nurses assume care of the simulated patient from an actor. These first responders may then call for help from a more specialized pool of health care providers, such as pediatric residents, who may eventually involve a participating anesthesia resident. The basic flow of this scenario algorithm tests the ability of the team to recognize the crisis early, identify a leader, declare an emergency, exchange information, and support the other team members. Because exchange of information among the group is a key element of team working, it is essential to provide opportunities for several handovers so that this action can be assessed. If appropriate, a new leader may be chosen, who is challenged to display new authority. This person must assess the capabilities of the evolving group and assign tasks appropriately.

In this example, the hypoxic event begins to unfold relatively quickly. If it is not resolved, the simulated patient becomes bradycardic, requiring the participants to initiate Pediatric Advanced Life Support (PALS) protocols. The scenario ends when the patient is stabilized, either by securing the airway or by reversing the medication error (Fig. 1).

Figure 1.

Figure 1.

Because the makeup of the team members present at the crisis varies, much like real-life emergencies, these scenarios require the group to support each other in achieving each goal. As the flow of the scenarios demand that the roles of the players change by taking on new roles and needing to continuously exchange old and new information, it becomes apparent which teams are supportive and work cohesively during the crisis. Additionally, the varying makeup of the teams demands good task allocation and distribution of work by the leader.23

Back to Top | Article Outline

Steps Unique to Each Scenario

Until this point, every scenario in this series has the same framework to create parallel scenarios. But by changing enough variables, such as location of the crisis event, the particular cause of the problem (slow incipient onset of hypoxia, followed by apnea), and the roles of the actors, the scenarios appear different, despite using identical pathophysiology.

Back to Top | Article Outline

5. Develop Events Leading up to Crisis

The first step in developing individual scenarios is to create varying background stories leading up to the crisis event. For example, each of the patients has a different specific reason for becoming hypoxic, such as accidentally receiving a narcotic overdose for a procedure. The sequence of events before the first responders arrive must be brief but adequately convoluted so that the participants will need to work together to ascertain the correct history. Here they have opportunities to use situational awareness skills, identify resources, gather details, and exchange information.

Back to Top | Article Outline

6. Select a Setting for the Crisis

The location and setting of each scenario is varied to create the sense of a new scenario and preserve suspension of disbelief. The participants are put in generalized, recognizable settings instead of locations unfamiliar to them. For example, in two of the scenarios, the nurses are asked to transport a patient to his room from the MRI scanner or postanesthesia recovery room. These are typical, reasonable tasks for this group of trainees.

Back to Top | Article Outline

7. Define the Distracters and Distractions

The next step is to define the distractions and interrupters that are present during the crisis. Task management is challenged by altering the position of the patient or the bed and adding a noise disturbance (e.g., a loud radio or loud TV). Live actors who divert the attention of the participants challenge team working and situation awareness. Two types of actors were chosen: a “clueless” personality who offers no help and constantly draws attention away from the patient, and a “potentially helpful” individual who possesses some relevant information regarding the pathogenesis of the problem and may be used to perform certain tasks during the crisis if requested.

Proper planning and preparing can help the team to prioritize current and possible future events. As equipment malfunctions are a common issue faced by multiple levels of health care providers, two types of equipment issues are represented: equipment failures and absent equipment. These failures can be managed by successful teamwork and create opportunities for decision making. For example, the nature of the crisis event requires the team members to assess the airway and provide oxygen for the hypoxic patient. Equipment failures and distractions rapidly create new problems and trigger new crisis events as the initial problem increases in severity.23 A faulty bag-valve-mask and empty or missing oxygen tank connector are smaller problems that can distract from the main issue (proper patient care) and add up to create a sequence of events more problematic than the initial crisis.23 If the participants never realize that there is an equipment failure, the crisis continues to worsen and challenges the participants to set new priorities and find the origin of new problems. In this way, the scenario regenerates opportunities to demonstrate effective task management.

This method creates symmetrical scenarios that require groups to deal with other types of distractions: predisposing factors and latent errors. These are the underlying aspects of the crisis that are present before the event.23 They are usually hidden factors and pose no distraction until the adverse consequences declare themselves in the midst of the crisis. The team is challenged by a clueless nursing student to assess his limited capabilities and use assertiveness to remove him from the crisis if he becomes a liability. His annoying presence is not important and he can be ignored initially until the nurses become preoccupied with the ensuing emergency. Then, his persistence challenges the leader's vigilance and may cause mismanagement of the crisis. He can thwart the team's ease in exchanging information and coordinating activities. Alternatively, an efficient team can make this person a nonmedical resource, and turn the most annoying distracter into a valuable helper or messenger.

Equipment failures also provide opportunities for team members to exchange information about missing or faulty equipment. The team leader is confronted with issues in coordinating activities, relegating distractions, and reducing fixation errors. The identification of some equipment issues may be limited to those with more in-depth knowledge of the tools; an anesthesiologist may recognize a faulty bag-valve-mask more readily than a pediatric nurse and will need to report this information to the leader. Since there are several failures in each scenario, there is a higher chance that the team will recognize and address this type of issue.

Back to Top | Article Outline

8. Make Resources Available

Despite introducing multiple distractions to challenge the subjects' CRM skills, multiple resources should be available during the crisis to assist them when they seek help and information. Replacement equipment and additional patient documentation, such as a history and physical, anesthesia record, or other pertinent medical records should be available. Actors can positively contribute to the efforts of the group; they should be instructed to give information when specifically questioned by the participants but refrain from volunteering any supplemental information. Incorporating teams of providers with different areas of expertise force them to turn towards each other as resources.

Back to Top | Article Outline

9. Write a Script for the Actors

Scripts of key phrases and comments for each scenario were written and distributed to the actors to serve as a guide for their dynamic portrayal as the human distracters. Specific cues such as “Wow, the patient looks really blue,” were mixed with unhelpful comments similar to, “This is my first day working in MRI,” and more distracting statements like, “You need to move this patient because we have a full schedule of scans to do.” Upon completion and finalization of the scenarios in our series, we provided wireless headsets to the actors and practiced scenario run-throughs to facilitate smooth and standardized performances.

Back to Top | Article Outline

RESULTS

The product of our practical example is the series of scenarios shown in Table 5. Organizing the data in this way not only helps to identify the symmetrical glitches among these four scenarios, but also aids in setting up the details of the actual simulation. Because we assessed the CRM skills of participants using the ANTS system, we were required to ensure that this scenario development plan includes adequate testing of the nontechnical skills described in this particular assessment tool. Taking each separate component of the scenario and matching it to a particular element in a scoring system produces a graphical representation like Table 6. Many elements are assessed more than once by each of the circumstances. This duplication is intentional and increases the probability that the participants will have opportunities to demonstrate these behaviors.

Back to Top | Article Outline

DISCUSSION

This is the first methodological description of the development of a series of interchangeable simulation scenarios for the purpose of teamwork skills evaluation. It allows researchers to create a series of symmetrical scenarios to test crisis behavioral skills such as communication, avoiding fixations, team working, leadership, and situation awareness. This method deemphasizes the importance of clinical knowledge and generates conditions within the scenario to reward effective task management and use of resources. These scenarios are likely amenable to evaluation by various behavioral rating scales, including ANTS (Table 6).

The scenarios created by using this method make it possible to assess a group's anticipation and avoidance of more serious problems. For example, the simple problem of hypoxia is critical enough to require the team to anticipate further problems. They must prepare for the possibility of cardiac arrest while simultaneously addressing the immediate concerns. As they evolve, these scenarios require team members to either avoid or anticipate intubation. They must make the decision to reverse the medication error with naloxone before the patient becomes too unstable and needs a definitive airway. Gaba and coauthors refer to this concept as the “prediction of future states” and point out that proper anticipation also affects the time frame available for future actions.23

If a leader is acknowledged early in each scenario, then the ability of the team to enact sound risk evaluation and decision making is improved.23 The leader should assume the role of primary decision maker and not be involved in the task management.23 Gaba labels this person the “supervisory controller,” who must reflect on past choices, identify options, balance risks, and make decisions.23 The leader should be vigilant and constantly improve his situation awareness, understand the data, verify the data, continually reevaluate circumstances, and determine which actions are effective.23 This scenario development scheme provides opportunities to gather information from the patient, monitors, and actors, who can be resources as well as distracters. As the crisis unfolds and becomes more serious, new data is introduced and it becomes the responsibility of the leader and the team members to recognize and appreciate the new information.

This method was also developed to assess effective communication between leaders and followers, as it is one of the most important qualities of a good team. The exchange of ideas involves patient assessment, resource identification, and decision making. The noise distractions obscure clear and specific instructions but they are manageable by the participants who can reduce the commotion by eliminating the distraction. Finding solutions to equipment errors requires followers to close the communication loop and report back to the leader.23 Effective communication also involves giving precise commands, allocating specific tasks to specific people, and supporting open exchange of information while avoiding conflict.

Most of the research performed on CRM exists in the aviation and anesthesiology literature. This methods paper expands those boundaries by incorporating multiple types of practitioners. We chose to assign pediatric nurses to the role of first responders, as nurses typically discover a crisis. However, the order in which groups arrive to the scene is interchangeable and creates a different dynamic with each new arrangement. Any mix of providers is possible within this framework, from nonmedical personnel and technicians to nurses and physicians. This can accurately simulate the sequence of genuine crisis events.

One of our first challenges was to make scenarios that are similar in design while preserving the suspension of disbelief. This concept is the ability of the participants to imagine themselves in a real-life crisis and put on hold the knowledge that they are in a simulated environment with an electronic patient. With each new 5-year-old patient, it may be more difficult for participants to imagine themselves in a brand new crisis. Furthermore, it is possible that they may expect a patient with hypoxia as they encounter each new scenario in the series. Their ability to predict the circumstances common to all scenarios may affect certain elements of crisis resource management, such as situation awareness and decision making. The nurses may first place oxygen on the patient because they recall that particular intervention from a previous crisis, not because they have properly assessed the situation and have recognized the problem. We did not specifically ask our participants for feedback regarding our series of scenarios and any perceived similarities. This may be one of the first steps in future validation of this method. Even so, a certain amount of predictability among the scenarios may be acceptable to preserve their design similarity.

The greatest advantage of this system is the ability to create many different scenarios based on the same framework and the same sequence of programmed simulation events. The number of comparable scenarios that can be generated is endless and their order can be randomized for use in controlled trials. Given the same basic equipment and number of people required to act in each scenario, multiple CRM sessions can be given to a group of participants within a short period of time with minimal preparation.

As other scoring systems for assessing CRM behaviors are developed, this method can be adapted for objective evaluation with those new tools. Informal subjective evaluation during CRM debriefings is also achievable if no validated scoring system is used to evaluate performance. Further research may involve replicating the method for the development of scenarios for a greater variety of health care providers interested in receiving CRM training. The method can be expanded to test clinical competency for simple clinical problems for any level of provider. It may also prove useful for developing crisis management scenarios that assess technical skills and procedural competency for courses such as PALS and ACLS.

Back to Top | Article Outline

CONCLUSION

Careful design of CRM scenarios is essential for assessment and future research of teamwork skills to move forward. This method can guide the development of simulation scenarios that consistently assess crisis behavioral skills. As one of the first methodologies described for scenario development, more rigorous testing and use by other researchers is necessary to further define its limitations. However, this novel concept of symmetrical design for a series of simulation scenarios represents an important new technique for the scenario developer's toolbox.

Back to Top | Article Outline

REFERENCES

1. Wayne DB, Butter J, Siddall VJ, et al.: Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med 2006;21:251–256.
2. Stefanidis D, Sierra R, Korndorffer JR, Jr, et al.: Intensive continuing medical education course training on simulators results in proficiency for laparoscopic suturing. Am J Surg 2006;191:23–27.
3. Hassan I, Maschuw K, Rothmund M, et al.: Novices in surgery are the target group of a virtual reality training laboratory. Eur Surg Res 2006;38:109–113.
4. Andreatta PB, Woodrum DT, Birkmeyer JD, et al.: Laparoscopic skills are improved with LapMentor training: results of a randomized, double-blinded study. Ann Surg 2006;243:854–860.
5. Sutherland LM, Middleton PF, Anthony A, et al.: Surgical simulation: a systematic review. Ann Surg 2006;243:291–300.
6. Chou B, Handa VL: Simulators and virtual reality in surgical education. Obstet Gynecol Clin North Am 2006;33:283–296.
7. Korndorffer JR, Jr, Dunne JB, Sierra R, et al.: Simulator training for laparoscopic suturing using performance goals translates to the operating room. J Am Coll Surg 2005;201:23–29.
8. Seymour NE, Gallagher AG, Roman SA, et al.: Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002;236:458–463.
9. Blum MG, Powers TW, Sundaresan S: Bronchoscopy simulator effectively prepares junior residents to competently perform basic clinical bronchoscopy. Ann Thorac Surg 2004;78:287–291.
10. Colt HG, Crawford SW, Galbraith O, 3rd: Virtual reality bronchoscopy simulation: a revolution in procedural training. Chest 2001;120:1333–1339.
11. Ost D, DeRosiers A, Britt EJ, et al.: Assessment of a bronchoscopy simulator. Am J Respir Crit Care Med 2001;164:2248–2255.
12. Holzman RS, Cooper JB, Gaba DM, et al.: Anesthesia crisis resource management: real-life simulation training in operating room crises. J Clin Anesth 1995;7:675–687.
13. Yule S, Flin R, Paterson-Brown S, Maran N: Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 2006;139:140–149.
14. Howard SK, Gaba DM, Fish KJ, et al.: Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 1992;63:763–770.
15. Reznek M, Smith-Coggins R, Howard S, et al.: Emergency medicine crisis resource management (EMCRM): pilot study of a simulation-based crisis management course for emergency medicine. Acad Emerg Med Apr 2003;10:386–389.
16. Yee B, Naik VN, Joo HS, et al.: Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology 2005;103:241–248.
17. Gaba DM, Howard SK, Flanagan B, et al.: Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998;89:8–18.
18. Fletcher G FR, McGeorge P, Glavin R, et al.: Rating non-technical skills: Developing a behavioural marker system for use in anaesthesia. Cognition Technol Work 2004:165–171.
19. Fletcher G, Flin R, McGeorge P, et al.: Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003;90:580–588.
20. Reader T, Flin R, Lauche K, Cuthbertson BH: Non-technical skills in the intensive care unit. Br J Anaesth 2006;96:551–559.
21. Flin R ML, Goeters K: Development of the NOTECHS (non-technical skills) system for assessing pilots' CRM skills. Human Factors Aerospace Safety 2003:95–117.
22. Kim J, Neilipovitz D, Cardinal P, et al.: A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med 2006;34:2167–2174.
23. Gaba DM, Fish KJ, Howard SK. Crisis management in anesthesiology. New York: Churchill Livingstone; 1994.
Keywords:

Simulation; Nontechnical skills; Crisis resource management; Crisis management behaviors; Scenario development; Behavioral skills; Anesthesia nontechnical skills; Education; Teamwork; Research; Communication

© 2007 Society for Simulation in Healthcare