Skip Navigation LinksHome > January 2014 - Volume 120 - Issue 1 > Building the Evidence on Simulation Validity: Comparison of...
Anesthesiology:
doi: 10.1097/ALN.0b013e3182a44bc5
Education: Original Investigations in Education

Building the Evidence on Simulation Validity: Comparison of Anesthesiologists’ Communication Patterns in Real and Simulated Cases

Weller, Jennifer M.D., M.B.B.S., M.Clin.Ed., F.A.N.Z.C.A., F.R.C.A.; Henderson, Robert M.Sc.; Webster, Craig S. Ph.D.; Shulruf, Boaz Ph.D.; Torrie, Jane M.B.Ch.B., F.A.N.Z.C.A.; Davies, Elaine R.N.; Henderson, Kaylene; Frampton, Chris B.Sc. Hons., Ph.D.; Merry, Alan F. M.B.Ch.B., F.F.P.M.A.N.Z.C.A., F.R.C.A., F.A.N.Z.C.A.

Free Access
Medical Education
Article Outline
Collapse Box

Author Information

Collapse Box

Abstract

Background: Effective teamwork is important for patient safety, and verbal communication underpins many dimensions of teamwork. The validity of the simulated environment would be supported if it elicited similar verbal communications to the real setting. The authors hypothesized that anesthesiologists would exhibit similar verbal communication patterns in routine operating room (OR) cases and routine simulated cases. The authors further hypothesized that anesthesiologists would exhibit different communication patterns in routine cases (real or simulated) and simulated cases involving a crisis.
Methods: Key communications relevant to teamwork were coded from video recordings of anesthesiologists in the OR, routine simulation and crisis simulation and percentages were compared.
Results: The authors recorded comparable videos of 20 anesthesiologists in the two simulations, and 17 of these anesthesiologists in the OR, generating 400 coded events in the OR, 683 in the routine simulation, and 1,419 in the crisis simulation. The authors found no significant differences in communication patterns in the OR and the routine simulations. The authors did find significant differences in communication patterns between the crisis simulation and both the OR and the routine simulations. Participants rated team communication as realistic and considered their communications occurred with a similar frequency in the simulations as in comparable cases in the OR.
Conclusion: The similarity of teamwork-related communications elicited from anesthesiologists in simulated cases and the real setting lends support for the ecological validity of the simulation environment and its value in teamwork training. Different communication patterns and frequencies under the challenge of a crisis support the use of simulation to assess crisis management skills.
Back to Top | Article Outline

What We Already Know about This Topic

* Simulation is used to evaluate and teach teamwork, but whether communication patterns during simulation reflect communication in the clinical environment is unknown
Back to Top | Article Outline

What This Article Tells Us That Is New

* In an evaluation of 20 anesthesiologists, communication pattern for routine cases was similar during simulation as during clinical care in the operating room, supporting the ecologic validity of simulation
HEALTH care is now acknowledged as being delivered by teams and not by individual health professionals.1 Effective teamwork is recognized as an important contributor to patient safety,1 and verbal communication underpins many dimensions of teamwork.2,3 Simulation-based education is often used to develop both effective teamwork and communication, and in assessment and recertification.4,5 However, important questions remain about the validity of simulation, and there is limited objective evidence comparing teamwork behaviors, including verbal communication, between clinical and simulated environments.
Assessment of anesthesiologists in the workplace typically takes place in routine situations. It is unclear whether effective patterns of communication displayed in routine situations will be sustained in a crisis. An advantage of simulation is the ability to recreate rare or challenging clinical events, including crises.
Therefore, in this study, we set out to explore communication among anesthesiologists, anesthesia technicians, and other members of the operating room (OR) team during simulations of cases with and without a crisis, and in comparable routine cases in the OR.
As our theoretical framework, we used the Brunswikian approach to ecological validity, where the validity of participant behaviors is a function of the validity of the environmental cues6 and, in the context of assessment, that the observed behaviors in the assessment condition (e.g., simulation environment) predict behaviors in the open environment.7 We found no previous observational studies comparing communication in the simulated environment with communication in the clinical environment. We designed simulations to provide comparable environmental cues with those seen in ORs in our institution. In an observational study, Manser et al.8 compared anesthesiologists’ activity patterns in a simulated and a real OR, using a structured computerized coding system consisting of 43 observable, predefined behaviors, and demonstrated similar patterns, although at higher activity levels in the simulation. We have applied this methodology to verbal communications and have examined the extent to which the simulation environment elicited similar verbal communications as a comparable clinical environment. Further justification for using simulation to teach and assess communication would be provided if the verbal communications elicited by the simulation environment resemble those elicited by a real case.
Our primary hypothesis was that anesthesiologists would exhibit similar verbal communication patterns in a routine OR case and routine simulated case. Our secondary hypothesis was that anesthesiologists would exhibit different communication patterns in routine cases (in the OR or simulated environment) and simulated cases involving a crisis.
Back to Top | Article Outline

Materials and Methods

Fig. 1
Fig. 1
Image Tools
Ethics approval was obtained from the Northern Ethics Committee Y, Auckland, New Zealand (approval NTY/081 1S IEXP). All study participants, and all staff members and patients present in the designated OR during filming, gave written informed consent. The study design is summarized in figure 1.
Back to Top | Article Outline
Participants
Each participant team comprised an anesthesiologist and an anesthesia technician. In New Zealand, an anesthesia technician is an assistant who has completed 3 yr of vocationally focused training at a tertiary institution. Within the OR team, the anesthesiologist and technician work closely together as a subunit. We included only anesthesiologist consultants or residents in their final year of training.
Back to Top | Article Outline
Sample Size
Manser et al.8 demonstrated differences in activity patterns with a sample of six anesthesiologists in two scenarios over five cases. In our previous work, we demonstrated clinically relevant effects with a sample of 20.9,10 Taking feasibility into account, we aimed to enroll 20 anesthesiologists and anesthesia technicians.
Back to Top | Article Outline
OR Filming
Using unobtrusive audiovisual equipment, we video recorded the anesthesiologists in the OR over the course of a full-day elective general surgical operating list (two to five cases). The automated anesthesia records were collected for each case.
Back to Top | Article Outline
Simulated Cases
We used a METI HPS full body human patient simulator (Medical Education Technology Inc., Sarasota, FL) in a simulated OR, using the same automated record-keeping system as used in the OR (Safersleep Inc., Nashville, TN)11 to create an accurate record of physiological data and events.
In the simulations, the study participants (the anesthesiologist and anesthesia technician) worked with a full OR team, comprising experienced faculty acting in the roles of surgeon, scrub nurse, and circulating nurse. The circulating nurse also assisted the participants in tasks specific to the simulation environment, e.g., fetching equipment, taking blood samples for analysis.
The routine simulation, modeled on the cases typical of the OR where filming took place, involved a 70-yr-old female undergoing elective laparoscopic surgery. The case proceeded uneventfully apart from a change in blood pressure of approximately 20% from baseline. In a second scenario, participants then managed one of two crisis simulations: profound hypotension with hyperkalemia or an unstable tachyarrhythmia. Although the crisis simulations differed, they were both manifestly more complex than the routine simulation.
Back to Top | Article Outline
Conduct of Simulator Study Days
We followed a standardized protocol for familiarization and briefing to the simulation environment. Simulations began when the anesthesiologist first entered the simulated OR to begin the anesthesia for the patient and ended with the completion of surgery. The routine case was followed by the case containing the crisis. The day concluded with an educational debrief.
Back to Top | Article Outline
Selection of Video Segments for Comparison
Table 1
Table 1
Image Tools
We wished to compare two video segments in each of the three conditions for each anesthesiologist: an “induction” segment and an “event” segment. We recorded several hours of video for each anesthesiologist in the OR consisting of two to five cases. From these, a research assistant (not involved in the simulation component of the study, the coding or analysis) selected video segments on the basis of the criteria listed in table 1.
Back to Top | Article Outline
The Coding Framework
We used a structured coding framework to code anesthesiologists’ verbal communications, allowing computerized recording and analysis of relevant communication events. The methodology was based on the observation system developed by Manser et al. to observe anesthesiologists’ actions during simulated and real OR cases.12,13 The starting point for the coding framework was well-established dimensions of teamwork taken from the extensive literature on effective teamwork, crew resource management, and nontechnical skills.14–16 These were workload management; problem solving and decision making; open communication environment; and sharing mental model. We then developed observable verbal manifestations on each dimension of teamwork. These were identified initially on the basis of published examples,16 then reviewed by a group of experts, and subsequently piloted. This resulted in a list of eight mutually exclusive items for communication types, within which a subset of four items that related to sharing of situational information.
All relevant communication events in the videos were coded initially on the basis of to whom the anesthesologist was communicating (six mutually exclusive items) and the communication type. A subset of these communication types involved sharing situational information about the patient (four items), and these were then coded against these items. The chosen communication events were not an exhaustive list of all possible communications.
Back to Top | Article Outline
Piloting the Instrument
The coding framework was piloted by three researchers viewing six video segments, following which descriptors for the items were refined to address ambiguity.
Back to Top | Article Outline
Coding the Videos
Two coders, both with clinical experience in the operating theatre, were trained in the use of the observation tool. The order of the videos was randomized for coding. Using Observer XT video-analysis software (Noldus Information Technology, Wageningen, The Netherlands), the two coders independently coded all videos using the coding framework. A 10% sample of these initial independent codings showed 85% agreement on coded verbal communications. The two coders then met and reviewed every communication event where their coding differed and reached consensus on each event and item over the course of a single viewing. The final codings for analysis were thus a consensus of the two coders.
Back to Top | Article Outline
Questionnaire
After the simulation and before any discussion, the anesthesiologist and anesthesia technician completed an on-line questionnaire exploring their perceptions of the realism of the simulations.
Back to Top | Article Outline
Statistical Analysis
The percentages of all coded items were then calculated for each anesthesiologist in each condition, and compared using a mixed-model ANOVA that included anesthesiologist as a random factor, and condition as a fixed within-subject effect. When the overall condition effect for a particular item was significant (two-tailed P < 0.05) from the ANOVA, we proceeded to pair-wise comparisons among the three conditions, using Fisher protected least significant difference test, applying the Bonferroni correction for multiple comparisons. The differences between conditions are represented as Cohen D effect sizes, calculated as the difference between the means divided by the SD. All analyses were undertaken using SPSS V19 (IBM SPSS Statistics, Armonk, NY).
Back to Top | Article Outline

Results

We recruited 20 anesthesiologists (17 consultants and 3 senior residents) from the pool of 58 eligible anesthesiologists and 20 anesthesia technicians.
Of the 20 anesthesiologists recorded in the simulated cases, we were unable to record three in the OR as they had moved to other hospitals. This generated a total time of 2,469.2 min of video, with an average per video segment of 21.66 min (median, 20 min; range, 6.6–84.8 min; interquartile range, 16.5–22.5 min).
Table 2
Table 2
Image Tools
From this data, we identified 2,502 relevant verbal communication events between the anesthesiologist and other members of the OR and simulated OR teams, 48.7% of which involved sharing of situational information about the patient. The different items, expressed as percentages of all 2,502 coded communication events, are shown in table 2. Items that were coded less often than 5% of the total were excluded from further analysis.
The number of communication events in the crisis simulations was considerably higher than in either of the other two conditions: 16% (400) were in the OR cases; 27% (683) in the routine simulations; and 57% (1,419) in the crisis simulations. The average number of coded communication events per video segment per condition was 12.5 (OR), 17.5 (routine simulation), and 36.5 (crisis simulation).
Table 3
Table 3
Image Tools
The mean percentages of all coded items for each anesthesiologist in each condition are shown in table 3.
The only significant differences between the OR and the routine simulation were the percentages of communications with the circulating nurse. The circulating nurse was a faculty member with the specific role of assisting participants with tasks related to working in the simulation environment.
There were, however, significant differences between the crisis simulation and the other two conditions in the items related to sharing of situational information. Significantly higher percentages of verbalizing patient status, assessing patient status, and proposing a plan of action occurred in the crisis simulation than in the real OR. Proposing a plan of action was also significantly different between the routine simulation and the crisis simulation. In addition, there was a trend toward differences in assessing patient status.
Table 4
Table 4
Image Tools
The pair-wise comparisons of the routine simulation with the other two study conditions (crisis simulation and OR; table 4) indicate that the effect sizes for the comparisons that were not statistically significant were small to moderate, generally less than 0.50. This would indicate that the sample size of 20 anesthesiologists provided a sufficient power to detect significant and potentially important differences between the conditions.
Back to Top | Article Outline
Questionnaire Results
Table 5
Table 5
Image Tools
Participants rated team communication as realistic in 86% of responses (n = 80). Participants considered their communications generally occurred with a similar frequency in the simulations as in comparable cases in the OR (table 5).
Back to Top | Article Outline

Discussion

Our results support our primary hypothesis that anesthesiologists would exhibit similar verbal communication patterns in a routine OR and a routine simulated case. Anesthesiologists directed their communications to different members of the team in similar percentages in the OR and in the simulations, with the exception of communications with the circulating nurse. The latter was a member of the faculty assigned to assist participants with tasks specific to the simulation environment and, not unexpectedly, significantly more communications were directed to this person in the simulations.
Our secondary hypothesis was that anesthesiologists would exhibit different communication patterns in routine cases (in the OR or simulated environment) and simulated cases involving a crisis. Again, our results support this hypothesis.
This is the first study that has attempted to compare anesthesiologists’ verbal communications in the operating theatre with those in the simulated environment. Manser et al. looked at another aspect of ecological validity of the simulation environment and provided evidence that anesthesiologists undertook similar tasks during simulated and OR cases. They suggested that understanding the effect of the simulation environment on teamwork behaviors was key to create valid learning experiences.8 In this study, we explored the effect of simulation and the occurrence of a crisis on key team communications and identified areas of similarity and difference. Although in this study we have not rated quality of communication, the items in our coding tool are closely linked to established dimensions of teamwork. This could conceivably allow more specific research into which types of communication are desirable; for example, does verbalizing more situational assessment information encourage more suggestions from other team members, and subsequently better patient management decisions?
Increased sharing of situational information about the patient in crisis simulations makes intuitive sense in view of the additional challenges involved and supports previous findings.17 Although the ability of our coding framework to detect this difference supports its construct validity, this finding suggests that communication patterns in routine simulations may not be predictive of those in crisis simulations. This finding may also apply in the clinical environment. Although future research to explore the validity of simulation for assessing crisis management should optimally compare real and simulated crises, this is not likely to be feasible. Our findings of similarity between routine simulated and OR cases, but differences in simulated cases containing a crisis, provide some limited support for the proposition that anesthesiologists’ ability to manage a crises may be better assessed in a simulator than by inference from the management of routine OR cases. Furthermore, far fewer team-related communications occurred in the OR than in the crisis simulations, compounding the difficulty in assessing teamwork communication in the real setting.
Modern approaches to validation consider a unified concept of validity where evidence from different sources is required to build the argument.18,19 Much of the evidence on simulation validity rests on comparisons of novice and expert performance in the simulator.20,21 Experts would be expected to perform better than novices in real life and, if this is also the case in a simulation, it would support the validity of the simulation. Although other types of validity evidence exist, in this study we looked for evidence on ecological validity, determining whether the simulation elicited similar or different responses from participants. From this and Manser’s study, we can now say that anesthesiologists’ activity patterns and teamwork-related verbal communications support these important components of ecological validity of the simulation environment.
Back to Top | Article Outline
Limitations
Selecting a random sample from the limited number of available anesthesiologists was not feasible and may introduce bias. However, the paired study design is a strength because it allowed participants to act as their own controls.
Our study was not designed to demonstrate equivalence between the conditions, but rather to explore similarities and possible differences. In general, demonstration of equivalence requires a larger study than the demonstration of differences. For this reason, we have reported the effect size of comparisons between the conditions. A larger sample size with more statistical power may well identify further more subtle differences between the three conditions; however, the small to moderate effect sizes in our study suggest it was adequately powered to detect any important differences. Our results may be useful to plan future studies of equivalence.
The coding framework that we used is novel. To ensure content validity, we consulted the literature and drew on the experience of senior anesthesiologists and a human factors expert. The ability of the framework to detect different communication patterns with the circulating nurse lends support to its validity in that it has detected an anticipated difference (due to this person’s role in the simulation). We had no previous knowledge of interrater agreement with the coding framework. Similar studies using observation tools for coding observable events, where no judgment on quality or completeness is required, have used a single coder.22 An interim reliability check before consensus supported high levels of agreement between our two coders, suggesting only one coder would be necessary for future studies.
Although we could manipulate the simulation environment, we were unable to manipulate the clinical environment and this inevitably led to some compromise in comparability of the chosen video segments. We chose to study intubation and a change in blood pressure of 20% from baseline, which could be expected to involve similar tasks and workload demands in the simulated and clinical environment.
In conclusion, in our study, anesthesiologists demonstrated similar communication patterns in routine simulated cases and routine OR cases, but we found differences between routine OR and simulated cases and crisis simulations. The similarity in teamwork-related communications elicited from anesthesiologists in routine simulated cases, and the real setting lends support for the ecological validity of the simulation environment and its value in teamwork training. Different communication patterns and frequencies under the challenge of a crisis support the use of simulation to assess crisis management skills.
Back to Top | Article Outline

References

1. Lingard LHodges B, Lingard L. Rethinking competence in the context of teamwork. The Question of Competence: Reconsidering Medical Education in the Twenty-first Century. 2012 New York Cornell University Press:42–69 . Edited by

2. Salas E, Sims DE, Burke CS. Is there a “Big Five” in teamwork? Small Group Res. 2005;36:555–99

3. Salas E, Wilson KA, Burke CS, Priest HA. Using simulation-based training to improve patient safety: What does it take? Jt Comm J Qual Patient Saf. 2005;31:363–71

4. Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesth Analg. 2006;102:853–8

5. McIvor W, Burden A, Weinger MB, Steadman R. Simulation for maintenance of certification in anesthesiology: The first two years. J Contin Educ Health Prof. 2012;32:236–42

6. Kirlik A. Brunswikian theory and method as a foundation for simulation-based research on clinical judgment. Simul Healthc. 2010;5:255–9

7. Reliability and Validity in Neuropsychological Assessments. 2000 New York Kluwer Academic/Plenum

8. Manser T, Dieckmann P, Wehner T, Rallf M. Comparison of anaesthetists’ activity patterns in the operating room and during simulation. Ergonomics. 2007;50:246–60

9. Weller JM, Merry AF, Robinson BJ, Warman GR, Janssen A. The impact of trained assistance on error rates in anaesthesia: A simulation-based randomised controlled trial. Anaesthesia. 2009;64:126–30

10. Merry AF, Weller JM, Robinson BJ, Warman GR, Davies E, Shaw J, Cheeseman JF, Wilson LF. A simulation design for research evaluating safety innovations in anaesthesia. Anaesthesia. 2008;63:1349–57

11. Merry AF, Webster CS, Hannam J, Mitchell SJ, Henderson R, Reid P, Edwards KE, Jardim A, Pak N, Cooper J, Hopley L, Frampton C, Short TG. Multimodal system designed to reduce errors in recording and administration of drugs in anaesthesia: Prospective randomised clinical evaluation. BMJ. 2011;343:d5543

12. Manser T, Wehner T. Analysing action sequences: Variations in action density in the administration of anaesthesia. Cognition, Technology & Work. 2002;4:71–81

13. Held J, Manser T. A PDA-based system for online recording and analysis of concurrent events in complex behavioral processes. Behav Res Methods. 2005;37:155–64

14. Helmreich R. Threat and error in aviation and medicine: Similar and different, Special Medical Seminar, Lessons for Health Care: Applied Human Factors Research, Australian Council of Safety and Quality in Health Care & NSW Ministerial Council for Quality in Health Care. 2000

15. Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS System: Reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care. 2009;18:104–8

16. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. Br J Anaesth. 2003;90:580–8

17. Manser T, Harrison TK, Gaba DM, Howard SK. Coordination patterns related to high clinical performance in a simulated anesthetic crisis. Anesth Analg. 2009;108:1606–15

18. Kane M. Current concerns in validity theory. J Educ Meas. 2001;38:319–42

19. Linn R Validity, Educational Measurement. 19893rd edition New York American Council on Education and Macmkllan:13–103 . Edited by

20. Devitt JH, Kurrek MM, Cohen MM, Cleave-Hogg D. The validity of performance assessments using simulation. ANESTHESIOLOGY. 2001;95:36–42

21. Boulet JR, Murray D, Kras J, Woodhouse J, McAllister J, Ziv A. Reliability and validity of a simulation-based acute care skills assessment for medical students and residents. A. 2003;99:1270–80

22. Manser T, Dieckmann P, Wehner T, Rallf M. Comparison of anaesthetists’ activity patterns in the operating room and during simulation. Ergonomics. 2007;50:246–60

© 2014 American Society of Anesthesiologists, Inc.

Publication of an advertisement in Anesthesiology Online does not constitute endorsement by the American Society of Anesthesiologists, Inc. or Lippincott Williams & Wilkins, Inc. of the product or service being advertised.
Login

Article Tools

Images

Share