Journal Logo

Empirical Investigations

Teamwork Assessment Tools in Obstetric Emergencies

A Systematic Review

Onwochei, Desire N. MBBS, BSc (Hons), FRCA; Halpern, Stephen MD, FRCP, MSc; Balki, Mrinalini MD

Author Information
doi: 10.1097/SIH.0000000000000210
  • Free

Abstract

Obstetric emergencies are commonplace on any labor and delivery ward. Such emergencies comprise, among others, postpartum hemorrhage, emergency cesarean section, and cardiorespiratory arrest. The management of these crises requires multidisciplinary collaboration including obstetrics, nursing, neonatology, and anesthesiology. Interdisciplinary collaboration is a regular daily occurrence in the labor ward environment; however, breakdown in communication can be a major problem, especially in emergency situations where it is most needed. Ineffective communication and inadequate care may cause poor maternal and fetal outcomes. High-quality care is evident when there is good communication and multidisciplinary information sharing.1 In addition, the Institute of Medicine identifies team-based training and simulation as methods to improve patient safety.2

Simulation, modelled on Crew Resource Management (CRM) developed by the aviation industry, has been used as an essential training tool by multidisciplinary medical teams. It has been shown to improve team building, manage fatigue and adverse events, and improve communication, decision making, and performance.3 In many specialties, including obstetrics, team simulation exercises can prove invaluable.4 Means of assessing the communication, behavioral, and nontechnical skills of a team are necessary, because these are important factors in team interactions. Many teamwork assessment tools exist to try to quantify the adequacy of teams working in various scenarios. However, whether or not a specific tool is better adapted to teamwork assessment in obstetric emergency scenarios is unclear and no general consensus exists.

The primary objective of this qualitative systematic review is to find the tools available to assess team effectiveness in obstetric emergencies. The secondary objective is to compare the strengths and weaknesses of these tools.

METHODS

Search Strategy

We searched the literature for prospective studies that evaluated nontechnical skills in multidisciplinary teams involving obstetric emergencies. Only studies describing the use of a specific evaluation tool or method were considered. Those evaluating only technical skills or nontechnical skills of individuals, rather than teams, were excluded. We searched Embase, Medline, PubMed, Web of Science, PsycINFO, CINAHL, and Google Scholar, using the following key words and strings: “group processes,” “team,” “nontechnical skills,” “nontechnical skills,” “obstetrics,” and “pregnancy.” The search included studies from 1944 until the present. The last search was conducted on January 11, 2016. Abstracts of scientific meetings and non-English studies were excluded. Opinion pieces and editorials were also excluded, unless they included new empirical evidence. Where appropriate, we attempted to contact the corresponding author for missing information. Each article that met the inclusion criteria was evaluated independently by one investigator (D.O.) and reviewed by another two (S.H. and M.B.). Disagreements were resolved by discussion with all investigators.

Outcomes

Data were collected on study population, study intervention, objectives, measurement tool used, and domains addressed by said tool. Data on any reliability and validity measures were collected and used for interpretation (Table 1),5–7 in addition to the study scenarios, number of raters, and type of rater training.

TABLE 1
TABLE 1:
Interpretation of Statistical Values for Reliability

Quality

The PRISMA guidelines8 were applied to the conduct of this systematic review. In addition, identified studies meeting the inclusion criteria were assessed for risk of bias, by means of quality, as per the PRISMA checklist.9 We used the following parameters as markers of quality: number of raters and whether raters were multidisciplinary, simulations involving appropriate scenarios with anesthesiology involvement, whether the tool had been used in other practices, and if any evidence of improved patient outcomes existed with its use or development.

Data Analysis

A descriptive analysis was performed, because quantitative meta-analysis would be inappropriate, given the nature of the data.

RESULTS

A total of 15,333 records were identified and 6168 were screened after removal of duplicates and non-English studies (Fig. 1). Seventy-one full text articles were assessed for eligibility. Thirteen studies fulfilled the inclusion criteria and were included in the final qualitative synthesis,10–22 and the rest were excluded (Appendix 1). All the studies assessed teams in the context of simulation scenarios and used some form of teamwork assessment tool (Table 2).10–22 Nine named tools were identified from the following thirteen studies: the Observational Teamwork Assessment for Surgery-Spanish version (OTAS-S),11 the Situational Awareness Global Assessment Technique (SAGAT),18 the Team Emergency Assessment Measure (TEAM),21 the Clinical Teamwork Scale (CTS),12,15 the Assessment of Obstetric Team Performance (AOTP) and Global AOTP (GAOTP),10 the Team Attitudes Questionnaire (TAQ),19 the Human Factor Rating Scale (HFRS), and the Global Rating Scale of performance (GRS).17 Two of the studies did not give an official title for the tool,20,22 and three studies did not specify the tool used.13,14,16

FIGURE 1
FIGURE 1:
Flowchart of systematic research.
TABLE 2
TABLE 2:
Summary of Studies on Teamwork Assessment Tools for Obstetric Emergencies

Scenarios

The simulation scenarios were based around the management of obstetric emergencies, with common themes being pre-eclampsia, postpartum hemorrhage, and maternal cardiac arrest. The study by Amaya-Arias et al11 was the only one that did not specifically mention the scenarios undertaken. The setup of simulation, in all the studies, was for teams to undertake one or more scenarios. Four studies used shoulder dystocia and eclampsia scenarios that were tailored more toward obstetric practice rather than anesthetic practice.12,13,20,21 All the studies undertook scenarios with some degree of multidisciplinary team involvement; however, only six of the thirteen studies specified that anesthesiologists were involved in the simulations.10,11,14,17,18,21 In general, teams consisted of physicians and nurses at varying levels of training. One study only included students19 and one used actors playing team member roles, which were not elaborated upon.12

Raters

Team assessment was undertaken using videos of the simulations in all of the studies except one.11 A minimum of two raters assessed teams in each study. The disciplines of the raters were not truly multidisciplinary but mirrored the composition of the simulation teams. In only one study were raters not mentioned at all.11 Training of raters was mentioned in all studies except three.11,16,18 Formal training was described as either courses or 3 to 8 hours of instructional orientations or workshops in four of the studies.10,19,21,22 Rater training was mentioned but not described in three studies.13,15,20 One study explained that raters were trained in crisis resource management but not extensively in the use of the teamwork assessment tool.12 In two of the studies,14,17 rater training was not undertaken, but raters were selected according to previous experience in simulation or based on their expertise.

Objective of the Studies

The objective of nine of the studies was either to evaluate team training and improve teamwork, performance, and attitudes11,13,15,16,19–22 or to develop and validate a specific simulation scenario.14 In only four of the studies10,12,17,18 was the objective to evaluate a specific teamwork assessment tool. Regardless of the objective, seven of the studies performed some form of assessment on the tool10,12,15,17–20 (Table 3).10–22 Five studies evaluated the teamwork tool using just reliability measures,10,15,17–19 one using just validity measures,20 and one used both.12 None of the studies measured the quality of the tool with patient outcomes. One study mentioned promoting better patient outcomes but did not go so far as exhibiting evidence for this.19 Siassakos et al20 attempted to relate team behaviors to patient outcomes; however, the marker for this was the administration of magnesium, a clinical focus rather than a nontechnical skill.

TABLE 3
TABLE 3:
Domains, Measures, Reliability, and Validity of Teamwork Tools

Use of Tools in Other Practices

The OTAS-S11 is the Spanish version of the OTAS, which has previously been used and validated within operating theater teams.25 It was specifically developed for the perioperative period but focuses on teamwork occurring within anesthesiology, surgery, and nursing,26 instead of teamwork as one group. The SAGAT focuses specifically on situational awareness within a simulation environment27 and has been used in the training of medical practitioners, trauma trainees, and anesthesiologists.28–30 The TEAM tool, developed by Cooper et al24 to assess the performance of resuscitation teams, similarly, has mostly been used outside of the obstetric context. The CTS was developed by the State Obstetric and Pediatric Research Collaboration Obstetric Safety Initiative Team31 specifically to assess the performance of obstetric team but has not been extensively used outside of obstetric practice. The AOTP and GAOTP are relatively new tools that have been targeted toward the formative and summative assessment of obstetric team performance, respectively,32 and have not been widely applied to other practices. The TAQ was developed by the Agency for Healthcare Research and Quality, based on the Team Strategies and Tools to Enhance Performance and Patient Safety model.33 It has been used within a military setting34 but rarely within obstetrics. The HFRS was adapted from the Operating Room Management Attitudes Questionnaire35 for use within an obstetric context.17 Again, it has not been widely used in an obstetric context. The untitled tool used by Walker et al22 was based on key skills developed by the Centre for Advanced Pediatric and Perinatal Education.23 Further information on the tool could not be obtained from the reference given, and attempts to contact the authors were unsuccessful. The untitled teamwork measurement tool used in the study by Siassakos et al20 was developed specifically for the study, with the aim of describing the teamwork aspects relating to clinical efficiency in obstetric emergencies. Hence, this has not been used outside of this context.

Performing a Web of Science citation search revealed that with the exception of the OTAS and SAGAT tools, which have been extensively cited outside the context of obstetrics within the literature, none of the other tools have been extensively cited for new research.

Commonly Included Domains in Teamwork Assessment

The domains assessed by the teamwork tools in the thirteen studies were aspects of nontechnical skills. Most incorporated communication,10–12,14–17,19,20,22 leadership and role responsibility,11,12,14,15,17,19–22 and situational awareness.10–12,19,20,22 Other domains included coordination, supervision, teamwork, task management, error, and decision making.

A 5-point scale was used for the tool in the following five of the studies: for the AOTP and GAOTP,10 an unspecified tool,13 the HFRS and GRS,17 an untitled teamwork measurement tool,20 and TEAM.21 A 7-point Likert scale was used for the OTAS-S,11 an 11-point rating scale for the CTS in two studies,12,15 and a dichotomous scoring system for the SAGAT.18 Subjective assessment was used for the unspecified tool in one study,16 and no measures were mentioned in three of the studies.14,19,22

Reliability Measures

Table 3 shows the reliability and validity scores where they were used in the studies. Where reliability testing of team assessment tools was performed, the interrater agreement was measured using the κ statistic and Kendall coefficient.15 One study used the κ statistic to measure intrateam consistency.18 Interrater reliability was measured using the intraclass correlation coefficient (ICC)10,12,15,17,19 and internal consistency using Cronbach α.10,17,19 Three studies also measured correlation using the Pearson coefficient.10,12,17

Morgan et al18 found poor intrateam consistency for the SAGAT (Table 3). Morgan et al10 found that the GAOTP had high interrater reliability with at least eight raters. Moderate test-retest reliability was also found with this tool. The authors found high internal consistency with the GAOTP and AOTP, both individually and when the two tools were combined. Posmontier et al19 provided Cronbach α scores for internal consistency for each of the domains measured by the TAQ. The scores suggested acceptable internal consistency for team structure, good to excellent internal consistency for leadership and situation monitoring, questionable internal consistency for mutual support, and poor internal consistency for communication.

Morgan et al17 presented low interrater reliability scores for the HFRS and GRS for single raters. Self-assessment single rater scores suggested good interrater reliability for the GRS in this area. The GRS showed slightly higher internal consistency than the HFRS. There was little correlation for external versus self-evaluation using the HFRS and GRS, but the two scales correlated well together. The authors point out that nurses rated scores higher than physicians, which was statistically significant for the HFRS but not the GRS. Guise et al12 describe substantial interrater agreement and high concordance for the CTS, as well as high interrater reliability. A Pearson coefficient of scored ratings demonstrated high correlation.

Marshall et al16 mentioned reliability measures using the κ statistic and Cronbach α for their unspecified tool but did not provide results.

Validity Measures

Siassakos et al20 compared their untitled teamwork measurement tool with the Weller generic teamwork score.36 Construct validity was measured using Kendall τb and demonstrated general and significant correlation for global situational awareness and supportive language, respectively. Guise et al12 did not specify a statistical variable to determine construct validity but examined the distribution and median score for each scenario, according to its predetermined teamwork level. In addition, the authors state that raters were able to rate almost every item with high accuracy, suggesting that the CTS was easy to use. Although some of the other tools, such as TEAM and TAQ, have been shown to demonstrate validity in other clinical environments,24,34,37 the authors of the reviewed studies did not undertake measures of validity when using this tool in their obstetric scenarios.

DISCUSSION

Team collaboration is essential in obstetric medicine, with multidisciplinary involvement forming the basis of any labor and delivery environment. The management of obstetric emergencies, therefore, not only requires technical ability but also requires a range of nontechnical skills, such as communication, leadership, and situational awareness. Simulation has become a useful tool for team training, having been shown to improve the nontechnical aspects of teamworking,3 and is highly advocated by the Institute of Medicine.2 Teamwork assessment tools for nontechnical skills are available for many areas of medicine, yet within obstetrics, consensus is lacking on which tool is best. Reliability and validity are important aspects to consider when determining the best tools.

The reliability of a tool describes how reproducible it is and may refer to test-retest reliability or interrater agreement.38 Test-retest reliability looks at how reliable the tool is on two occasions separated by a certain time interval, such as a self-test where there are no raters.39 Interrater agreement looks at the homogeneity of rater scores and is generally quantified using the ICC, which is better when there are more than two raters; the κ statistic, a variant of the ICC; or Pearson product-moment correlation. To an extent, interrater agreement depends on how much training the raters have received.39 It can also depend on how clearly the scale is defined and how easily it can be used. Reliability can also be measured by internal consistency, classically described by the Cronbach α, and is influenced by the number of items the tool assesses. The ideal teamwork assessment tool should display high reliability, with the ICC recommended as the superior test of “external reliability,” because it takes into account the agreement that could occur by chance and allows determination of the tool's ability to differentiate between candidates.38,39 Internal consistency may also be used to describe the reliability of the ideal tool, but because this measure can be applied with only one “sitting” of the tool's use and does not include all sources of variance, values should be interpreted carefully.39 When raters are trained, checklists and global rating scales tend to display excellent reliability, markedly so in the framework of didactic teaching and simulation.38 Therefore, an ideal tool would do well to incorporate such tests.

The validity of a tool defines whether the tool is actually measuring what it is meant to measure and it includes content validity, construct validity, and face validity. Content validity ensures all important domains are measured in the tool and the domains are often derived using expert opinion.39 Construct validity describes whether the tool is able to differentiate between candidates of varying capabilities. Within the context of teamwork assessment, this also means that the tool should show that candidates do better clinically after undergoing training and, hence, patient outcomes improve. Therefore, an ideal tool should display a high degree of construct validity. Face validity simply reflects whether people think the tool is “good enough for its job” and should include the opinions of the candidates, not just the raters. Checklists and global rating scales tend to display good construct validity, but face validity tends to vary depending on the type of simulation scenario used. This can present a challenge when designing and testing the ideal teamwork assessment tool.38

Other than demonstrating reliability and validity, the ideal tool must encompass the proper domains to assess the nontechnical skills of multidisciplinary teams. The main domains are communication, leadership, mutual support, and situational awareness, derived from CRM.2,3 The Team Strategies and Tools to Enhance Performance and Patient Safety model, a widely implemented curriculum in this area, incorporates these principles and provides evidence base for them.40,41 The ideal tool should certainly cover these domains, but there is some argument to adding others that would be relevant to the multidisciplinary team management of obstetric emergencies, such as role clarity40 and decision making, which could essentially form subdomains.

In terms of quality, the ideal tool should be tested within the appropriate context. All disciplines, encompassing obstetrics, anesthesiology, nursing, and neonatology, should be represented as both candidates and raters. Therefore, the scenarios used to test the tool must reflect the multidisciplinary environment of the labor and delivery ward. Multidisciplinary raters must be adequately trained in the use of the tool, and there should ideally be more than two raters, helping to ensure reliability. An agreed set of quality measures to grade teamwork assessment tools would be valuable and is lacking. The development of such a grading system would further improve tool structure and inform design of the ideal tool, which is an area to be explored. Designing a tool specifically for the context of obstetric emergencies is certainly feasible; however, a well-designed tool should have applicability across other specialties when assessing different scenarios, a trait that few of the identified tools in this review displayed. Finally, the ideal tool should provide evidence of improved patient outcomes, such that training with the tool has actual clinical value to the participants being assessed, with results and feedback leading to improved clinical practice.

The nine named teamwork assessment tools identified by this review were the OTAS-S,11 SAGAT,18 TEAM,21 CTS,12,15 AOTP and GAOTP,10 TAQ,19 HFRS, and GRS.17 In addition, two untitled tools were also found.20,22 None of the studies satisfied all of the quality markers with the use of the associated teamwork tool. The strongest tools, in terms of rater credentials and multidisciplinary scenarios, were TEAM, AOTP, and GAOTP. However, only TEAM has been used in other clinical settings and no patient outcome improvement had been demonstrated with any of the tools. The strengths of the most applicable tools found—CTS, GAOTP, and GRS—lie in the addressed domains, the ease of use, and the reliability measures performed. The benefit of the GAOTP and GRS, however, is that anesthesiology involvement occurred within the scenarios. The main weakness of these two tools is that no validity measures were performed. The advantage of the CTS is that construct validity was demonstrated in addition to the reliability measures, but unfortunately, anesthesiologists were not part of the scenarios.

Overall, issues exist with all of the identified tools with regard to reliability and validity. Although many of the studies made efforts to describe some aspects of reliability, the main issue was that most lacked any type of validity measure. Furthermore, many of the domains described by the tools were not clear, and rater training was not consistently adequate. The running theme identified was that where a tool demonstrated high quality in one area, it had poor quality in another. Therefore, none of the tools measure up to the concept of the ideal teamwork assessment tool in this context. Simulation itself is associated with improved patient outcomes, which is an important factor that informs investment in training programs,42 but the use of these tools was not presented with any such evidence. Methodology for determining patient outcome improvement with team training43,44 could be applied to teamwork assessment tool studies, although controlling for confounding factors may present a challenge.

The limitations of this systematic review include those associated with the studies themselves and those associated with the review process. The influence of publication bias cannot be overlooked, considering the possibility that studies with lesser impact may not have been published. Bias within the studies themselves may also limit the findings of the review, which we attempted to overcome using prespecified quality parameters. With regard to the review process, excluding abstracts and non-English studies may have affected the number of included studies; however, the rationale behind this was to ensure that we could obtain published data from fully reported studies and avoid the potential difficulties of translating qualitative reports. Another important point to consider is that teamwork assessment domains, such as communication, leadership, and situational awareness, are not exclusive characteristics to teamwork within obstetric emergencies. These domains apply to teams across all medical specialties. However, we restricted our review to obstetric emergencies given the subspecialty interests of the authors and associated knowledge limitation in other subspecialties. Moreover, specific teamwork assessment domains in other specialties may have more or less significance than those identified in obstetric emergencies, a point that is beyond the scope of this review.

CONCLUSIONS

The current tools available for team assessment in obstetric emergencies are lacking in some areas, whether it be full reliability and validity scoring, domains assessment, or multidisciplinary applicability. More work needs to be conducted to establish the validity of teamwork tools for nontechnical skills, and the development of an ideal tool is warranted. In addition, an agreed set of defined quality measures to grade teamwork assessment tools would be valuable, not only within obstetric emergencies, but across all medical specialties. Further studies are required to assess how outcomes, such as performance and patient safety, are influenced when using teamwork assessment tools.

REFERENCES

1. MBBRACE-UK. Saving Lives, Improving Mothers’ Care: Surveillance of Maternal Deaths in the UK 2011–13 and Lessons Learned to Inform Maternity Care from the UK and Ireland Confidential Enquiries into Maternal Deaths and Morbidity 2009–13; 2015.
2. Institute of Medicine (US) Committee on Quality of Health Care in America; Kohn LT, Corrigan JM, Donaldson MS. To Err Is Human: Building a Safer Health System. Washington, DC: National Academies Press; 2000.
3. Grogan EL, Stiles RA, France DJ, et al. The impact of aviation-based teamwork training on the attitudes of health-care professionals. J Am Coll Surg 2004;199:843–848.
4. Guise J, Segel S. Teamwork in obstetric critical care. Best Pract Res Clin Obstet Gynaecol 2008;22:937–951.
5. Cicchetti D. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 1994;6:284–290.
6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174.
7. George D, Mallery P. SPSS for Windows Step by Step: A Simple Guide and Reference, 11.0 Update. Boston: Allyn and Bacon; 2003.
8. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535.
9. Liberati A, Altman D, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700.
10. Morgan PJ, Tregunno D, Pittini R, et al. Determination of the psychometric properties of a behavioural marking system for obstetrical team training using high-fidelity simulation. BMJ Qual Saf 2012;21:78–82.
11. Amaya-Arias AC, Idarraga D, Giraldo V, Gómez LM. Effectiveness of a program for improving teamwork in Operanting Rooms. Colomb J Anesthesiol 2015;43:68–75.
12. Guise JM, Deering SH, Kanki BG, et al. Validation of a tool to measure and promote clinical teamwork. Simul Healthc 2008;3:217–223.
13. Ellis D, Crofts JF, Hunt LP, Read M, Fox R, James M. Hospital, simulation center, and teamwork training for eclampsia management: a randomized controlled trial. Obstet Gynecol 2008;111:723–731.
14. Fialkow MF, Adams CR, Carranza L, Golden SJ, Benedetti TJ, Fernandez R. An in situ standardized patient-based simulation to train postpartum hemorrhage and team skills on a labor and delivery unit. Simul Healthc 2014;9:65–71.
15. Fransen AF, Van De Ven J, Merién AER, et al. Effect of obstetric team training on team performance and medical technical skills: a randomised controlled trial. BJOG 2012;119:1387–1393.
16. Marshall NE, Vanderhoeven J, Eden KB, Segel SY, Guise JM. Impact of simulation and team training on postpartum hemorrhage management in non-academic centers. J Matern Fetal Neonatal Med 2015;28:495–499.
17. Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology 2007;106:907–915.
18. Morgan P, Tregunno D, Brydges R, et al. Using a situational awareness global assessment technique for interprofessional obstetrical team training with high fidelity simulation. [Erratum appears in J Interprof Care. 2015 Mar;29(2):175; PMID: 25559685]. J Interprof Care 2015;29:13–19.
19. Posmontier B, Montgomery K, Smith Glasgow ME, Montgomery OC, Morse K. Transdisciplinary teamwork simulation in obstetrics-gynecology health care education. J Nurs Educ 2012;51:176–179.
20. Siassakos D, Bristowe K, Draycott TJ, et al. Clinical efficiency in a simulated emergency and relationship to team behaviours: a multisite cross-sectional study. BJOG 2011;118:596–607.
21. Sørensen JL, van der Vleuten C, Rosthøj S, et al. Simulation-based multiprofessional obstetric anaesthesia training conducted in situ versus off-site leads to similar individual and team outcomes: a randomised educational trial. BMJ Open 2015;5:e008344.
22. Walker D, Cohen S, Fritz J, et al. Team training in obstetric and neonatal emergencies using highly realistic simulation in Mexico: impact on process indicators. BMC Pregnancy Childbirth 2014;14:367.
23. Center for Advanced Pediatrics & Perinatal Education (CAPE): Simulation-based training: Key behavioral skills. 2012. Available at: http://cape.lpch.org/courses/logistics/skills.html. Accessed April 24, 2016.
24. Cooper S, Cant R, Porter J, Sellick K, Kinsman L, Nestel D. Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). Resuscitation 2010;81:446–452.
25. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg 2011;212:234–243.
26. Undre S, Healey AN, Darzi A, Vincent CA. Observational assessment of surgical teamwork: a feasibility study. World J Surg 2006;30:1774–1783.
27. Endsley MR. Measurement of situation awareness in dynamic systems. Hum Factors J Hum Factors Ergon Soc 1995;37:65–84.
28. Wright MC, Taekman JM, Endsley MR. Objective measures of situation awareness in a simulated medical environment. Qual Saf Health Care 2004;13(Suppl 1):i65–i71.
29. Hogan MP, Pace DE, Hapgood J, Boone DC. Use of human patient simulation and the situation awareness global assessment technique in practical trauma skills assessment. J Trauma 2006;61:1047–1052.
30. Zhang Y, Drews FA, Westenskow DR, et al. Effects of integrated graphical displays on situation awareness in anaesthesiology. Cogn Technol Work 2002;4:82–90.
31. State Obstetric and Pediatric Research Collaboration (STORC). OB Safety Initiative - Home 2007. Available at: http://www.obsafety.org/. Accessed April 28, 2016.
32. Tregunno D, Pittini R, Haley M, Morgan PJ. Development and usability of a behavioural marking system for performance assessment of obstetrical teams. Qual Saf Health Care 2009;18:393–396.
33. Agency for Healthcare Research & Quality. Teamwork Attitudes Questionnaire (T-TAQ). 2014. Available at: http://www.ahrq.gov/professionals/education/curriculum-tools/teamstepps/instructor/reference/teamattitude.html. Accessed May 2, 2016.
34. Baker DP, Amodeo AM, Krokos KJ, Slonim A, Herrera H. Assessing teamwork attitudes in healthcare: development of the TeamSTEPPS teamwork attitudes questionnaire. Qual Saf Health Care 2010;19:e49.
35. Sexton J, Helmreich R, Glenn D, Wilhelm J, Merritt A. Operating Room Management Attitudes Questionnaire (ORMAQ). Human Factors Research Project: University of Texas at Austin; 2000.
36. Weller JM, Bloch M, Young S, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. Br J Anaesth 2003;90:43–47.
37. McKay A, Walker ST, Brett SJ, Vincent C, Sevdalis N. Team performance in resuscitation teams: comparison and critique of two recently developed scoring tools. Resuscitation 2012;83:1478–1483.
38. Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in anaesthesia. Br J Anaesth 2009;103:472–483.
39. Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. 4th ed. Oxford, UK: University Press Scholarship Online; 2008.
40. Deering S, Johnston LC, Colacchio K. Multidisciplinary teamwork and communication training. Semin Perinatol 2011;35:89–96.
41. Agency for Healthcare Research and Quality. TeamSTEPPS 2.0. 2015. Available at: http://www.ahrq.gov/professionals/education/curriculum-tools/teamstepps/instructor/fundamentals/index.html. Accessed June 6, 2016.
42. Draycott TJ, Collins KJ, Crofts JF, et al. Myths and realities of training in obstetric emergencies. Best Pract Res Clin Obstet Gynaecol 2015;29:1067–1076.
43. Bergh AM, Baloyi S, Pattinson RC. What is the impact of multi-professional emergency obstetric and neonatal care training? Best Pract Res Clin Obstet Gynaecol 2015;29:1028–1043.
44. van de Ven J, Houterman S, Steinweg RA, et al. Reducing errors in health care: cost-effectiveness of multidisciplinary team training in obstetric emergencies (TOSTI study); a randomised controlled trial. BMC Pregnancy Childbirth 2010;10:59.

APPENDIX 1. Full-text articles assessed for eligibility and excluded.

No mention of teamwork tool

  1. Andreatta P, Frankel J, Boblick Smith S, Bullough A, Marzano D. Interdisciplinary team training identifies discrepancies in institutional policies and practices. Am J Obstet Gynecol. 2011;205:298–301.
  2. Gum L, Greenhill J, Dix K. Clinical simulation in maternity (CSiM): interprofessional learning through simulation team training. Qual Saf Health Care. 2010;19:e19.
  3. Haller G, Garnerin P, Morales MA, et al. Effect of crew resource management training in a multidisciplinary obstetrical setting. Int J Qual Heal Care. 2008;20:254–263.
  4. Haller G, Morales M, Pfister R, et al. Improving interprofessional teamwork in obstetrics: a CRM based training programme. J Interprof Care. 2008;22:545–548.
  5. Mackintosh N, Berridge EJ, Freeth D. Supporting structures for team situation awareness and decision making: insights from four delivery suites. J Eval Clin Pract. 2009;15:46–54.
  6. Madden E, Sinclair M, Wright M. Teamwork in obstetric emergencies. Evid Based Midwifery. 2011;9:95–101
  7. Miller KK, Riley W, Davis S, Hansen HE. In situ simulation: a method of experiential learning to promote safety and team behavior. J Perinat Neonatal Nurs. 2008;22:105–113.
  8. Minehart RD, Pian-Smith MC, Walzer TB, et al. Speaking across the drapes: communication strategies of anesthesiologists and obstetricians during a simulated maternal crisis. Simul Healthc J Soc Med Simul. 2012;7:166–170.
  9. Nelissen E, Ersdal H, Ostergaard D, et al. Helping mothers survive bleeding after birth: an evaluation of simulation-based training in a low-resource setting. Acta Obstet Gynecol Scand. 2014;93:287–295.
  10. Nielsen PE, Goldman MB, Mann S, et al. Effects of teamwork training on adverse outcomes and process of care in labor and delivery: a randomized controlled trial. Obstet Gynecol. 2007;109(1):48–55.
  11. Noblot E, Raia-Barjat T, Lajeunesse C, et al. Training program for the management of two obstetric emergencies within a French perinatal care network. Eur J Obstet Gynecol Reprod Biol. 2015;189:101–105.
  12. Phipps MG, Lindquist DG, McConaughey E, O'Brien JA, Raker CA, Paglia MJ. Outcomes from a labor and delivery team training program with simulation component. Am J Obstet Gynecol. 2012;206:3–9.
  13. Riley W, Davis S, Miller KM, et al. Detecting breaches in defensive barriers using in situ simulation for obstetric emergencies. Qual Saf Health Care. 2010;19 (Suppl 3):i53–i56.
  14. Robertson B, Schumacher L, Gosman G, et al. Simulation-based crisis team training for multidisciplinary obstetric providers. Simul Healthc 2009;4:77–83.
  15. Shaw-Battista J, Belew C, Anderson D, van Schaik S. Successes and challenges of interprofessional physiologic birth and obstetric emergency simulations in a nurse-midwifery education program. J Midwifery Womens Heal. 2015;60:735–743.
  16. Siassakos D, Draycott T, Montague I, Harris M. Content analysis of team communication in an obstetric emergency scenario. J Obstet Gynaecol. 2009;29:499–503.
  17. Singh A, Nandi L. Obstetric emergencies: role of obstetric drill for a better maternal outcome. J Obstet Gynaecol India. 2012;62:291–296.
  18. Sonesh SC, Gregory ME, Hughes AM, et al. Team training in obstetrics: a multi-level evaluation. Fam Syst Heal. 2015;33(3):250–261.
  19. Strachan B. How effective is training to help staff deal with obstetric emergencies. J Health Serv Res Policy. 2010;15(Suppl 1):37–39.
  20. Walker DM, Cohen SR, Estrada F, et al. PRONTO training for obstetric and neonatal emergencies in Mexico. Int J Gynaecol Obstet. 2012;116:128–133.
  21. Yeboah-Antwi K, Snetro-Plewman G, Waltensperger KZ, et al. Measuring teamwork and taskwork of community-based “teams” delivering life-saving health interventions in rural Zambia: a qualitative study. BMC Med Res Methodol. 2013;13:84.
  22. Zabari M, Suresh G, Tomlinson M, et al. Implementation and case-study results of potentially better practices for collaboration between obstetrics and neonatology to achieve improved perinatal outcomes. Pediatrics. 2006;118(Suppl):S153–S158.

Assessing individuals

  1. Freeth D, Ayida G, Berridge EJ, et al. Multidisciplinary obstetric simulated emergency scenarios (MOSES): promoting patient safety in obstetrics with teamwork-focused interprofessional simulations. J Contin Educ Health Prof. 2009;29:98–104
  2. Jackson S, Brackley K, Landau A, Hayes K. Assessing nontechnical skills on the delivery suite: a pilot study. Clin Teach. 2014;11:375–380.
  3. Miller K, Riley W, Davis S. Identifying key nursing and team behaviors to achieve high reliability. J Nurs Manag. 2009;17:247–255.
  4. Monod C, Voekt CA, Gisin M, Gisin S, Hoesli IM. Optimization of competency in obstetrical emergencies: a role for simulation training. Arch Gynecol Obstet. 2014;289:733–738.
  5. Guise JM, Lowe NK, Deering S, et al. Mobile in situ obstetric emergency simulation and teamwork training to improve maternal-fetal safety in hospitals. Jt Comm J Qual Patient Saf. 2010;36:443–453.

Not assessing obstetric emergencies

  1. Riley W, Davis S, Miller K, Hansen H, Sainfort F, Sweet R. Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Jt Comm J Qual Patient Saf. 2011;37:357–364.
  2. Rovamo L, Nurmi E, Mattila MM, Suominen P, Silvennoinen M. Effect of a simulation-based workshop on multidisciplinary teamwork of newborn emergencies: an intervention study. BMC Res Notes. 2015;8:671.
  3. Zevin B, Levy JS, Satava RM, Grantcharov TP. A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery. J Am Coll Surg. 2012;215(4):580–586.

Retrospective

  1. Deering S, Rosen MA, Salas E, King HB. Building team and technical competency for obstetric emergencies: the mobile obstetric emergencies simulator (MOES) system. Simul Healthc. 2009;4:166–173.

Articles/reviews

  1. Tregunno D, Pittini R, Haley M, Morgan PJ. Development and usability of a behavioral marking system for team performance assessment of obstetrical teams. Qual Saf Health Care. 2009;18:393–396.
  2. Banga FR, Truijens SEM, Fransen AF, et al. The impact of transmural multiprofessional simulation-based obstetric team training on perinatal outcome and quality of care in the Netherlands. BMC Med Educ. 2014;14(1):175.
  3. Auguste TC, Goffman D, Deering S, et al. Simulation-based team training in obstetric emergencies. Contemp Ob Gyn. 2012;57:30–39.
  4. Cameron M, Hinshaw K. A systematic review of training in acute obstetric emergencies. BJOG. 2004;111:288.
  5. de Mello WF, O'Flaherty D. The importance of teamwork in obstetrics. Br J Hosp Med. 1992;48:776.
  6. Deering S, Johnston LC, Colacchio K. Multidisciplinary teamwork and communication training. Semin Perinatol. 2011;35:89–96.
  7. Freeth D, Ayida G, Berridge EJ, Sadler C, Strachan A. MOSES: multidisciplinary obstetric simulated emergency scenarios. J Interprof Care. 2006;20:552–554.
  8. Guise J, Segel S. Teamwork in obstetric critical care. Best Pract Res Clin Obstet Gynaecol. 2008;22:937–951.
  9. Jackson KS, Hayes K, Hinshaw K. The relevance of nontechnical skills in obstetrics and gynaecology. Obstet Gynaecol. 2013;15:269–274.
  10. Merién a ER, van de Ven J, Mol BW, Houterman S, Oei SG. Multidisciplinary team training in a simulation setting for acute obstetric emergencies: a systematic review. Obstet Gynecol. 2010;115(5):1021–1031.
  11. Murray D, Enarson C. Communication and teamwork: essential to learn but difficult to measure. Anesthesiology. 2007;106:895–896.
  12. Rouse DJ. [Commentary on] Effects of teamwork training on adverse outcomes and process of care in labor and delivery: a randomized controlled trial. Obstet Gynecol Surv. 2007;62:294–295.
  13. Siassakos D, Draycott TJ, Crofts JF, Hunt LP, Winter C, Fox R. More to teamwork than knowledge, skill and attitude. BJOG 2010;117:1262–9.
  14. Smith A, Edwards S, Siassakos D. Effective team training to improve outcomes in maternal collapse and perimortem caesarean section. Resuscitation. 2012;83:1183–1184.
  15. Sorensen JL, Van der Vleuten C, Lindschou J, et al. “In situ simulation” versus “off-site simulation” in obstetric emergencies and their effect on knowledge, safety attitudes, team performance, stress, and motivation: study protocol for a randomized controlled trial. Trials [Electronic Resour.] 2013;14:220.

Not assessing nontechnical skills

  1. Abha S, Lily N. Obstetric emergencies: role of obstetric drill for a better maternal outcome. J Obstet Gynecol India. 2012;62:291–296.
  2. Banayan J, Blood A, Park YS, Shahul S, Scavone BM. A modified Delphi method to create a scoring system for assessing team performance during maternal cardiopulmonary arrest. Hypertens Pregnancy. 2015;34:314–331.
  3. Dadiz R, Weinschreider J, Schriefer J, et al. Interdisciplinary simulation-based training to improve delivery room communication. Simul Healthc. 2013;8:279–291.
  4. Kurrek MM, Morgan P, Howard S, et al. Simulation as a new tool to establish benchmark outcome measures in obstetrics. PLoS One. 2015;10:e0131064.
  5. Lipman SS, Carvalho B, Cohen SE, Druzin ML, Daniels K. Response times for emergency cesarean delivery: use of simulation drills to assess and improve obstetric team performance. J Perinatol. 2013;33:259–263.
  6. Maslovitz S, Barkai G, Lessing JB, Ziv A, Many A. Recurrent obstetric management mistakes identified by simulation. Obstet Gynecol. 2007;109:1295–1300.
  7. Olson K, Maietta R. Sim Huddles: a team STEPPS approach for emergency preparedness. J Obstet Gynecol Neonatal Nurs. 2014;43:S92.
  8. Pak KM, Hardasmalani M. A multidisciplinary obstetric trauma resuscitation using in situ high-fidelity simulation. Adv Emerg Nurs J. 2015;37:51–57.
  9. Simpson E, Daniel L. Obstetric emergency simulation training: team training for crisis care management. J Obstet Gynecol Neonatal Nurs. 2010;39:S76–S77.
  10. Sorensen A, Poehlman J, Bollenbacher J, et al. Training for teamwork through simulations. BMJ Innov. 2015;1:144.
  11. Strachan B. How effective is training to help staff deal with obstetric emergencies. J Health Serv Res Policy. 2010;15(Suppl 1):37–39.
  12. van der Nelson HA, Siassakos D, Bennett J, et al. Multiprofessional team simulation training, based on an obstetric model, can improve teamwork in other areas of health care. Am J Med Qual. 2014;29:78–82.
Keywords:

Obstetric emergencies; Teamwork; Teamwork assessment; Assessment tools; Nonechnical skills

© 2017 Society for Simulation in Healthcare