Journal Logo

Empirical Investigations

Benefits of Screen-Based Postpartum Hemorrhage Simulation on Nontechnical Skills Training

A Randomized Simulation Study

Michelet, Daphné MD; Barré, Jessy PhD; Job, Anais MD; Truchot, Jennifer MD; Cabon, Philippe PhD; Delgoulet, Catherine PhD; Tesnière, Antoine PhD

Author Information
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare: December 2019 - Volume 14 - Issue 6 - p 391-397
doi: 10.1097/SIH.0000000000000395


Postpartum hemorrhage (PPH) is defined as a postpartum blood loss 500 mL or greater (severe PPH if the blood loss ≥1000 mL), within 24 hours after childbirth (up to 6 weeks) due to, for example, poor contraction of the uterus, uterine laceration, or poor blood clotting.1–4 In France, there are nearly 800,000 births per year, 80 of which result in maternal death. Postpartum hemorrhage is the leading cause with 10 deaths annually.5 Rapid recognition of PPH6,7 but also organizational and human factors, such as communication, stress management, teamwork, and situation awareness, are important elements to consider for optimal management.8 These “nontechnical” skills (NTS) contribute to improved performances and enhanced patient safety.9 Nontechnical skills, though intrinsic to each individual, can be improved with practice through education, practical workshops, or simulations.10 In healthcare, among simulation devices, computer-based technologies have become increasingly prevalent. Screen-based simulation is defined as “a simulation presented on a computer screen using graphical images and text, similar to popular gaming formats, where the operator interacts with the interface using keyboard, mouse, joystick, or other input device.” The programs provide feedback and track the actions of the learners for assessment without the need for an instructor.11 These simulators show many advantages: they are autonomous (no need for an instructor), transportable (a software on a computer), and more affordable than high-fidelity simulation.12,13 More immersive simulators, such as virtual reality, are also available, but they are associated with adverse effects, such as motion sickness. Interest in virtual environments (immersive or not) is illustrated by the great variety of screen-based simulations or serious games available for medical education14 and the growing number of publications assessing their pedagogical benefit.15,16

In the field of obstetrics, simulation has been used to train and acquire technical and nontechnical skills for specific obstetric emergencies, such as shoulder dystocia, postpartum hemorrhage, eclampsia, and cord prolapse.17,18

The objective of this study was to evaluate whether a screen-based simulation specifically designed to train midwifes for nontechnical skills during PPH led to a better performance regarding NTS than the basic screen-based simulation (without NTS training). Positive results could strengthen the benefit of screen-based simulations to acquire and train nontechnical skills in the context of medical emergencies.


This randomized controlled simulation study was performed from November 2017 to June 2018 at Ilumens, a platform of medical simulation in Paris Descartes University. It was approved by the CERAR (French ethics committee for research in anesthesia and critical care). All participants signed an informed written consent. The study was registered on clinical trial. It was part of the MacCoy Critical ANR (national research agency) project (Models for Adaptative feedback enriChment and Orchestration based virtual realitY in Critical situations).


Volunteer participants were recruited from classes of midwifery school from Paris (fifth year) or during their first year of professional activity. All of them received the same courses on PPH management during their academic training. They had limited experience with clinical management of PPH at the time of the simulation sessions. None of the participants had any previous experience of NTS training, high-fidelity simulation, or screen-based simulation. No sample size calculation was performed for this research; a convenience sample was used.

Screen-Based Simulation: Design and Scenarios

The screen-based simulation PerinatSims (Developed by Medusims, partner of the ANR project) used for the experiment was a point-and-click screen-based simulation with a first-person view, including several scenarios on the management of a postpartum hemorrhage.19 The player embodied a midwife evolving in the virtual environment of a birth room. The midwife started in the virtual room with a nurse avatar but without interacting. During the PPH management, she had the possibility to call an obstetrician and anesthetist for help. The midwife could also interact with the mother through dialog boxes and actions were proposed after a mouse click on the parturient (Fig. 1). In addition, a toolbar with graphic representation was available in the software to apply specific interaction (eg, drug administration, blood bank call, patient file, etc). The software scenarios were based on the expertise of a committee of French medical experts (obstetricians, midwives, anesthetists, and human factors/ergonomics researchers) from Cochin, Versailles, Poissy, or Robert Debré Hospitals and Paris Descartes University. The screen-based simulation used in our experimentation included 2 categories of scenarios, with or without additional event requiring specific NTS (leadership, situation awareness, decision-making, and emotion/stress management19,20). At the end of each exercise, an automated feedback focusing on the technical skills of the PPH algorithm management was offered to the learner.

Screenshot of interface and virtual environment. Down, toolbar with graphic representations. From left to right: parturient interaction, hygiene gestures, medical devices and equipment, phone call, drug administration, laboratory analysis, orientation, patient files, time, and exit/end (page 5).


Each participant was exposed individually to 3 scenarios of PPH management. Before the beginning of the simulation, one of the researchers presented the device with a systematic briefing. The participants then had 10 minutes to test and explore the virtual environment. After each scenario/exercise, participants received an in-game automated feedback regarding their technical actions (according to the French National Societies of Anesthesiology and Obstetric Guidelines) with a color code: green (well-performed action), orange (partially performed action), and red (absent or wrong action). The participants were randomly divided in 2 groups ( basic scenarios followed by a verbal postsimulation structured interview on their technical performance (named control group)1 and NTS scenarios followed by a verbal postsimulation structured interview on their technical performances and the NTS (named NTS group).2 To maintain comparability between groups, the postsimulation interview was standardized and lasted 5 to 10 minutes. Laminated interview sheets were used to further assure standardization (see Supplemental Digital Content 1, which contains the 2 laminated sheets, For the control group, the sheet contained the PPH management algorithm from the French societies guidelines. For the NTS group, the sheet contained a table introducing the 4 categories of the Anaesthesia Non-Technical Skills (ANTS21) framework with subcategories (situation awareness, task management, team working, and decision-making22). No specific NTS classification existed for the management of PPH by midwives. Most of the medical NTS classification include similar NTS, and we chose the ANTS classification because it is extensively used in the operative room23,24 and obstetric fields.25,26 The screen-based simulation sessions were performed in an experimentation room at Ilumens Simulation laboratory (Paris Descartes University) on a Desktop Computer (23-inch screen with 1920 × 1080 resolution).

On the same day, all participants individually performed a high-fidelity simulation session on a case of PPH (Fig. 2). During the scenario, 2 confederates were present to interact with the participant, a nurse, and an obstetrician. An anesthetist was available via telephone. The high-fidelity manikin NOELLE S554.100 (Gaumard) and the dedicated UNI software ( Version) were used (see Supplemental Digital Content 2,, which contains the high-fidelity scenario). The sessions were recorded with the help of the SimView software (Laerdal, Norway) using 3 dome cameras and 1 microphone (Axis Communication, Sweden). Before the beginning of the simulation, a briefing on the environment including the manikin, the material, and the possible actions (call for help, intravenous line insertion…) was provided to the participants.

Experimentation design (page 7).

Primary Endpoint: ANTS Scores

Three independent raters (an anesthetist and 2 human factors experts specialized in health sciences) performed the NTS assessment. They were all simulation instructors and had followed a specific training in ANTS scoring. The training consisted of visualization and assessment of at least 50 videos of high-fidelity PPH simulation with midwives before this study started. They were blinded to the allocated group since the analysis took place after the study using video recording of the sessions. The ANTS observation system was used to assess the NTS in various simulation situations and population, ranging from emergencies for medical students27 to neonatal resuscitation for midwives (with a specific modified ANTS version,25). The measurement properties of the ANTS score has been extensively investigated.28 The ANTS scores were recorded as overall categories ranging from 1 to 4 (poor performance1 to good performance4) as described by the authors of the ANTS. Interrater reliability calculations were performed.

Secondary Endpoint: Analysis of the Number of Verbal Exchanges During High-Fidelity Simulation

To analyze the effect of the screen-based simulator on communication skills (included in teamwork skills), 2 blinded raters (both students in ergonomics, trained in verbalization analysis) viewed high-fidelity simulation videos to measure the number of verbal exchanges between the midwife and her colleagues, but also with the patient. The analysis was quantitative with raters counting the number of sentences and words.

Statistical Analysis

Data are presented as median (interquartile range [IQR]) for continuous data given the nonnormal distribution of data (small sample). Agreement between raters for ANTS was evaluated for each category using percent agreement and corresponding Fleiss κ coefficient (interrater agreement). Values of κ can range from 0 (or less) to 1. Landis and Koch (1977) gave a table for interpreting the κ values. The κ = 1 was a complete agreement among raters, whereas the κ = 0 illustrates a poor agreement. Calculation of interrater reliability between experts was performed by an online κ calculator.29 Statistical analyses were performed with the mean values obtained from the 3 raters. Comparison between NTS group and control group regarding the NTS performance was performed using the Mann-Whitney U test for independent samples, completed by an effect size calculation using a Pearson correlation test. All tests were two-tailed, and statistical significance was considered for P < 0.05. Statistical analyses were performed using SPSS 25.0 software (IBM Company).


Twenty-four participants (Fig. 3) were included in the study, 12 were randomly assigned to the control group and 12 to the NTS group. They were students in midwifery in their last year of training in France (fifth year) or first year of professional activity. All participants were women. The median age was 25 (22–39) years.

Consort flow diagram of the study (page 9).

Primary Endpoint: ANTS Scores

The median (IQR) value of the ANTS score was higher in the NTS group than the control group [15.5, IQR (13.25–16) vs. 8 (7.4–9.7), P < 0.0001] (Fig. 4). The detailed analysis for each ANTS category shows that the NTS group had higher ANTS scores in each category when compared with the control group (Table 1). The r value of the global ANTS score was 0.9, highlighting a large effect size.

Box plot comparing ANTS scores between NTS group and control group (min-max, median, Q1, Q3) (page 9).
Comparison of NTS Scores of NTS Group and Control Group During High-Fidelity Simulation (IQR, Median)

Interrater Agreement

Raters assessed each ANTS category on a 4-point scale. The overall agreement was 90.3% in average with a corresponding average Fleiss κ was 0.87, indicating an almost perfect agreement (Table 2).

Interrater Reliability Between Experts (Percent Agreement, Corresponding κ Coefficient and Classification) of ANTS

Secondary Endpoint: Analysis of the Number of Verbal Exchanges During High-Fidelity Simulation

During the high-fidelity simulation session, there were more verbal exchanges between participants and colleagues (obstetrician and nurses/facilitators) in the NTS group than in the control group [37 (31.25–50.5) sentences vs. 25.5 (22–33.25) sentences, U = 28.5, P < 0.01, r = 0.51], whereas the number of communication between midwife and patient was similar for both groups [16 (13.25–18.75) sentences vs. 15.5 (12–21) sentences, U = 65, NS, r = 0.07]. Concerning all verbalizations (with colleagues and parturient), significant differences were found between groups: midwives of the NTS group generated more sentences/communication [55 (48.25–66.5) sentences vs. 41.5 (30.5–52.75) sentences, U = 32, P < 0.05, r = 0.45].


The main finding of this study is that training on a screen-based simulation, with dedicated NTS scenarios and followed by a debriefing discussion surrounding the 4 domains of the ANTS,30 led to a better performance during a high-fidelity simulation session. The large effect size supports the magnitude of our intervention effect. In this trial, we enrolled midwifery students or midwives during the first year of their professional activity. We observed limited NTS for the control group during a high-fidelity simulated PPH. We emphasize that these young midwives had never participated in a high-fidelity simulation and had never received training regarding NTS. These results confirm the positive impact of such a training tool for the management of critical situations.16

The mastery of NTS is a major goal for practitioners.31 Early acquisition of NTS through dedicated training program is beneficial in the early phases of a health professional's career.32,33 A recent study even found an improvement in clinical performances and outcomes associated with an increased self-confidence after the implementation of a simulation-based education program for NTS.34

In this study, we evaluated the training on a screen-based simulator. Virtual or graphical environments create interactive and engaging learning environments.35 A recent review highlights the positive effect of serious games on knowledge acquisition despite the inconsistency of evidence to support the superiority of serious games over traditional learning approaches.16 Serious games seemed to be effective for knowledge retention, as they have been shown to be equivalent to traditional learning approaches.13,36,37 In 2 recent reviews, screen-based simulations also showed promise for NTS training in the health sciences, with improvements noted in error rates, multitasking, and teamwork.38,39

This project was the first to study the impact of training NTS with a screen-based simulation for a specific obstetrical emergency. Many obstetric situations are life-threatening emergencies. Postpartum hemorrhage is the leading cause of maternal death in the world, responsible for more than 100,000 deaths annually, mainly in developing countries.40 Rapid recognition of PPH is the main condition to reduce maternal mortality,6,7 but organizational and human factors are very important (eg, communication, team working, etc.), especially regarding the multidisciplinary aspect. A recent Australian study showed that the implementation of obstetric multiprofessional simulation training (half day of high-fidelity simulation training on obstetric emergencies with 508 participants) led to an improvement in both clinical and NTS during a postpartum hemorrhage situation. These results highlight the importance of teamwork, communication, leadership, and prioritization (pretest/posttest questionnaires). The improvement in the management of PPH was observed during the 2 following years after this training.13

To deepen the scope of our results, let's take the example of teamwork and more precisely communication. During the high-fidelity simulation session, the quantitative analysis of verbal exchanges analysis showed the participant communicated more with their colleagues in the NTS group than in the control group, whereas the number of communications between midwife and patient was stable for both groups. The previously trained midwifes seemed to have a better overall understanding of the importance of interprofessional communication during emergency situations. However, the quantitative aspect of our analysis limits the scope of this result, as the quality of these exchanges was not assessed. The causality between communication and case resolution remains unsure.

Limitations of the Study

First, the NTS assessment was performed with the ANTS scoring tool, which was not originally developed and validated for the studied population. The absence of published data on the use of the ANTS score for midwives is a limitation. However, the almost perfect agreement between raters highlights the feasibility and reliability of this score for our population.

Secondly, we did not collect baseline measurement for the participant's technical and NTS. This impairs the quality of evidences regarding the improvement in the NTS group. However, the 2 studied groups showed the same baseline professional experience and training level, and no participants had previous NTS training. This supports the equivalence in technical and NTS between groups.

Thirdly, the evaluation of the NTS was performed during a high-fidelity simulation session and not during a real-life case of PPH. The scarcity of the event makes it almost impossible to evaluate such real-life clinical impact. Its random occurrence would render a clinical evaluation in situ very laborious. Here, simulation can be a powerful tool as it allows us to recreate and study rare and/or critical events, such as this in a standardized way.26 A recent study41 highlighted this advantage by comparing results of simulation-based study versus real clinical situations: simulation might be a valid method to evaluate clinical issues.

Fourthly, the study did not evaluate the unique effect of the screen-based simulator on the NTS performance, but the effect of the simulation associated with a structured interview. Indeed, there is no simulation without debriefing, the opportunity for reflection and sharing, to transform experience into learning.42,43 The screen-based simulation did not include NTS automated feedback. Because it was necessary to stay as close as possible to an in-game/automated feedback, we chose a structured interview on NTS based on the classification of Flin et al.22

Finally, the control group received screen-based simulation education but no specific NTS training. Therefore, we did not answer the question of screen-based approach efficiency compared with other ways of teaching NTS skills. At last, as training and assessment happened during the same day, no evaluation of learning retention was performed. These 2 points could be subjects for further studies.


Our study suggests that NTS screen-based simulation, with specific scenarios and debriefing, leads to higher midwives “NTS performance during simulated postpartum hemorrhage.” This result is important given the scarcity and life-threatening nature of this complication, and there are still too numerous resulting deaths even in developing countries.


The authors thank the Baudelocque Midwifery School, Paris Descartes University, and especially Ms. Vérot and all the students who participated in the study. The authors also thank L'Agence Nationale de la Recherche. This article is part of “MacCoy Critical” project (Models for Adaptative feedback enriChment and Orchestration based virtual realitY in Critical situations) (ANR CE14-24 0021,


1. Lynch CB. A Textbook Of Postpartum Hemorrhage: A Comprehensive Guide To Evaluation, Management And Surgical Intervention. 1/E edition. Duncow: Jaypee Brothers Publishers; 2006.
2. World Health Organization. WHO recommendations for the prevention and treatment of postpartum haemorrhage. 2012. Available at: Accessed October 13, 2018.
3. Deneux-Tharaux C, Bonnet MP, Tort J. Epidemiology of post-partum haemorrhage [in French]. J Gynecol Obstet Biol Reprod (Paris) 2014;43(10):936–950.
4. Weeks A. The prevention and treatment of postpartum haemorrhage: what do we know, and where do we go to next? BJOG 2015;122(2):202–210.
5. Les morts maternelles en France : mieux comprendre pour mieux prévenir. 5e rapport de l'Enquête nationale confidentielle sur les morts maternelles (ENCMM) 2010-2012;231.
6. Marshall NE, Vanderhoeven J, Eden KB, Segel SY, Guise JM. Impact of simulation and team training on postpartum hemorrhage management in non-academic centers. J Matern Fetal Neonatal Med 2015;28(5):495–499.
7. Maslovitz S, Barkai G, Lessing JB, Ziv A, Many A. Improved accuracy of postpartum blood loss estimation as assessed by simulation. Acta Obstet Gynecol Scand 2008;87(9):929–934.
8. Cornthwaite K, Edwards S, Siassakos D. Reducing risk in maternity by optimising teamwork and leadership: an evidence-based approach to save mothers and babies. Best Pract Res Clin Obstet Gynaecol 2013;27(4):571–581.
9. Sevdalis N, Hull L, Birnbach DJ. Improving patient safety in the operating theatre and perioperative care: obstacles, interventions, and priorities for accelerating progress. Br J Anaesth 2012;109(Suppl 1):i3–i16.
10. Gordon M, Darbyshire D, Baker P. Non-technical skills training to enhance patient safety: a systematic review. Med Educ 2012;46(11):1042–1054.
11. Ventre KM, Schwid HA. Computer and web based simulators. In: Levine AI, De Maria S, Schwartz AD, Sim AJ, éditeurs. The Comprehensive Textbook of Healthcare Simulation [Internet]. New York, NY: Springer New York; 2013.
12. Haerling KA. Cost-utility analysis of virtual and mannequin-based simulation. Simul Healthc 2018;13(1):33–40.
13. Liaw SY, Chan SW, Chen FG, Hooi SC, Siau C. Comparison of virtual patient simulation with mannequin-based simulation for improving clinical performances in assessing and managing clinical deterioration: randomized controlled trial. J Med Internet Res 2014;16(9):e214.
14. Kapralos B, Fisher S, Clarkson J, van Oostveen R. A course on serious game design and development using an online problem-based learning approach. Interact Technol Smart Educ 2015;12(2):116–136.
15. Gorbanev I, Agudelo-Londoño S, González RA, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online 2018;23(1):1438718.
16. Sipiyaruk K, Gallagher JE, Hatzipanagos S, Reynolds PA. A rapid review of serious games: from healthcare education to dental education. Eur J Dent Educ 2018;22(4):243–257.
17. Raynal P. Simulation benefits in obstetrical emergency: which proof level? [in French]. Gynecol Obstet Fertil 2016;44(10):584–590.
18. Schornack LA, Baysinger CL, Pian-Smith MCM. Recent advances of simulation in obstetric anesthesia. Curr Opin Anaesthesiol 2017;30(6):723–729.
19. Barré J, Michelet D, Job A, et al. Does repeated exposure to critical situations in a screen-based simulation improve the self-assessment of non-technical skills in postpartum hemorrhage management? Simul Gaming 2019;1046878119827324.
20. Bracco F, Masini M, De Tonetti G, et al. Adaptation of non-technical skills behavioural markers for delivery room simulation. BMC Pregnancy Childbirth 2017;17(1):89.
21. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists' Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003;90(5):580–588.
22. Flin RH, O'Connor P, Crichton M. Safety at the Sharp End: A Guide to Non-technical Skills. Burlington, VT: Ashgate Publishing Company; 2008.
23. Lyk-Jensen HT, Jepsen RM, Spanager L, Dieckmann P, Østergaard D. Assessing nurse anaesthetists' non-technical skills in the operating room. Acta Anaesthesiol Scand 2014;58(7):794–801.
24. Cole DC, Giordano CR, Vasilopoulos T, Fahy BG. Resident physicians improve nontechnical skills when on operating room management and leadership rotation. Anesth Analg 2017;124(1):300–307.
25. Cavicchiolo ME, Cavallin F, Staffler A, et al. Decision making and situational awareness in neonatal resuscitation in low resource settings. Resuscitation 2019;134:41–48.
26. Balki M, Chakravarty S, Salman A, Wax RS. Effectiveness of using high-fidelity simulation to teach the management of general anesthesia for cesarean delivery. Can J Anaesth 2014;61(10):922–934.
27. Hagemann V, Herbstreit F, Kehren C, et al. Does teaching non-technical skills to medical students improve those skills and simulated patient outcome? Int J Med Educ 2017;8:101–113.
28. Boet S, Larrigan S, Martin L, Liu H, Sullivan KJ, Etherington N. Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties. Br J Anaesth 2018;121(6):1218–1226.
29. Randolph JJ. Online Kappa Calculator [Computer software]. 2008. Available at: Accessed November 14, 2019.
30. Flin Rhona, O'Connor Paul. Safety at the sharp end: a guide to non-technical skills. Available at: Accessed October 23, 2018.
31. Makary MA, Daniel M. Medical error-the third leading cause of death in the US. BMJ 2016;353:i2139.
32. de Feijter JM, de Grave WS, Hopmans EM, Koopmans RP, Scherpbier AJ. Reflective learning in a patient safety course for final-year medical students. Med Teach 2012;34(11):946–954.
33. Gordon M, Hill E, Stojan JN, Daniel M. Educational interventions to improve handover in health care: an updated systematic review. Acad Med 2018;93(8):1234–1244.
34. Coggins A, Desai M, Nguyen K, Moore N. Early acquisition of non-technical skills using a blended approach to simulation-based medical education. Adv Simul (Lond) 2017;2:12.
35. Prensky M. Don't Bother Me Mom-I'm Learning! St Paul, MN: Paragon House Publishers; 2006.
36. Akl EA, Kairouz VF, Sackett KM, et al. Educational games for health professionals. Cochrane Database Syst Rev 2013;(3):CD006411.
37. Blakely G, Skirton H, Cooper S, Allum P, Nelmes P. Educational gaming in the health sciences: systematic review. J Adv Nurs 2009;65(2):259–269.
38. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg 2012;99(10):1322–1330.
39. Wang R, DeMaria S Jr., Goldberg A, Katz D. A systematic review of serious games in training health care professionals. Simul Healthc 2016;11(1):41–51.
40. Say L, Chou D, Gemmill A, et al. Global causes of maternal death: a WHO systematic analysis. Lancet Glob Health 2014;2(6):e323–e333.
41. Merry AF, Hannam JA, Webster CS, et al. Retesting the hypothesis of a clinical randomized controlled trial in a simulation environment to validate anesthesia simulation in error research (the VASER Study). Anesthesiology 2017;126(3):472–481.
42. Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul Healthc J Soc Simul Healthc 2007;2(2):115–125.
43. Garden AL, Le Fevre DM, Waddington HL, Weller JM. Debriefing after simulation-based non-technical skill training in healthcare: a systematic review of effective practice. Anaesth Intensive Care 2015;43(3):300–308.

Screen-based simulation training; education; nontechnical skills; postpartum hemorrhage

Supplemental Digital Content

Copyright © 2019 Society for Simulation in Healthcare