Adverse events in the operating room (OR) are often linked to failures in nontechnical skills (NTS).1,2 Although various definitions of NTS are available, the majority include intraoperative communication, situational awareness, decision-making, teamwork, and leadership.3,4 Specifically for orthopedic surgery, 44% of deaths had a failure in NTS with missing situational awareness accounting for the majority (51.7%) of reported incidents.5 Therefore, NTS are fundamental for procedure efficiency, patient outcomes, surgical success, and patient satisfaction.2,6 To the best of our knowledge, there are no studies on NTS in spine surgery available. Technical skills (TS), however, are defined as psychomotor actions or related mental faculties acquired through practice and learning pertaining to a particular craft or profession.7 Low levels of NTS, especially situational awareness, are associated with a higher likelihood of technical errors.3,8,9 Although NTS and TS are considered to be related, specific effects of NTS on patient outcomes as well as their interrelations to TS remain unclear.10 The current literature base on surgeon's experience and intraoperative NTS is inconsistent.11–13 Moreover, the role of surgical experience within the interplay of NTS and TS has not been surveyed yet.3,8,9 Simulation-based performance scenarios of surgical teams in realistic OR environments provide unique opportunities to scrutinize the interplay of surgeons’ TS and NTS during spine procedures.14 In other specialties, simulation studies revealed positive correlations between TS and NTS during intraoperative crisis resource management15 as well as open gastrojejunostomy16 and ureteroscopy.17
The purpose of this study was to investigate for associations between surgeons’ intraoperative NTS and TS and to determine the influence of surgeons’ experience on these associations.
MATERIALS AND METHODS
We followed the reporting guidelines for health care simulation research.18 Ethical approval was obtained from the Ethics Committee of the Faculty of Medicine, LMU University (Nr 773-15). All participants were informed about the purpose of the study, confidentiality of data, and provided written consent before data collection.
Study Design and Setting
We conducted an observational study with a simulated mixed-reality procedure of a vertebroplasty procedure (VP) in a full-scale and realistic OR environment. We recruited surgeons from two university teaching hospitals in Germany. We applied a convenience sampling approach through snowball invitations to all eligible surgeons within the respective departments. The study took place in the simulation center of the Institute for Emergency Medicine and Management in Medicine (INM). No changes were made to the study protocol after trial commencement.
Mixed-reality Surgical Simulation Environment
We simulated a two-sided VP of a fractured lumbar vertebra (L2). The setup allows for complete intraoperative workflow of a VP for a surgical team consisting of surgeon, anesthetist, scrub nurse and circulating nurse. The previous definition of procedural workflow, specification of role demands, and requirements for simulation environment are reported elsewhere.19 For this study, scripted confederates performed the roles of anesthetist, scrub nurse, and circulating nurse.
The mixed-reality simulation environment included a synthetic patient model, a decommissioned C-arm and a trocar which were equipped with tracking markers to determine their spatial relationships via an optical tracking system. A 3D printed synthetic spine model derived from a real patient CT dataset was inserted into the mannequin, covered with gel wax and synthetic skin which allowed for realistic incision and suture. Using the patient's CT data and the spatial relationship between the C-arm, the patient model and the trocar, radiation-free simulated fluoroscopic x-ray images identical of the surgical scene are generated.20 An injection system was employed to ensure precise measurement of bone cement injection. Lastly, standard tools for VP were available, that is, scalpel, trocar, clamps, hammer, and suturing material.
Participating surgeons were acquainted with the OR-set up, the team and the patient case. They subsequently performed a VP procedure under experimental conditions. Assessments started with the team time out and ended after suturing and surgeon stepping back from the table. Afterwards, each participant filled out a questionnaire. All confederate actors had a medical background and adhered to the predefined script. The procedure was captured on audio- and video recordings.
Surgeons’ NTS Assessment
Two trained observers rated surgeons’ intraoperative behaviors with the Observational Teamwork Assessment for Surgery (OTAS) scale. OTAS is a well-established and valid tool for OR teamwork assessment.21 It assesses NTS on five dimensions (communication, coordination, cooperation/backup behavior, leadership, and monitoring/situational awareness) on a seven-point Likert scale (from “0 = problematic behavior” to “6 = exemplary behavior”). OTAS ratings of surgeons’ NTS were established by two trained observers based on video and audio recordings. Interobserver agreement was high with intraclass correlation coefficient = 0.83.
Surgeons’ TS Assessment and Surgical Outcomes
Surgical performance was determined with the Objective Structured Assessment of Technical Skill (OSATS). OSATS is valid and reliable22 and has been applied in spine surgery.23 OSATS consists of three parts: 10-item task-specific rating scale (range 0–10), six-item global rating scale (range 6–30), and pass–fail rating.
For surgical outcome assessment we developed a measure based on literature review and interviews with five senior spine surgeons.19 It consists of a 14-item rating scale on trocar positioning with items on No-Go-Areas harmed, number of positioning attempts, rating of trocar depth, and others. Overall sum scores range between 0 and 22 points with higher scores indicative of better outcome. The scale can be found as Appendix 1, http://links.lww.com/BRS/B445.
TS performances were evaluated by an orthopedic expert (author C.M.) who had >15 years of experience in VP and >1000 VPs performed. He was completely blinded toward participants′ identity. All TS ratings were based on video and audio recordings, a fully interactive 3D virtual reconstruction playback, a postoperative CT showing the cement injection, and the actual printed and treated bone model.
Surgical experience was based on participants′ professional tenure (years since acquisition of license to practice) and number of VPs performed.
Eleven surgeons were included, with eight being male (72.7%). Six participants were trauma surgeons, three orthopedic surgeons, and two trauma and orthopedic specialists. Their working experience ranged between 0 years and 33 years (mean [M] = 7.82, standard deviation [SD] = 9.38). Numbers of VPs performed were between 0 and 200 (M = 35.0, SD = 62.75). For NTS, OTAS score ranged between 1.70 and 4.60 with a mean score of 3.01 (SD = 1.10). Regarding TS, task-specific OSATS score ranged between 5 and 10 with mean score of 8.09 (SD = 1.87) and mean global OSATS was 25.09 (SD = 3.88, range 20–30). Concerning the third OSATS outcome, 6 participants (54.5%) were evaluated with “pass,” whereas 5 (45.5%) failed. For surgical outcome, mean score of trocar positioning was 17.09 (SD = 4.91, range 8–22).
Data were tested for normality using the Kolmogorov–Smirnov test. As most variables were not normally distributed, nonparametric tests were utilized for testing our study objectives. For determination of relationships, we applied one-tailed Kendall's tau b; for group difference tests, we used one-tailed Mann–Whitney U tests. Kendall′s tau b were converted to r2.24 Values of P < 0.05 were considered statistically significant. SPSS 25 (IBM Corp., Armonk, NY) was used.
We first tested for relationships between NTS and TS: OTAS scores correlated positively with task-specific OSATS scores (τ = 0.63, r2 = 0.70, p = 0.006) and trocar positioning scores (τ = 0.60, r2 = 0.65, P = 0.007). Therefore, surgeons with better nontechnical behaviors also achieved better technical performance and surgical outcomes (see Table 1). Association with global OSATS scores was close to significance (τ = 0.37, r2 = 0.30, P = 0.065). Regarding associations between technical performance and surgical outcome, we observed significant correlations between OSATS task-specific and trocar positioning scores (τ = 0.67, r2 = 0.75, P = 0.004) as well as global scores and trocar positioning scores (τ = 0.73, r2 = 0.83, P = 0.002). Therefore, surgeons with better performance achieved superior outcomes.
For group differences between OSATS’ “pass” versus “fail” evaluation, nontechnical performance scores were significantly higher in “pass” group (M = 3.73, SD = 0.88) compared to “fail” group (M = 2.14, SD = 0.58; U = 3.00, P = 0.013). Respective scores of all five OTAS dimensions were significantly higher for “pass” group (see Appendix 2, http://links.lww.com/BRS/B445 for details).
We then determined associations between surgeons′ experience and intraoperative performance measures. Professional tenure correlated significantly with nontechnical performance, that is, OTAS overall score (τ = 0.71, r2 = 0.81, P = 0.001), task-specific OSATS scores (τ = 0.49, r2 = 0.48, P = 0.022), global OSATS scores (τ = 0.47, r2 = 0.45, P = 0.024), and trocar positioning scores (τ = 0.55, r2 = 0.58, P = 0.011). Identically, number of VPs performed correlated significantly with OTAS scores (τ = 0.72, r2 = 0.82, P = 0.001), task-specific OSATS scores (τ = 0.50, r2 = 0.50, P = 0.022), global OSATS scores (τ = 0.52, r2 = 0.53, P = 0.016), and trocar positioning scores (τ = 0.52, r2 = 0.53, P = 0.016). All correlations are depicted in Table 1.
We finally ran partial correlations for NTS and TS while controlling for both surgical experience measures, respectively. After taking account of professional tenure, correlations of OTAS scores with task-specific OSATS scores were τ = 0.45 (r2 = 0.42, P = 0.094), τ = 0.05 (r2 = 0.01, P = 0.441) with global OSATS scores, and τ = 0.36 (r2 = 0.29, P = 0.152) with trocar positioning scores. Associations of OTAS scores controlled for number of VPs performed were τ = 0.44 (r2 = 0.41, P = 0.100) with taskspecific OSATS scores, τ = −0.01 (r2 = 0.00, P = 0.487) with global OSATS scores, and τ = 0.38 (r2 = 0.32, P = 0.137) with trocar positioning scores. See Appendix 3, http://links.lww.com/BRS/B445 for details.
We investigated the relationship between technical and NTS in a simulated spine procedure. Our results revealed positive correlations such that higher NTS levels were related with increased TS, and vice versa. We further determined the influence of surgical experience that attenuated the strength of the association. To the best of our knowledge, this study is the very first in the field of spine surgery that comprehensively determines perioperative technical and NTS as well as experience. It thus contributes in several ways to the limited literature base.
First, concerning the role of surgical experience, we investigated the effect of experience on the relationship between technical and NTS. Previous studies showed inconsistent results: a simulation study including surgeons with 2 to 34 years of practice, surgical experience was found to have an effect on NTS and TS.25 In the same line, in vascular surgeons an investigation reported significant correlations between technical and NTS with significant differences between different experience groups.11 A third study reported a lack of respective correlations for urological surgeons.13 Further studies reported moderate to strong correlations between TS and NTS in simulation studies with anesthesiology residents,15 residents performing an open gastrojejunostomy,16 and novice surgeons conducting an ureteroscopy.26 Another study showed a non-linear relationship between NTS and experience, with NTS peaking around the time of fellowship in a sample of various surgical specialties and different experience levels.12 Summing up, it is possible that these inconsistencies can be attributed to the influence of experience on the relationship between NTS and TS. The results of our study resonate well with this assumption such that experience was highly related to NTS as well as TS.11,27 Despite the observation that NTS and TS improve through surgical practice, other post-hoc explanations may apply. Older, more experienced surgeons have a higher likelihood of previously completed NTS trainings, given the observation that NTS do not improve over time without practice and training.28 A further explanation stems from Cognitive load theory29: experienced surgeons have more mental resources to focus on NTS since their working memory is less occupied by technical performance demands. Moreover, intraoperative stress is proposed to contribute to poor TS.3 Hence, the relationship between NTS and TS is influenced by external stressors and individual coping behaviors.25,30
Owing to its pilot character, our study has some limitations. First, as this was a simulation study it is possible that participants did not perform to their full potential. Yet, the simulation procedure and environment were based on a rigorous, in-depth development process to reproduce a realistic OR setting with authentic OR team members.19 A systematic review on skills transfer after surgical simulation-based training indicates good transferability to real life surgery.31 We used ad-hoc teams instead of fixed teams that know each other; this may have influenced the results, as fixed teams show better NTS than ad-hoc teams.10 For TS, few surgeons attained maximum scores with ceiling effects. However, it is inevitable that experienced surgeons will achieve maximum performance levels in a routine procedure with moderate demands. In addition, our sample size is limited. As we used a convenience sampling approach a self-selection of the participants cannot be excluded. Regarding our assessment instruments, further research on the validity and reliability of the newly developed scale for surgical outcome is required. Additionally, we did not assess stress as contributing factor. This study was a cross-sectional study. Therefore, we cannot draw any inferences on the causality of the scores pertaining to surgeons′ NTS, TS, and experience. Nonetheless, our study has various strengths such as its comprehensive assessment of technical and NTS as well its employment of a full-scale OR team work environment.
The findings of this study advance future comprehensive assessment of TS and NTS in spine surgery. Future studies should investigate the effect of surgical experience on the relationship between NTS and TS with bigger sample sizes and across various surgical settings. Although simulation studies are a good opportunity to assess skills without harming patients, future studies may include real OR performance and patient outcomes for comparison. Moreover, our study simulated a VP, which is minimally invasive. Future studies may also include open surgeries as they show differences to minimally invasive surgeries regarding NTS.10 Finally, future investigations should include an assessment of intraoperative stress and learning curves when investigating NTS.
We examined the relationship between NTS and TS as well as surgical experience using a newly developed mixed-reality simulation environment for VP. We applied standardized observational tools and were able to demonstrate moderate to strong relationships specifically to spine surgery. Our results stress the importance of controlling the effect of professional experience when investigating NTS and TS. Moreover, our work provides further insights in establishing assessment and training environments to improve safety and patient care in spine surgery.
The authors acknowledge Angelika Richter und Nicole Ostermeier for supporting us as confederates. Moreover, the authors like to thank Michael Wengler, Aljoscha Kullmann, and Alexander Winkler for supporting us in the scenario setup.
1. Flin R, Mitchell L. Safer Surgery: Analysing Behaviour in the Operating Theatre. Boca Raton, FL: CRC Press; 2017.
2. Siu J, Maran N, Paterson-Brown S. Observation of behavioural markers of non-technical skills
in the operating room and their relationship to intra-operative incidents. Surgeon
3. Hull L, Arora S, Aggarwal R, et al. The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg
4. Yule S, Flin R, Paterson-Brown S, et al. Non-technical skills
for surgeons in the operating room: a review of the literature. Surgery
5. Panesar SS, Carson-Stevens A, Mann BS, et al. Mortality as an indicator of patient safety in orthopaedics: lessons from qualitative analysis of a database of medical errors. BMC Musculoskelet Disord
6. Bible JE, Shau DN, Kay HF, et al. Are low patient satisfaction scores always due to the provider?: Determinants of patient satisfaction scores during spine clinic visits. Spine (Phila Pa 1976)
7. Trumble WR, Stevenson A, Brown L. Shorter Oxford English dictionary on historical principles. 5th edOxford; New York: Oxford University Press; 2002.
8. McCulloch P, Mishra A, Handa A, et al. The effects of aviation-style non-technical skills
training on technical performance and outcome in the operating theatre. Qual Saf Health Care
9. Mishra A, Catchpole K, Dale T, et al. The influence of non-technical performance on technical outcome in laparoscopic cholecystectomy. Surg Endosc
10. Gjeraa K, Spanager L, Konge L, et al. Non-technical skills
in minimally invasive surgery teams: a systematic review. Surg Endosc
11. Black SA, Nestel DF, Kneebone RL, et al. Assessment of surgical competence at carotid endarterectomy under local anaesthesia in a simulated operating theatre. Br J Surg
12. Gostlow H, Marlow N, Thomas MJW, et al. Non-technical skills
of surgical trainees and experienced surgeons: Non-technical skills
of surgical trainees and experienced surgeons. Br J Surg
13. Lee JY, Mucksavage P, Canales C, et al. High fidelity simulation
based team training in urology: a preliminary interdisciplinary study of technical and nontechnical skills in laparoscopic complications management. J Urol
14. Coelho G, Defino HLA. The role of mixed reality simulation
for surgical training in spine: phase 1 validation. Spine (Phila Pa 1976)
15. Riem N, Boet S, Bould MD, et al. Do technical skills
correlate with non-technical skills
in crisis resource management: a simulation
study. Br J Anaesth
16. Rao R, Dumon KR, Neylan CJ, et al. Can simulated team tasks be used to improve nontechnical skills in the operating room? J Surg Educ
17. Brunckhorst O, Shahid S, Aydin A, et al. Simulation
-based ureteroscopy skills training curriculum with integration of technical and non-technical skills
: a randomised controlled trial. Surg Endosc
18. Cheng A, Kessler D, Mackinnon R, et al. Reporting guidelines for health care simulation
research: extensions to the CONSORT and STROBE statements. Adv Simul
19. Pfandler M, Stefan P, Wucherer P, et al. Stepwise development of a simulation
environment for operating room teams: the example of vertebroplasty
. Adv Simul
20. Stefan P, Habert S, Winkler A, et al. A radiation-free mixed-reality
training environment and assessment concept for C-arm-based surgery. Int J Comput Assist Radiol Surg
21. Hull L, Arora S, Kassab E, et al. Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg
2011; 212:234–243. e5.
22. Hatala R, Cook DA, Brydges R, et al. Constructing a validity argument for the Objective Structured Assessment of Technical Skills
(OSATS): a systematic review of validity evidence. Adv Health Sci Educ
23. Boody BS, Rosenthal BD, Jenkins TJ, et al. The effectiveness of bioskills training for simulated open lumbar laminectomy. Glob Spine J
24. Walker DA. JMASM9: converting Kendall's Tau for correlational or meta-analytic analyses. J Mod Appl Stat Methods
25. Wetzel CM, Black SA, Hanna GB, et al. The effects of stress and coping on surgical performance during simulations. Ann Surg
26. Brunckhorst O, Shahid S, Aydin A, et al. The relationship between technical and nontechnical skills within a simulation
-based ureteroscopy training environment. J Surg Educ
27. Anderson DD, Long S, Thomas GW, et al. Objective Structured Assessments of Technical Skills
(OSATS) does not assess the quality of the surgical result effectively. Clin Orthop Relat Res
28. Dedy NJ, Zevin B, Bonrath EM, et al. Non-technical skills
of surgery residents: does experiential learning lead to competence? J Am Coll Surg
29. Van Merrienboer JJG, Sweller J. Cognitive load theory in health professional education: design principles and strategies: Cognitive load theory. Med Educ
30. Krage R, Zwaan L, Tjon Soei Len L, et al. Relationship between non-technical skills
and technical performance during cardiopulmonary resuscitation: does stress have an influence? Emerg Med J
31. Dawe SR, Pena GN, Windsor JA, et al. Systematic review of skills transfer after surgical simulation
-based training. Br J Surg