A comprehensive and reliable assessment tool for those who provide anesthesia has content validity for NTS known as the Anaesthetists’ Non-Technical Skills (ANTS) system; ANTS consists of a scoring system involving task management, situational awareness, teamwork, and decision making.3,14 The ANTS system has been tested in a simulated operating room (OR) environment3,15 and in a clinical environment.16,17 The ANTS system is available free of charge for noncommercial use18 and identifies the 4 areas of focus for NTS in this study: teamwork, task management, clinical decision making, and situational awareness.
The aim of this study was to measure NTS improvement during a 2-week senior elective rotation that taught and built upon the level of existing clinical leadership skills. Senior residents received training and exercised their decision-making skills during this innovative rotation. Daily debriefings and real-time guidance were provided by the attending anesthesiology officer of the day (AOD).
This study was an observational cohort of the first 10 consecutive senior anesthesiology residents (clinical anesthesiology residents, level third year of training [CA3]/ fourth year of postgraduate training [PGY-4]) who chose an OR management and leadership elective rotation. The residents were enrolled in this study, which spanned from March 2013 to March 2015. This elective was available for 1 resident during each 2-week block. To qualify for this elective, the resident had to be judged by the department clinical competency committee to be in good academic standing, including having obtained the prerequisite medical knowledge. This study was approved by the University of Florida institutional review board, and all participants consented to participate. During this 2-week rotation, a 2-step cross-sectional assessment of senior residents was performed at the beginning and conclusion of the rotation. The modified Delphi method was used for the instruction of NTS techniques to the senior residents by the attending AOD as the exemplar and evaluator.19 OR raters (composed of attending AOD and charge nurse raters) were trained to evaluate using the 4 ANTS techniques and skill areas of NTS (task management, situational awareness, teamwork, and decision making)3 that were identified as necessary to successfully complete the goals of the 2-week rotation. These evaluations were based on the 6 Accreditation Council on Graduate Medical Education competencies (Table 1) and the milestones appropriate for the rotation (Table 2). This training included background literature relevant to ANTS as well as training in the evaluation parameters by using clinical scenarios. The list of skill areas provided the framework for senior residents to exhibit these NTS when directing clinical teams. The ANTS tool used in this study was developed for NTS; however, it was not specifically developed as a tool to be used to evaluate OR management decisions. The goals and objectives for the rotation, including all supplemental educational materials, the areas for evaluation, and the evaluation tool, were distributed before the start of each rotation for the resident to review.
Rotation resources were collected from a literature review and used as the rotation guide for instructional feedback regarding individual performance. Textbook and journal articles describing NTS and other articles relevant to excellence in anesthesiology, clinical leadership, and hospital economics were used as materials for the rotation.2–5,20–28 To assess NTS, a questionnaire survey was distributed to the resident, the attending AOD, and the OR charge nurse that compiled 14 elements to assess the 4 areas of the ANTS system: teamwork, task management, decision making, and situational awareness.3 Residents were assessed by only 1 attending and 1 charge nurse at each time point. The ANTS system has been utilized and validated in both simulation and clinical studies (see Discussion), with good interrater agreement on both the element (.55 to .67) and category levels (.56 to .65) and shows high internal consistency (Cronbach α = .79 to .86).3,14,16,17
Resident clinical duties during the rotation included leading anesthesia care teams based at an adult cancer and trauma hospital. Our model included managing the flow and daily assignments of 19 ORs, self-clinical assignment to 2 to 4 ORs based on OR volume that were staffed with nurse anesthetists, as well as being available for all emergent cases that presented to the trauma OR. The case mix included general surgery, urology, and orthopedic services, as well as non-OR anesthesia for gastroenterology and interventional radiology cases. The management of OR personnel included staffing changes that were required (eg, unplanned illness), leading interactions surrounding OR schedule changes with surgeons and the OR charge nurse, as well as shift relief and handoffs for cases extending into the evening. For each resident participant, the goals were to establish a logical sequence for first case starts for OR teams, staggering initial start times to allow smooth OR workflow, providing staff for different individual surgical needs, opening additional ORs when clinical volume demanded, and managing additional cases that emerged throughout the day. As the OR manager, resident AODs were also called to lead the response to daily intraoperative emergencies within the ORs, including, but not limited to, airway emergencies, surgical bleeding, embolic phenomena, and cardiac pathology. Resident AODs also helped direct and complete patient preoperative evaluations and obtain informed consents.
Study measures included both written comments and questionnaire rankings (at the beginning and end of the rotation) independently performed by resident self-evaluation and OR raters, which included the attending AOD and the OR charge nurse. Debriefings with attending AODs were conducted at the conclusion of each day, providing time for questions and the necessary instruction when reviewing NTS resources. During daily debriefings, self-reflection questions about individual performance effectiveness were posed, and feedback was provided by the attending AOD. In addition, to solidify the importance of the rotation goals, the following self-inquiry questions were asked daily:
- How has this rotation changed your view of managing OR teams?
- Explain how being an AOD has prepared you for becoming a clinical leader in your future practice?
- What personal attributes do you feel are necessary for leading clinical teams?
- What resources helped you improve your clinical leadership skills?
The ranking of questionnaire items with their associated activities were reported on a 4-point scale that ranged from a 1 for rarely, a 2 for sometimes, a 3 for often, and a 4 for always. For each of the 10 CA-3 residents who participated in this elective rotation, there were 14 items that required feedback from 3 different sources: resident self-assessment, assessment from OR raters, and assessment from the attending AOD and charge nurse. There were a total of 420 ranks at the beginning of the rotation and 420 ranks at the end of the rotation, each of which included 140 ranks from the residents’ self-evaluations and 280 ranks from the OR raters. Residents were blinded from ranks by the OR raters to enable them to complete the survey without being influenced by the other ratings. In addition, the OR raters were also blinded from the resident assessments. Scaled rankings were analyzed for the Kendall coefficient of concordance (ω) for rater agreement across resident, AOD, and charge nurse raters. For agreement, 0 means no agreements at all, and 1 represents perfect agreement. Results for each of the 14 items of the questionnaire were compared between the beginning and the end of the rotation with the Wilcoxon signed-rank test for matched samples (using SAS JMP Pro 12.0, SAS Institute, Cary, NC). Comparisons were run separately for attending AOD assessments (n = 10 per item), charge nurse assessments (n = 10 per item), and resident self-assessments (n = 10 per item). Hodges–Lehmann estimates for median differences with 95% confidence were calculated for each item for each rater. P < .05 was considered statistically significant. False-discovery rate methods were used to adjust for multiple comparisons.29 Power analyses showed that n = 10 in a matched-paired design (Wilcoxon signed-rank test, correlation between groups = .45, logistic distribution) was able to detect differences of .75 (SD = .65) at α = .05 (2-tailed), power = 90%.
A total of 420 ranks were completed at each time point, with 140 ranks from resident self-evaluations and 280 ranks from the OR raters. All 14 areas of the NTS ranks by questionnaire survey were compared at the beginning of the rotation and end of the rotation for both of the attending AODs (Figure 1), charge nurse assessments (Figure 2), and resident self-assessments (Figure 3). Resident performance of NTS improved from the beginning to the end of the rotation on 12 of the 14 NTS items (P < .05; Figures 1 and 2) according to either AOD or charge nurse assessments. Examining the proportion of AOD and charge nurse responses for each skill (Figures 1 and 2), by the end of the rotation, at least 90% of attending AOD and charge nurse raters reported that residents always demonstrated a focused method of approaching tasks, were humble to the complexity of the work, had personal contact with patients, and anticipated the changing environments. However, for some areas, 35% or fewer attending and charge nurse raters reported residents as always appropriately using authority and assertiveness, briefing teams about a plan of action, and reevaluating and debriefing individuals and teams, although all of these areas did show significant improvement from the beginning of the rotation. A majority of the items related to teamwork and situational awareness showed at least a 65% increase in being rated as “always” by AOD attending and charge nurse assessments.
For self-assessment (Figure 3), residents were less likely to report that they always performed the actions related to the most NTS items, with only 4 areas in which 50% or more of the residents reported that they always performed that action by the end of the rotation. Interestingly, 3 of these items (demonstrates focused method of approaching tasks, humble to complexity of work, and personal contact with patients) were the same tasks rated highly by AOD attending/charge nurse assessments. Furthermore, there were multiple areas that had low “always” ratings (10% or fewer residents) by the end of the rotation, although they showed increases in ratings of “often.”
Secondary analyses evaluated the relationships among the 4 content areas (teamwork, task management, clinical decision making, and situational awareness). Items within each content area were summarized so that correlations among content areas could be performed. Correlations (Spearman ρ) ranged from low to moderate. The lowest correlation (ρ = .206, 95% CI, −.142–.546, P = .276) occurred between situational awareness and task management. The highest correlation (ρ = .582, 95% CI, .303–.788, P = .001) occurred between clinical decision making and task management. These correlations suggest that although there is some overlap across teamwork, task management, clinical decision making, and situational awareness, these areas still measure distinct content.
The reliability of the questionnaire was high across all raters (Cronbach α = .73–.83), which was similar to previous uses of the ANTS system in both simulation and clinical settings. Interrater reliability (across the charge nurse, resident, and AOD raters) ranged from ω = .36 to .61 at the beginning of the rotation and ω = .27 to .70 at the end of the rotation. This indicates fair to moderate agreement across all 3 raters.
Common themes identified by residents at daily debriefings, as well as those recorded on written feedback forms, were recurrence of challenging situations and abilities needed to instruct and manage clinical teams. The resident comments about the abilities needed to instruct and manage teams included essentials of team management and delegation, skills needed to respond to surgical emergencies, maintaining closed-loop communication, and ensuring a safe environment. Resident AODs were repeatedly challenged on a daily basis by trauma OR cases when they were placed in the lead anesthesia role for resuscitative measures and invasive procedures. AOD residents were also the primary resource for responding to emergent intraoperative events in other ORs, which typically included the diagnosis and treatment of a presenting crisis.
During this 2-week rotation, resident participants at the University of Florida commented that OR management was more involved than simply making daily schedule assignments and responding to clinical needs. They found that this role requires a personal knowledge of collegial skills and personalities that enable the AOD to proficiently assign clinical work. Under the right circumstances of self-reflection and effective feedback, and implanting the seeds of leadership principles, anesthesiology residents can make the transition from physicians providing clinical care to physician leaders.30
The residents consistently ranked the clinical decision-making section of reevaluates and debriefs individuals and teams lower than some other NTS, revealing a responsibility that residents did not have as much experience with before the rotation. Reevaluation and debriefing are not typical skills learned during residency, as reflected in the lower scores and the improvement with exposure to these areas. This may be an area of future emphasis with additional training because “debriefs individuals and teams at the end of the rotation” only received an “always” 26.7% of the time. The highest “always” responses at the beginning of the AOD rotation to “humble to complexity of work” and “personal contact with patients” were expected. The “humble to complexity of work” is learned on the first day of residency with exposure to the complex health system and is reflected in these results.
This study was limited to a single institution. Furthermore, another limitation is our small sample (n = 10), which may limit the generalizability of our results. However, there were a large number of ranks at each point in time, across multiple raters. We used the ANTS tool because studies have shown it to be a reliable and validated measure of NTS in both simulation environments3,15 and clinical settings.16,17 Furthermore, the results of this study support that the ANTS scoring system has content validity. The expected outcome aimed at demonstrating improvement in NTS through an in situ NTS education experienced during this OR management and leadership rotation was achieved. This was confirmed by the significant improvement in NTS from baseline. Several studies have examined the validity of the ANTS, particularly in terms of content validity and observability. In terms of content validity, the ANTS was developed using literature review, expert interviews, comparison to existing metrics, theoretical models, cognitive task analysis, and iterative workshop development.14 To test its content validity, feedback from 50 consulting anesthesiologists was assessed, rating the ANTS completeness and observability after viewing 8 experimental videos.3 Results from this study supported a high level of content validity, with 100% of the anesthesiologists reporting that the tool addressed the key behaviors in questions, 84% stating that the tool was not missing key elements, and 82% reporting that the ANTS categories were very easy to observe.3 In addition, the good interrater reliability of the ANTS in this study was similar to that found in both simulation and clinical implementation of the ANTS. The study is limited by the tools that were used (in this study, that is the ANTS tool); however, this study demonstrated that this tool was able to be used for OR decision making and may help develop better assessment tools. The outcomes for this study documented improvements in evaluations; however, improvements in other quantifiable measures involving operating room performance metrics (eg, room turnover time among cases) were not measured. These quantifiable measurements potentially could provide additional information on whether or not the residents during the rotation improved their skills in the domains that were assessed.
In this study, only residents in their final year of training with the prerequisite clinical skills, as determined by the departmental clinical competence committee, were eligible for this elective. Because this was an elective, only those individuals motivated to learn the leadership principles that were offered in this rotation chose to enroll, and this may be partly explain why all of these residents achieved the final goals. However, the goals may have to be more challenging because even motivated students do not have the capacity to routinely obtain every goal during a rotation. In the future, the addition of quantitative data with OR performance metrics may provide additional assessment data. Based on this initial experience involving senior anesthesiology residents interested in leadership, the final goals potentially could have been more advanced. This might not be the case if this were a required rotation for all senior residents. Other potential explanations for all residents achieving the final goals include the parameters chosen, or the assessments did not permit differentiation between these levels.
Each resident was provided a 2-week rotation in a suite of ORs that provided similar case volumes and types of cases with 3 faculty attending AOD supervisors. The anesthesia team members, including other anesthesia providers (eg, certified registered nurse anesthetists) who changed on a daily basis and for this initial study, were not used as a part of the evaluation. Furthermore, a limitation of this study was that they were unable to account for potential differences in ratings across the attending AOD and charge nurse. Future plans include obtaining additional feedback from the anesthesia providers being supervised by the resident AOD and evaluating for improvement in OR performance metrics during the rotation. The average volume for the resident AOD was 6 to 10 cases per day, 2 to 3 traumatic cases per week, and 1 OR crisis per day. Specific details regarding the nature of the individual intraoperative crises were not tracked, and this represents an area for future data collection to ensure similar experiences among the residents as well as to identify areas to strengthen rescue training for residents.
Management of OR personnel included staffing changes required throughout daily shift work, with the goal of managing OR flow and throughput. Hospital economics in relation to utilization was discussed, but we were unable to address detailed implications of staffing decisions with an emphasis on economics and utilization because of the 2-week rotational time frame. Having the residents make decisions before receiving additional education targeted to that decision making is an area for future investigation and may increase resident interest in the topic. However, because these were senior residents, their prerotation self-assessments should be reflective of their familiarity with these skills before formal education. Whether these skills were maintained after the completion of the rotation and whether residents practiced the skills learned on the rotation are important pieces of additional information and represent future areas for further assessment. Education sufficient for basic OR management decisions requires 15 hours or 1.5 days (http://www.franklindexter.net/PDF%20Files/CourseAlternatives.pdf). To be able to provide additional background on OR management decisions that involve utilizing written formulas shown to increase trust would entail 35 hours.31–33 Complicated topics,34 including overutilized OR time, are not intuitive or learned through OR work experiences.35,36 In fact, one study using a highly standardized scenario-based simulated environment for OR management decisions showed that decision making was no better than random chance.36 This represents a future area for additional educational opportunities and potentially further study as well.
A commitment to teaching in the workplace gives residents the opportunity to observe successful NTS.18 Placing senior residents in a leadership role, such as the OR manager, also provides the scenarios needed for the practice of perioperative care and the development of clinical leadership skills. This formula requires committed faculty who are willing to devote part of their day to real-time instruction and feedback, as well as daily debriefings, and therefore should be comprised of specific attending AODs. It can be a challenge for the attending AOD to relinquish workflow decision making to a resident in a clinically demanding area such as the OR, and this represented one of the challenges when developing this rotation. When given the opportunity to lead with real-time support, activity-based assessments of NTS yields improved skills in the clinical arena. On the basis of our experience of this rotation, we believe that an OR management and leadership rotation focusing on NTS provides a framework for the development and assessment of perioperative leadership skills.
The authors would like to thank Kenneth Hartis (Charge Nurse, South Tower Operating Room, UF Health Shands, Gainesville, FL) for his responsiveness, patience, and feedback.
Name: Devon C. Cole, MD.
Contribution: This author helped design the study and the management, data analysis, and drafting of the manuscript.
Conflicts of Interest: None.
Name: Christopher R. Giordano, MD.
Contribution: This author helped design the study and the management, data analysis, and drafting of the manuscript.
Conflicts of Interest: None.
Name: Terrie Vasilopoulos, PhD.
Contribution: This author made substantial contributions to analysis and interpretation of data.
Conflicts of Interest: None.
Name: Brenda G. Fahy, MD, MCCM.
Contribution: This author helped design the study, analyze the data, and write the manuscript.
Conflicts of Interest: Dr. Fahy received no financial support for this manuscript. Dr. Fahy receives financial support from the American Board of Anesthesiology (ABA) as a board member, as well as ABA-International, LLC, as a consultant. The institution receives salary support from the Foundation of Anesthesia Education and Research and the National Institutes of Health (NIH) P = 50, “PICS: A New Horizon for Surgical Critical Care.”
This manuscript was handled by: Edward C. Nemergut, MD.
1. Flin R, O’Connor P, Crichton M.Safety at the Sharp End: A Guide to Non-Technical Skills
. 2008Burlington, VT: Ashgate Publishing.
2. Fletcher GC, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88:418429.
3. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90:580588.
4. Gaba D, Fish K, Howard S.Crisis Management in Anesthesiology
. 1994Edinburgh: Churchill Livingstone.
5. Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills: evaluation of a behavioural marker. Br J Anaesth. 2010;105:38.
6. Kohn L, Corrigan J, Donaldson M. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. 2000.Washington, DC: National Academy Press.
7. Vetter TR, Boudreaux AM, Jones KA, Hunter JM Jr, Pittet JF. The perioperative surgical home: how anesthesiology can collaboratively achieve and leverage the triple aim in health care. Anesth Analg. 2014;118:11311136.
8. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27:759769.
9. Mintz LJ, Stoller JK. A systematic review of physician leadership and emotional intelligence. J Grad Med Educ. 2014;6:2131.
10. Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system–rationale and benefits. N Engl J Med. 2012;366:10511056.
11. The anesthesiology milestone project. J Grad Med Educ. 2014;6:1528.
12. Cate O. Competency based education, entrustable professional activities, and the power of language. J Grad Med Educ. 2013;5:67.
13. Hafferty FW. Professionalism—the next wave. N Engl J Med. 2006;355:21512152.
14. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Rating non-skills: developing a behavioral marker system for use in anaesthesia. Cong Tech Work. 2004;6:165171.
15. Yee B, Naik VN, Joo HS, et al. Nontechnical skills in anesthesia crisis management with repeated exposure to simulation-based education. Anesthesiology. 2005;103:241248.
16. Taylor KL, Ferri S, Yavorska T, Everett T, Parshuram C. A description of communication patterns during CPR in ICU. Resuscitation. 2014;85:13421347.
17. Siu J, Maran N, Paterson-Brown S. Observation of behavioural markers of non-technical skills in the operating room and their relationship to intra-operative incidents. Surgeon. 2016;14:119128.
18. Industrial Psychology Research Centre. ANTS: a behavioural marker system for rating anaesthetists’ non-technical skills. Available at http://www.abdn.ac.uk/iprc/ants
. Accessed April 29, 2016.
19. McLaughlin M. The Rand change agent study revisited: macroperspectives and micro realities. Educ Res. 1990;19:1116.
20. Larsson J, Holmström I, Rosenqvist U. Professional artist, good Samaritan, servant and co-ordinator: four ways of understanding the anaesthetist’s work. Acta Anaesthesiol Scand. 2003;47:787793.
21. Eisenach J. In search of excellence in anesthesiology. Anesthesiology. 2009;110:45.
22. Glavin RJ. Excellence in anesthesiology: the role of nontechnical skills. Anesthesiology. 2009;110:201203.
23. Eisenach J. Studying tacit knowledge in anesthesiology. Anesthesiology. 2009;110:443444.
24. Smith AF, Greaves JD. Beyond competence: defining and promoting excellence in anaesthesia. Anaesthesia. 2010;65:184191.
25. Baker DP, Day R, Salas E. Teamwork as an essential component of high-reliability organizations. Health Serv Res. 2006;41:15761598.
26. Hill LL, Evers AS. Perspective: hospital support for anesthesiology departments: aligning incentives and improving productivity. Acad Med. 2012;87:348355.
27. Collins J. Level 5 leadership: the triumph of humility and fierce resolve. Harvard Bus Rev. 2005;83:211.
28. Christensen C. How will you measure your life. Harvard Bus Rev. 2010;88:4651.
29. Vasilopoulos T, Morey TE, Dhatariya K, Rice MJ. Limitations of significance testing in clinical research: a review of multiple comparison corrections and effect size calculations with correlated measures. Anesth Analg. 2016;122:825830.
30. Goleman D. What makes a leader? IQ and technical skills are important, but emotional intelligence is the sine qua non of leadership. Harvard Bus Rev. 2004;82:8291.
31. Prahl A, Dexter F, Braun MT, Van Swol L. Review of experimental studies in social psychology of small groups when an optimal choice exists and application to operating room management decision-making. Anesth Analg. 2013;117:12211229.
32. Wachtel RE, Dexter F. Curriculum providing cognitive knowledge and problem-solving skills for anesthesia systems-based practice. J Grad Med Educ. 2010;2:624632.
33. Dexter F, Van Swol LM. Influence of data and formulas on trust in information from journal articles in an operating room management course. A A Case Rep. 2016;6:329334.
34. Dexter F, Wachtel RE, Epstein RH. Decreasing the hours that anesthesiologists and nurse anesthetists work late by making decisions to reduce the hours of over-utilized operating room time. Anesth Analg. 2016;122:831842.
35. Dexter EU, Dexter F, Masursky D, Garver MP, Nussmeier NA. Both bias and lack of knowledge influence organizational focus on first case of the day starts. Anesth Analg. 2009;108:12571261.
Copyright © 2016 International Anesthesia Research Society
36. Dexter F, Willemsen-Dunlap A, Lee JD. Operating room managerial decision-making on the day of surgery with and without computer recommendations and status displays. Anesth Analg. 2007;105:419429.