Institutional members access full text with Ovid®

Share this article on:

Simulation-Based Assessment Identifies Longitudinal Changes in Cognitive Skills in an Anesthesiology Residency Training Program

Sidi, Avner MD*; Gravenstein, Nikolaus MD*†; Vasilopoulos, Terrie PhD*; Lampotang, Samsun PhD*†‡§

doi: 10.1097/PTS.0000000000000392
Original Article: PDF Only

Objectives We describe observed improvements in nontechnical or “higher-order” deficiencies and cognitive performance skills in an anesthesia residency cohort for a 1-year time interval. Our main objectives were to evaluate higher-order, cognitive performance and to demonstrate that simulation can effectively serve as an assessment of cognitive skills and can help detect “higher-order” deficiencies, which are not as well identified through more traditional assessment tools. We hypothesized that simulation can identify longitudinal changes in cognitive skills and that cognitive performance deficiencies can then be remediated over time.

Methods We used 50 scenarios evaluating 35 residents during 2 subsequent years, and 18 of those 35 residents were evaluated in both years (post graduate years 3 then 4) in the same or similar scenarios. Individual basic knowledge and cognitive performance during simulation-based scenarios were assessed using a 20- to 27-item scenario-specific checklist. Items were labeled as basic knowledge/technical (lower-order cognition) or advanced cognitive/nontechnical (higher-order cognition). Identical or similar scenarios were repeated annually by a subset of 18 residents during 2 successive academic years. For every scenario and item, we calculated group error scenario rate (frequency) and individual (resident) item success. Grouped individuals' success rates are calculated as mean (SD), and item success grade and group error rates are calculated and presented as proportions. For all analyses, α level is 0.05.

Results Overall PGY4 residents' error rates were lower and success rates higher for the cognitive items compared with technical item performance in the operating room and resuscitation domains. In all 3 clinical domains, the cognitive error rate by PGY4 residents was fairly low (0.00–0.22) and the cognitive success rate by PGY4 residents was high (0.83–1.00) and significantly better compared with previous annual assessments (P < 0.05). Overall, there was an annual decrease in error rates for 2 years, primarily driven by decreases in cognitive errors. The most commonly observed cognitive error types remained anchoring, availability bias, premature closure, and confirmation bias.

Conclusions Simulation-based assessments can highlight cognitive performance areas of relative strength, weakness, and progress in a resident or resident cohort. We believe that they can therefore be used to inform curriculum development including activities that require higher-level cognitive processing.

From the *Department of Anesthesiology, †Center for Safety, Simulation & Advanced Learning Technologies, ‡Clinical & Translational Science Institute Simulation Core, and §UF Health Shands Experiential Learning Center, University of Florida, Gainesville, FL.

Correspondence: Avner Sidi, MD, Department of Anesthesiology, University of Florida College of Medicine, 1600 SW Archer Rd, PO Box 100254, Gainesville, FL 32610-0254 (e-mail:

This work was supported by the Department of Anesthesiology at the University of Florida College of Medicine.

The authors disclose no conflicts of interest.

Supplemental digital contents are available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s Web site (

Copyright © 2018 Wolters Kluwer Health, Inc. All rights reserved