Secondary Logo

Institutional members access full text with Ovid®

Augmented Reality Future Step Visualization for Robust Surgical Telementoring

Andersen, Daniel S., MS; Cabrera, Maria E., MSc; Rojas-Muñoz, Edgar J., BS; Popescu, Voicu S., PhD; Gonzalez, Glebys T., BS; Mullis, Brian, MD; Marley, Sherri, RN; Zarzaur, Ben L., MD; Wachs, Juan P., PhD

doi: 10.1097/SIH.0000000000000334
Technical Reports

Introduction Surgical telementoring connects expert mentors with trainees performing urgent care in austere environments. However, such environments impose unreliable network quality, with significant latency and low bandwidth. We have developed an augmented reality telementoring system that includes future step visualization of the medical procedure. Pregenerated video instructions of the procedure are dynamically overlaid onto the trainee's view of the operating field when the network connection with a mentor is unreliable.

Methods Our future step visualization uses a tablet suspended above the patient's body, through which the trainee views the operating field. Before trainee use, an expert records a “future library” of step-by-step video footage of the operation. Videos are displayed to the trainee as semitransparent graphical overlays. We conducted a study where participants completed a cricothyroidotomy under telementored guidance. Participants used one of two telementoring conditions: conventional telestrator or our system with future step visualization. During the operation, the connection between trainee and mentor was bandwidth throttled. Recorded metrics were idle time ratio, recall error, and task performance.

Results Participants in the future step visualization condition had 48% smaller idle time ratio (14.5% vs. 27.9%, P < 0.001), 26% less recall error (119 vs. 161, P = 0.042), and 10% higher task performance scores (rater 1 = 90.83 vs. 81.88, P = 0.008; rater 2 = 88.54 vs. 79.17, P = 0.042) than participants in the telestrator condition.

Conclusions Future step visualization in surgical telementoring is an important fallback mechanism when trainee/mentor network connection is poor, and it is a key step towards semiautonomous and then completely mentor-free medical assistance systems.

From the Departments of Computer Science (D.S.A., V.S.P.) and Industrial Engineering (M.E.C., E.J.R.M., G.T.G., J.P.W.), Purdue University, West Lafayette; and Indiana University, School of Medicine (B.M., S.M., B.L.Z.), Bloomington, IN.

Reprints: Daniel S Andersen, 305 N University St, West Lafayette, IN 47907 (e-mail:

Supported by both the Office of the Assistant Secretary of Defense for Health Affairs under Award No. W81XWH-14-1-0042, and the National Science Foundation under Grant DGE-1333468. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the funders.

The authors declare no conflict of interest.

© 2019 Society for Simulation in Healthcare