We created a modified Objective Structured Assessments of Technical Skills that can be used for objective assessment of performance during robotic training. This assessment form demonstrates construct validity and interrater and intrarater reliability in inanimate simulation training.
There have been numerous recent reports focusing on robotic technical skills training.15–17 To date, the more comprehensive studies have been performed using a series of nine inanimate technical skills—five of these based on prior Fundamentals of Laparoscopic Surgery exercises with four newly developed tasks for robotic surgery.15,18,19 For these drills, investigators chose to assess performance using a combination of time and errors. Using this assessment technique, construct validity was established in a small study involving eight faculty and Fellows and four medical students.15 Thus, prior studies have shown that based on time and errors, trained surgeons perform better than medical students. We were less interested in the ability of an assessment technique to distinguish between novice and expert levels of proficiency and more interested in the ability to distinguish performance among various levels of resident and Fellow trainees. Furthermore, we wanted to offer a competency-based instead of time-based approach to assessment, which is currently lacking in robotic simulation training.20 In our current study, we used a modified version of Objective Structured Assessments of Technical Skills, because this method of assessment has demonstrated validity and reliability across numerous types of technical skills.3,4,6,12 Also, because our goal is to assess trainee performance during simulation training, we performed our validation study in a resident and Fellow trainee population.
The strengths of our study lie in the methodology that we used. Because of the multiinstitutional design, we were able to include a larger number of resident and Fellow trainees than typical for an educational study. We used rigorous methods for standardization of study sessions and blinding of scoring judges to reduce bias. Furthermore, our participants encompassed multiple surgical disciplines, and the judges represented many gynecologic subspecialties, allowing greater generalizability of our findings. Through our study design, we also demonstrated reproducibility of our results. This is important because, even if an educational tool is valid, it must also be reliable among different judges to be useful for a training program.
Our study also has some notable limitations. Although there are multiple types of validity, we focused only on construct validity. We indeed demonstrated that our assessment form could discriminate between levels of training and, furthermore, between levels of prior robotic surgical experience. We did not, however, include experienced nonrobotic surgeons in our study. Inclusion of nonrobotic faculty surgeons as a control group would have allowed us to assess for whether surgical experience alone, rather than robotic console experience, contributes to higher scores. Another limitation of our study is that we were unable to assess predictive validity—the ability of a tool to predict future performance (eg, in the live operating room). Lastly, although our intrarater reliability coefficients were excellent, some of the interrater reliability scores were only moderately high (α of 0.78 for tower transfer, big dipper, and train tracks drills). This suggests that subtle differences in interpretation may exist, although our intraclass correlation coefficient values were well within the 0.7 to 0.95 range reported in other Objective Structured Assessments of Technical Skills studies.11,21–23
Residency programs are continually being challenged to improve surgical training, and simulation training is becoming an increasingly popular method. For trainees wishing to learn robotic surgery, they often spend time practicing simulation drills but lack an objective method to demonstrate their performance. We offer an assessment tool that can be used to assess trainees during simulation training. It is important to note that to use these drills and assessment form as a “test of readiness” for live surgery, rigorous methodology should be used to establish benchmark scores for resident performance. Importantly, for residents, who are still in a training environment, these benchmarks may be very different than those that are set for faculty surgeons. Furthermore, we recognize that robotic surgery not only involves technical console skills, but also presents unique challenges for the bedside assistant, communication in the operating room, and for cost containment in a training environment. Thus, we would consider incorporating technical skill training as one component of a comprehensive robotic training curriculum. One such curriculum exists and is currently undergoing further study.24
In summary, we present an assessment form for robotic simulation training that demonstrates validity and reliability in resident and Fellow trainees. This tool, when paired with inanimate robotic skill drills, may prove useful for competency-based training and assessment.
1. Lenihan JP Jr, Kovanda C, Seshadri-Kreaden U. What is the learning curve for robotic assisted gynecologic surgery? J Minim Invasive Gynecol 2008;15:589–94.
2. Woelk JL, Casiano ER, Weaver AL, Gostout BS, Trabuco EC, Gebhart JB. The learning curve of robotic hysterectomy. Obstet Gynecol 2013;121:87–95.
3. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al.. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273–8.
4. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative ‘bench station’ examination. Am J Surg 1997;173:226–30.
5. Buerkle B, Rueter K, Hefler LA, Tempfer-Bentz EK, Tempfer CB. Objective Structured Assessment of Technical Skills (OSATS) evaluation of theoretical versus hands-on training of vaginal breech delivery management: a randomized trial. Eur J Obstet Gynecol Reprod Biol 2013;171:252–6.
6. Goff BA, Nielsen PE, Lentz GM, Chow GE, Chalmers RW, Fenner D, et al.. Surgical skills assessment: a blinded examination of obstetrics and gynecology residents. Am J Obstet Gynecol 2002;186:613–7.
7. Rackow BW, Solnik MJ, Tu FF, Senapati S, Pozolo KE, Du H. Deliberate practice improves obstetrics and gynecology residents' hysteroscopy skills. J Grad Med Educ 2012;4:329–34.
8. Gélinas-Phaneuf N, Del Maestro RF. Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery 2013;73(suppl 1):S30–8.
9. Steehler MK, Chu EE, Na H, Pfisterer MJ, Hesham HN, Malekzadeh S. Teaching and assessing endoscopic sinus surgery skills on a validated low-cost task trainer. Laryngoscope 2013;123:841–4.
10. Tsagkataki M, Choudhary A. Mersey deanery ophthalmology trainees' views of the objective assessment of surgical and technical skills (OSATS) workplace-based assessment tool. Perspect Med Educ 2013;2:21–7.
11. Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, et al.. Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 2013;216:955–965.e8.
12. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, et al.. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107–13.
13. Messick S. Standards of validity and the validity of standards in performance assessment. Educ Meas Issues Pract 1995;14:5–8.
14. Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc 2003;17:1525–9.
15. Dulan G, Rege RV, Hogg DC, Gilberg-Fisher KM, Arain NA, Tesfay ST, et al.. Proficiency-based training for robotic surgery: construct validity, workload, and expert levels for nine inanimate exercises. Surg Endosc 2012;26:1516–21.
16. Hung AJ, Jayaratna IS, Teruya K, Desai MM, Gill IS, Goh AC. Comparative assessment of three standardized robotic surgery training methods. BJU Int 2013;112:864–71.
17. Lyons C, Goldfarb D, Jones SL, Badhiwala N, Miles B, Link R, et al.. Which skills really matter? proving face, content, and construct validity for a commercial robotic simulator. Surg Endosc 2013;27:2020–30.
18. Arain NA, Dulan G, Hogg DC, Rege RV, Powers CE, Tesfay ST, et al.. Comprehensive proficiency-based inanimate training for robotic surgery: reliability, feasibility, and educational benefit. Surg Endosc 2012;26:2740–5.
19. Dulan G, Rege RV, Hogg DC, Gilberg-Fisher KM, Arain NA, Tesfay ST, et al.. Developing a comprehensive, proficiency-based training program for robotic surgery. Surgery 2012;152:477–88.
20. Schreuder HW, Wolswijk R, Zweemer RP, Schijven MP, Verheijen RH. Training and learning robotic surgery, time for a more structured approach: a systematic review. BJOG 2012;119:137–49.
21. Jabbour N, Reihsen T, Payne NR, Finkelstein M, Sweet RM, Sidman JD. Validated assessment tools for pediatric airway endoscopy simulation. Otolaryngol Head Neck Surg 2012;147:1131–5.
22. Marglani O, Alherabi A, Al-Andejani T, Javer A, Al-Zalabani A, Chalmers A. Development of a tool for Global Rating of Endoscopic Surgical Skills (GRESS) for assessment of otolaryngology residents. B-ENT 2012;8:191–5.
23. Nimmons GL, Chang KE, Funk GF, Shonka DC, Pagedar NA. Validation of a task-specific scoring system for a microvascular surgery simulation model. Laryngoscope 2012;122:2164–8.