Obstetrics and gynecology trainees in the United States are deemed surgically competent if they pass the written and oral examinations of the American Board of Obstetrics and Gynecology.1 As part of the validity of the testing process, it is assumed that volume of surgery along with adequate sampling of a trainee's knowledge using case log scenarios can reassure surgical competency. This cognitive assessment is very different from direct observation of a trainee's ability to operate in practice because competency in performing a surgical skill requires demonstration of both cognitive and motor skills. Defining surgical competency begins with identifying key skills that are required to perform a procedure. This unique approach in assessing clinical decision-making is referred to as the key-feature approach.2 A key-feature is a focused step in which trainees are likely to make errors in the resolution of a problem.
Vaginal hysterectomy is a good example of a procedure gynecologists should be able to demonstrate competently in training. For the more than 500,000 hysterectomies performed each year for benign disease,3 the vaginal approach is preferred owing to lower postoperative pain, quicker recovery, and fewer complications than either the laparoscopic or abdominal approaches.4 Despite increased safety, the number of vaginal hysterectomies performed in the United States is decreasing, and, as case volume decreases, some training programs may find it difficult to reassure the public that trainees are becoming competent in this procedure. During the initial stages of learning vaginal hysterectomy, one of the key skills involves three simple, discrete steps that are repeated: placing a clamp, cutting the tissue, and suturing the vascular pedicle. These three steps are linked together to form a more complex skill, one that usually is assessed globally as a single, fluid movement. At our institution, we have observed trainees struggle while performing this skill. More importantly, difficulty performing these steps appears to be indicative of overall performance during the procedure.
Until recently, assessment of technical skills has included the use of case logs, time taken to perform a procedure, and global assessment by a teacher through observation.5,6 Case logs lack content validity because they document only participation in a procedure and do not assess technical ability of the individual performing the procedure. They are a surrogate measure, implying that volume alone equates with how well one performs a technical skill. Time taken to perform a procedure depends on multiple factors and therefore lacks reliability, and most forms of global assessment have poor reliability and unknown validity.5
One method to improve learning and assessment of surgical skills is through video. Coaches have successfully analyzed motion captured on video to refine an athlete's performance.7 Motion analysis also has been used during surgery to measure a surgeon's dexterity, spatial orientation, and operative flow along with formal assessment.8 In fact, surgeons in Canada, the United Kingdom, and Japan use video as a formal assessment tool in general surgery, urology, and vascular surgery to verify technical competency.9–11
The objective of this study was to determine the feasibility of using video motion analysis to quantify a key step of vaginal hysterectomy. We hypothesized that there would be measurable differences between novice and experienced trainees. If identified, these differences could not only be used by teaching surgeons to improve teaching of vaginal hysterectomy but potentially used as defined values that could support future competency-based assessment of this procedure.
MATERIALS AND METHODS
After institutional review board exemption status was assigned, appropriate cases of vaginal hysterectomy procedures were collected. These procedures were performed by obstetrics and gynecology residents and female pelvic medicine and reconstructive surgery fellows (postgraduate years 1 through 7) from our institution between May 2007 and June 2008, and they had been video recorded in a standardized fashion for a previously conducted study validating a new intraoperative surgical scale for vaginal surgery. All videos had been filmed with an effort to blind reviewers to the identity of the operating surgeon. All videos were collected and reviewed by a single investigator (G.D.) who was blinded to the identity and training level of the operating trainee. Scores from two valid and reliable rating scales, the Vaginal Surgical Skills Index and the Global Rating Scale12,13 had been completed by the staff gynecologist immediately after each surgery.
For purposes of this study, the skill of clamping, cutting, and suturing the left uterosacral ligament was selected to demonstrate the feasibility of video motion analysis. This skill was chosen because it encompasses a discrete set of tasks that are easily captured on film and incorporates important procedural skills such as flow, planning, and dexterity. Videos were excluded if footage of this step was not captured clearly during the entire task or if the surgeon's hands or instruments were not visualized clearly.
Video analyses were performed using two motion-tracking software programs, LoggerPro (Vernier Software and Technology, Beaverton, OR) and SiliconCoach Pro (SiliconCoach, Dunedin, New Zealand). LoggerPro allowed measurements of hand-motion trajectory, distance, and velocity. After the video clip was imported into the program, the software allowed an object to be tracked while simultaneously extracting X and Y position data as a function of time. A standard point at the center of the clamp was identified and tracked during performance of suturing. Hand-motion trajectory was measured during insertion of the needle into the tissue pedicle until removal of the needle at the conclusion of suture placement. SiliconCoach Pro software was used to measure total time to complete the entire task and angles between instruments. The bladder retractor represented the vertical Y axis, and, therefore, all angle measurements originated from the bladder retractor (0°). Angles were measured between the vertical Y axis and clamp on the left uterosacral ligament and vertical Y axis and needle-driver (Fig. 1). Subtracting these two angles determined the angle between the clamp and needle-driver. Angles were measured immediately before insertion of the needle into the pedicle and then at removal of the needle from the pedicle. Total time was measured from placement of the clamp until the last knot was placed.
Cumulative translational motion was calculated using the X and Y data points generated by the LoggerPro analysis of hand trajectory. Cumulative translational motion reflects efficiency and smoothness of movement. A novice trainee is expected to have a greater frequency of unnecessary movements, resulting in greater cumulative motion. This is reflected in the additional X and Y points and greater differences between X and Y points compared with an experienced trainee with smoother, more efficient movement. The Pythagorean theorem was used to calculate the change in distance between two consecutive frames. The hypotenuse, Z, was calculated from the change in distance between consecutive X and Y points using the formula [(X2–X1)2+(Y2–Y1)2]0.5. The absolute values of the change in distance, or Z values, were summed to calculate total change in motion. Maximum and mean velocities (cm/sec) of hand motion also were calculated from the velocity data generated by LoggerPro analysis.
Trainees were separated into novice (n=9) and more experienced (n=14) trainees based on the total number of career vaginal hysterectomies performed. A minimum cutoff of 25 vaginal hysterectomies was used to separate the groups (novice, fewer than 25; experienced, 25 or more). This cutoff was determined by Jelovsek et al as the volume of cases needed to reach minimum competency scores on the Vaginal Surgical Skills Index and the Global Rating Scale while performing a vaginal hysterectomy.
The population for this study was a convenience sample of obstetrics and gynecology trainees who underwent a standardized videotaping while performing vaginal hysterectomy for another purpose. Given this fixed sample size and the fact that no preliminary data were available on the use of video motion analysis in gynecologic surgery, no a priori sample size estimate was performed for this feasibility study. Motion-analysis data were compared between the novice and experienced trainees using the Wilcoxon rank-sum test. Multivariate linear regression was used to control for level of difficulty of the hysterectomy. P<.05 was considered statistically significant. The level of difficulty was assigned by the staff surgeon immediately after completing the hysterectomy by answering the question, “Compared with the average level of difficulty for your operations, would you say this operation was:” using a Likert scale ranging from 0 (“much more difficult than my average operation”) to 4 (“much easier than my average operation”). The Contrasting Groups method was used to determine competency cutoff values for measurements that were significantly different between experienced and novice trainees.14 Contrasting Groups is an accepted standard-setting method in which a distribution of trainees are divided into noncompetent and competent groups by an external criterion (in this case, 25 vaginal hysterectomies), the distribution of scores are plotted, and passing scores are set at the intersection of the distributions, with false-positives and false-negatives having equal weight. Statistical analyses were performed using JMP 7.0 (SAS Institute, Inc., Cary, NC).
Of the 43 procedures that were evaluated on video, the selected skill of clamping, cutting, and suturing the left uterosacral ligament was demonstrated adequately and allowed video motion analysis in 23 (53%). Although a trainee often was videotaped multiple times during the course of study, only a single tape demonstrating the selected skill from each individual was included in the analysis. These 23 videos included novice (n=9) and experienced trainees (n=14). Table 1 demonstrates findings from video motion analysis. Novice trainees took longer than did experienced trainees to clamp, transect, and suture the uterosacral ligament (112 seconds, 95% confidence interval [CI] 95–143, compared with 84 seconds, CI 65–102, P=.07, respectively). Novice trainees also had greater cumulative translational motion compared with experienced trainees (92 cm CI 58–125 compared with 49 cm CI 22–76, P=.05, respectively) (Fig. 2). At the start of suturing the uterosacral ligament, experienced trainees placed the Heaney clamp closer to a right angle to the vertical axis compared with novice trainees (experienced 96°, 95% CI 89–104, compared with novice 109°, 95% CI 100–118, P=.03) (Fig. 3). All findings persisted after controlling for level of difficulty of the hysterectomy. The angle between the vertical axis and needle-driver and angles (vertical axis to needle-driver and vertical axis to clamp) at removal of the needle from the pedicle were not significantly different between the groups. Furthermore, maximum and mean velocities were not significantly different between groups.
Contrasting Groups analysis demonstrates minimum values that separate more experienced trainees from novice trainees when performing a skill (Table 2). Using this method, the experienced-trainee group's total time to perform clamping, transecting, and suturing the uterosacral ligament was 112 seconds or less, and the angle between the vertical axis and clamp while suturing was 105° or less. If future assessment instruments such as video analysis or simulation are able to quantitate motion, a surgeon who has performed 25 or more vaginal hysterectomies would be expected to have an average cumulative translational motion of 75 cm or less.
Accurate and reliable evaluation of surgical performance is essential for teaching surgery and to reassure the public that a surgeon is competent to operate. In this study, we have demonstrated that video motion analysis is a feasible technique for objectively quantifying vaginal surgical skills. Through video motion analysis, we were able to measure and compare objectively cumulative hand motion, hand velocity, instrument angles, and time taken to complete a discrete set of surgical tasks used during vaginal hysterectomy in obstetrics and gynecology trainees. Although most assessments of surgical skill are subjective and qualitative, video motion analysis has the potential for providing an objective and quantifiable assessment of a trainee's surgical skills, which should prove valuable for surgical teaching. Moreover, we were able to identify quantifiable differences between novice and more experienced trainees, with experienced trainees demonstrating more efficient hand movement (less cumulative motion) and consequently quicker time to complete the discrete surgical task evaluated in this study. This suggests that video motion analysis may be able to provide objective data that could be used along with case logs and evaluations by attending surgeons to document the competency of a particular trainee to perform a specific procedure. Although there are a wide range of costs associated with the use of the motion analysis software and videotaping equipment, and additional personnel are required in the operating room, an advantage of this method is that analysis of a surgery may be performed by a single evaluator an unlimited number of times after completing the procedure. The software can be used efficiently by any surgical educator after understanding how to navigate through the functions of the software. Now that we have demonstrated that video motion analysis is feasible for a specific step in the performance of vaginal hysterectomy, our next goal is to break down the entire procedure into a number of clinically relevant, discrete surgical steps that can be analyzed using video motion analysis. This process, referred to as cognitive task analysis, identifies procedural steps, critical intraoperative decision-making points, and human-factor–related error information. This method has been shown to be effective in identifying relevant steps and decision points of an operative procedure that might otherwise be omitted during traditional teaching because of difficulties associated with expert knowledge automation.15
One of the more interesting findings of this study was that experienced trainees consistently held the Heaney clamp at a more horizontal angle than did novices, who held their clamp below the horizontal. This finding was not anticipated and not something the teaching surgeons in this study had recognized previously by casual observation. Careful evaluation of the data provided by the video motion analysis revealed that the decreased angle between the clamp and the vertical axis allows the surgeon to visualize the needle tip easily after it is driven through the uterosacral pedicle. Novice trainees were less likely to visualize the needle tip with a greater clamp angle, resulting in excessive, unnecessary attempts to visualize the needle. This was demonstrated by an increased total time and cumulative motion in the novice surgeons. Thus, the objective analysis provided by video motion analysis helped us identify a critical aspect of the set of skills under study that separated novice from experienced trainees. Since discovering this, we have made a conscious effort when teaching vaginal hysterectomy to correct this “error” in clamp angle during suturing that was encountered in novice surgeons. Our hope is that, by applying video motion analysis to other steps of vaginal hysterectomy, we will identify other critical differences between novice and experienced surgeons that will aid in teaching.
The findings of this study have several implications for teaching trainees how to perform vaginal hysterectomy. The measurements obtained can provide a gynecologic trainee specific learning goals that must be reached when performing the task of clamping, transecting, and suturing the uterosacral ligament. The data also provide the surgical instructor with the means to convey objective, quick feedback when teaching this step. This feedback can be in the form of a narrative comment such as, “Your efficiency may improve by holding the clamp as close to 90° to the bladder retractor as possible.” These data also demonstrate that competency cutoff values can be determined to separate trainees, potentially providing a method to assess competency in this single task. Motion analysis also may be useful for teaching other steps of vaginal hysterectomy, such as anterior or posterior entry into the peritoneal cavity, or in improving overall operative efficiency. Once the remaining critical steps required to perform vaginal hysterectomy are evaluated, we envision that measurements and cutoff scores could be obtained for each key step to aid in creating competency-based learning objectives for vaginal hysterectomy that surgical trainees could use as benchmarks that eventually lead to performing this procedure in a competent, safe manner.
This study was performed as part of a previously conducted study validating a new intraoperative surgical scale for vaginal surgery. The objective of this study was to demonstrate the feasibility of performing video motion analysis to assess a trainee while performing vaginal hysterectomy. Therefore, our sample size was limited to the videos available for analysis from the primary study and was not predetermined. We were able to demonstrate statistical differences in operating time, cumulative translational motion, and clamp angle between experienced and novice trainees during the step of suture ligation of the right uterosacral ligament. Variables for which we did not demonstrate a statistical difference between experienced and novice trainees, such as minimum and maximum velocity, should be interpreted with caution because of the risk of type II error given our small sample size.
Another limitation includes the potential error introduced by the investigator's measurements, which entailed manual placement of computer cursors on a particular object (clamp) for each video frame. The placement may have been altered as the frames advanced. Also, the placement likely varied slightly between different videos. This error was minimized by a single investigator performing all measurements. All 23 videos included in the study were similarly videotaped with regard to camera angles and distance from the surgical field. For every analysis, the measurements were calibrated against a stationary object present throughout the footage being analyzed. The width of the bladder retractor was used in this calibration. Therefore, it is unlikely that measurements differed between videos owing to slight variations in the distance between the video camera and surgical field. However, the degree of error that may be introduced by different assessors (interrater reliability) is currently unknown but could be determined easily. Recently, devices have been developed to measure motion during surgery with an effort to reduce these errors and provide motion in a three-dimensional space. These devices include electromagnetic motion tracking systems, hand sensors, and motion acquisition software and have been reported in laparoscopic and open general surgical procedures.16 Software can extrapolate the positional data to number and speed of hand movement, distance traveled, and total time taken to perform the task.17 However, such instruments can be associated with increased costs and setup and may not be feasible for most training sites. Second, it is not currently feasible to apply video motion analysis of surgical footage to determine competency in a procedure. Although software programs may be able to perform this function in the future, motion analysis could have a more immediate effect by providing useful data for high-fidelity surgical simulators. By incorporating standard setting data on key features of the vaginal hysterectomy, computerized simulators could improve their content validity and come closer to developing useful models for use in competency assessment in addition to repetitive practice.
In conclusion, video motion analysis is a feasible technique to quantify and compare objectively surgical skills during vaginal surgery. There are measurable differences between novice and more experienced surgical trainees performing vaginal hysterectomy that can be quantified using motion analysis and that may aid in surgical teaching. Ongoing research currently is being conducted to determine whether video motion analysis can be applied to other key steps of vaginal hysterectomy. The use of video motion analysis for the evaluation of laparoscopic or open procedures is also of interest and should be investigated. As in competitive athletics, video motion analysis holds promise as a tool to improve surgical teaching and refine surgical technique.
1. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63–7.
2. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med 1995;70:194–201.
3. Wu JM, Wechter ME, Geller EJ, Nguyen TV, Visco AG. Hysterectomy rates in the United States, 2003. Obstet Gynecol 2007;110:1091–5.
4. Johnson N, Barlow D, Lethaby A, Tavender E, Curr E, Garry R. Surgical approach to hysterectomy for benign gynaecological disease. Cochrane Menstrual Disorders and Subfertility Group. The Cochrane Database of Systematic Reviews 2006, Issue 2. Art. No.: CD003677. DOI: 10.1002/14651858.CD003677.pub3.
5. Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165:358–61.
6. Cuschieri A, Francis N, Crosby J, Hanna GB. What do master surgeons think of surgical competence and revalidation? Am J Surg 2001;182:110–6.
7. Nicholls R, Fleisig G, Elliott B, Lyman S, Osinski E. Accuracy of qualitative analysis for assessment of skilled baseball pitching technique. Sports Biomech 2003;2:213–26.
8. Tsue TT, Dugan JW, Burkey B. Assessment of surgical competency. Otolaryngol Clin North Am 2007;40:1237,59, vii.
9. Matsuda T, Ono Y, Terachi T, Naito S, Baba S, Miki T, et al. The endoscopic surgical skill qualification system in urological laparoscopy: A novel system in Japan. J Urol 2006;176:2168,72.
10. Beard JD. Education and Training Committee of the Vascular Society of Great Britain and Ireland. Setting standards for the assessment of operative competence. Eur J Vasc Endovasc Surg 2005;30:215–8.
11. Dath D, Regehr G, Birch D, Schlachta C, Poulin E, Mamazza J, et al. Toward reliable operative assessment: the reliability and feasibility of videotaped assessment of laparoscopic technical skills. Surg Endosc 2004;18:1800–4.
12. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997;84:273–8.
13. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg 1997;173:226–30.
14. Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006;18:50–7.
15. Ericsson KA, Charness N, Feltovich P. The Cambridge handbook of expertise and expert performance. New York (NY): Cambridge University Press; 2006.
16. Dosis A, Aggarwal R, Bello F, Moorthy K, Munz Y, Gillies D, et al. Synchronized video and motion analysis for the assessment of procedures in the operating theater. Arch Surg 2005;140:293–9.
17. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. BMJ 2003;327:1032–7.