Essentials in Minimally Invasive Gynecology Manual Skills Construct Validation Trial : Obstetrics & Gynecology

Secondary Logo

Journal Logo

Contents: Original Research

Essentials in Minimally Invasive Gynecology Manual Skills Construct Validation Trial

Munro, Malcolm G. MD; Advincula, Arnold P. MD; Banks, Erika H. MD; Auguste, Tamika C. MD; Chahine, E. Britton MD; Grace Chen, Chi Chiung MD, MHS; Curlin, Howard L. MD; Jorgensen, Elisa M. MD; Kim, Jin Hee MD, MS; King, Cara R. DO, MS; Lucas, Joelle MD; Milad, Magdy P. MD, MS; Mourad, Jamal DO; Siedhoff, Matthew T. MD, MSCR; Solnik, M. Jonathon MD; Destephano, Christopher C. MD, MPH; Thayn, Kim PhD; for the Essentials in Minimally Invasive Gynecology (EMIG) Steering Committee

Author Information
Obstetrics & Gynecology 136(1):p 83-96, July 2020. | DOI: 10.1097/AOG.0000000000003936
  • Open
  • SDC
  • Associated Video
  • Editors' Pick Podcast

Simulation-based assessments have shown promise in evaluating the performance of trainees and have correlated positively with patient-reported outcomes.1 Required objective measures of technical skills for certification in obstetrics and gynecology have been relatively absent in the curricula of residency training programs, in formative examinations offered by the Council on Resident Education in Obstetrics and Gynecology (CREOG) and in the certification examination for the American Board of Obstetrics and Gynecology (ABOG). However, ABOG has recently established passing the Fundamentals of Laparoscopic Surgery cognitive and manual skills examination as a prerequisite for board certification starting in 2020.

Adaptation of the Fundamentals of Laparoscopic Surgery high-stakes examination to evaluate gynecologic surgical cognitive and technical skills is problematic for several reasons. A meta-analysis suggested more validity evidence was required to support content (selection of tasks) and consequences (intended and unintended) of the assessment.2 Regarding content validity evidence, the multiple choice cognitive examination is specific to general surgery trainees and the manual skills test lacks assessment of the hysteroscopic skills essential for the gynecologic surgeon.3 In the general surgery literature, questions were recently raised about the utility of the Fundamentals of Laparoscopic Surgery system for residents, because no appreciable improvement in outcomes of laparoscopic cholecystectomies was noted.4 No comparable studies of consequence validity evidence have been performed in obstetrics and gynecology.

The AAGL responded to these gaps by commencing development of the Essentials in Minimally Invasive Gynecology project, comprising a cognitive question database with supportive online didactic materials and a psychomotor skills simulation system with laparoscopic and hysteroscopic exercises specific to gynecologic surgery.5 In 2018, the American College of Obstetricians and Gynecologists (ACOG) and CREOG joined the AAGL to begin the process of compiling evidence evaluating the validity of the psychomotor component when it is applied to trainees in obstetrics and gynecology as well as gynecologic surgeons. A pilot study was designed to refine the Essentials in Minimally Invasive Gynecology laparoscopic and hysteroscopic systems and explore, in a preliminary fashion, the utility of seven tasks—five laparoscopic and two hysteroscopic—in distinguishing among those with varying levels of skill and experience.6 The pilot study suggested that these systems may be useful for distinguishing between novice and mid-level trainees and also indicated that those expert surgeons who had completed a postresidency fellowship in minimally invasive gynecologic surgery would likely perform these skills at a measurably higher level.

The specific hypothesis tested was that Essentials in Minimally Invasive Gynecology performance is related to participants' training levels.

METHODS

Messick's validation framework7 was chosen for Essentials in Minimally Invasive Gynecology to remain consistent with a previous systematic review on FLS8 and because it is advocated by the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education.9,10 The design was a prospective cohort study comparing two sets of two groups of participants based on their level of training and their self-reported experience with both hysteroscopic and laparoscopic surgery and surgical simulation. The first two groups comprised novice (postgraduate year [PGY]-1) and mid-level (PGY-3) residents, each in the first 100 days of their training year; the second pairing included those considered proficient (ABOG-certified obstetrician–gynecologists [ob-gyns]) and experts who were 2-year fellowship–trained in minimally invasive gynecologic surgery.

The simulators used in this trial (Appendices 2–4, available online at https://links.lww.com/AOG/B925) have been described previously, as have the seven exercises—five laparoscopic (L-1 to L-5), and two hysteroscopic (H-1 and H-2).6 Maximum times allocated for each of the laparoscopic exercises were determined by evaluating previous performances in the pilot study.6 No maximum time was required for the hysteroscopic skills, because all participants completed these in the pilot. A 13.5-minute orientation video (Video 1) was provided to standardize exposure to both systems and each of the seven exercises. Then, assisted by the orientation proctor, participants had approximately 45 minutes of structured orientation to the systems and were provided videos of each exercise.

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 1.","caption":"Participant orientation video. To standardize orientation, this video was presented to each participant and covered at a high level, the rationale for the trial and the specifics of the exercises. Detailed video descriptions were also made available to the participants. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_jlm2li9o"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

The participants performed all of the laparoscopic exercises while standing on their preferred side of the simulator to replicate standard positioning alongside the operating table. The two hysteroscopic exercises were performed in a sitting position. Before the timing started, the testing proctor, blinded to the trainee status and different from the orientation proctor, described the exercises and the measured parameters with the aid of a laminated instruction card.

The following study exercises were performed:

  • L-1. Laparoscopic sleeve-peg transfer (Fig. 1A, Video 2)
    • The participant was supplied with two laparoscopic Maryland grasping forceps to transfer six cylindrical sleeves from the floor of the Essentials in Minimally Invasive Gynecology LaparoBowl to one of six peg targets located on five contiguous panels and then back to the original location. The participant's exercise time was calculated, as were potential errors such as dropped sleeves and failure to properly execute a transfer. The maximum allowable time was 330 seconds.
  • L-2. Laparoscopic pattern cut (Fig. 1B, Video 3)
    • This task required that the participant use laparoscopic scissors and a Maryland grasper to cut a circular pattern from the top layer of a double layer surgical gauze mounted on the LaparoBowl. The participant's exercise time was calculated, as were errors such as crossing lines, cutting both layers and either avulsing the gauze from the platform or the platform from the LaparoBowl. The maximum allowable time was 300 seconds.
  • L-3. Laparoscopic extracorporeal tie (Fig. 1C, Video 4)
    • Participants used a standard Fundamentals of Laparoscopic Surgery laparoscopic needle driver and another grasping instrument to pass a 90-cm 2-0 silk suture with a swedged-on tapered and curved needle through marks on a short, fenestrated portion of Penrose drain affixed at a 45-degree angle to a sponge block placed on the floor of the LaparoBowl. The linear defect was approximated with a knot comprising three, single, extracorporeally formed throws, each sequentially transferred into the trainer and tightened with a knot manipulator. The exercise was completed by cutting both ends of the suture with the laparoscopic scissors. The maximum allowable time was 600 seconds. The specimen was evaluated for apposition of the fenestration's edges and knot formation.
  • L-4. Laparoscopic intracorporeal knot (Fig. 1D, Video 5)
    • The participant was provided laparoscopic scissors and the same choices of needle driver or grasper offered in L-3. The target and its configuration were also identical, but the suture was a 15 cm length 2-0 braided polygalactin construct with a swedged-on and tapered curved needle. After passing the suture through the marks on the Penrose, the defect was approximated with a knot comprising three intracorporeally formed throws, the first of which was a double throw; the exercise ended when the suture was cut. The maximum allowed time and errors recorded were identical to those for L-3.
  • L-5. Laparoscopic running suture (Fig. 1E, Video 6)
    • The participant was asked to use laparoscopic instruments and a running 20-cm 2–0 polygalactin suture with a swedged-on curved needle to approximate the long fenestration in a piece of Penrose drain marked with five pairs of black “targets and attached to three contiguous internal panels of the LaparoBowl.” Exercise time was recorded as were targeting, approximation, avulsion and other errors. Maximum allowable time was 600 seconds.
  • H-1. Hysteroscopic targeting (see Appendix 2, https://links.lww.com/AOG/B925, Video 7)
    • The participant was provided a hysteroscope with a 30-degree lens prepositioned in a sheath that included a 5-Fr operating channel containing a specially designed 4 Fr probe. The participant was asked to identify, properly orient and then use the probe to depress the 10 numbered targets in the simulated endometrial cavity in the order announced by the testing proctor. Exercise time and errors were recorded.
  • H-2. Hysteroscopic foreign body (or polyp) removal (see Appendix 2, https://links.lww.com/AOG/B925, Video 8)
    • The participant was provided the same hysteroscope assembly as described for H-1 but with grasping forceps prepositioned in the operating channel. The forceps were used to grasp, detach and remove 10 small silicone “polyps” from the simulated endometrial cavity in the order announced by the testing proctor. Elapsed time and errors including dropped targets were recorded.

F1
Fig. 1.:
Essentials in Minimally Invasive Gynecology laparoscopic exercises. A. L-1 sleeve-peg transfer. Participants were provided two Maryland grasping forceps. B. L-2 pattern cut. Each participant was given a Maryland grasping forceps and curved laparoscopic scissors to cut the circle pattern. C. L-3 extracorporeal tie. Participants were provided their choice of two laparoscopic needle drivers (left inset) or one needle driver and a Maryland grasping forceps. Ties were thrown extracorporeally (left) and transferred into the trainer box using one of two choices of knot manipulators (center and right insets). D. L-4 intracorporeal knot. The same choice of needle drivers provided for L-3 used to form a knot with three intracorporeal throws. E. L-5 running suture. The same choice of needle drivers was offered (inset) to close the fenestration in the long Penrose with five paired targets. Photos of participants reprinted from Essentials in Minimally Invasive Gynecology manual skills pilot validation trial. Munro MG, Brown AN, Saadat S, Gomez N, Howard D, Kahn B, Stockwell E, et al. J Minim Invasive Gynecol 2020; 27:518–534, Copyright 2020, with permission from Elsevier.Munro. Essentials in Minimally Invasive Gynecology. Obstet Gynecol 2020.

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 2.","caption":"Exercise L-1: sleeve-peg transfer. Participants were asked to select a side of the trainer box and were provided two 5-mm laparoscopic outside diameter laparoscopic Maryland forceps. The exercise comprised moving each of the six sleeves on the floor-mounted peg module to one of the six pegs located in panels one through five of the LaparoBowl, making at least one transfer from one forceps to the other in the process. Once this was accomplished, the participant was asked to reverse the process. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_wzpojhrq"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 3.","caption":"Exercise L-2: pattern cut. The participant selected a side of the trainer box and was provided 5-mm outside diameter (OD) laparoscopic scissors and a 5-mm OD Maryland grasper. The exercise goal was to cut out the central circle, incising only the top layer of the gauze, and without the cut transecting the inner or outer boundaries. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_spvll5oq"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 4.","caption":"Exercise L-3: extracorporeal tie. After selecting a side of the trainer box, each participant was provided laparoscopic scissors and then chose a standardized, non–self-righting Fundamentals of Laparoscopic Surgery laparoscopic needle driver and another grasping instrument—either another identical needle driver or one of the Maryland forceps. The exercise goal was to pass a 90-cm 2-0 silk suture with a swedged-on 26-mm, 1/2 curved and tapered needle through the paired target marks on the Penrose. Then, participants approximated the linear defect with a knot comprising three, single, extracorporeally formed throws, each sequentially transferred into the trainer and tightened with a choice of two knot manipulators. To end the exercise, the participant was required to cut both ends of the suture with the laparoscopic scissors. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_kdcqfnuv"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 5.","caption":"Exercise L-4: intracorporeal tie. The participant is positioned and provided laparoscopic scissors and the same choices of needle drivers or grasper offered in Exercise L-3 (Video 4). The exercise goal was to pass a 15-cm 2-0 polygalactin 910 suture (Vicryl), swedged onto a 26-mm, 1/2 curved tapered needle through the marks on the Penrose and approximate the defect with a knot comprising three intracorporeally formed throws, the first of which was a double throw—a surgeon’s knot. The maximal allowed time and errors recorded were identical to those for Exercise L-3. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_zb5eb3bt"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 6.","caption":"Exercise L-5: running suture. Standing on one side of the trainer box, participants used a running suture to approximate a 5.4-cm longitudinal fenestration in a 9.5-cm-long, and 1.9-cm-wide (when flattened), thick-caliber Penrose drain. The Penrose, with five pairs of 2-millimeter diameter black targets, each one 3 mm from the fenestration and spaced at 0.95-cm intervals. The suture, a braided 2-0 polygalactin 910, swedged onto a 26-mm, 1/2 curved tapered needle, was 20 cm long with an anchoring knot located 18 cm from the swedge point of the needle. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_t30b9kvq"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 7.","caption":"Exercise H-1: targeting. After positioning the target module on the base unit, the participant was seated and provided a hysteroscope with a 30-degree Foroblique lens prepositioned in a 5.5 mm outside diameter sheath with a 5-Fr operating channel. A 4-Fr probe was used to depress targets in the targeting module in a sequence announced by the testing proctor. Once all 10 targets were depressed successfully, the participant removed the hysteroscopic assembly, whereupon timing was stopped by the testing proctor. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_j3yhuluj"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

{"href":"Single Video Player","role":"media-player-id","content-type":"play-in-place","position":"float","orientation":"portrait","label":"Video 8.","caption":"Exercise H-2: polyp (foreign body) removal. The foreign body removal exercise module was positioned with a cassette loaded with 10 detachable 5-mm long silicone foreign bodies or polyps. The participant was provided the same hysteroscope assembly as described for Exercise H-1 (Video 7) but with a prepositioned 5-Fr hysteroscopic grasping forceps used to grasp, detach, and remove the polyp in an order announced by the testing proctor. The participant was asked to repeat the process until all 10 targets were removed. Video created by AAGL. Used with permission.","object-id":[{"pub-id-type":"doi","id":""},{"pub-id-type":"other","content-type":"media-stream-id","id":"1_29pj7w11"},{"pub-id-type":"other","content-type":"media-source","id":"Kaltura"}]}

Approval for the overall study was obtained from the ACOG's institutional review board, Approval No. 38. Each of the academic sites also received approval from their local human participants institutional review board or appropriate ethics committee.

Candidates at the academic medical centers were recruited by the local Principal Investigator and asked to complete the anonymous web-based survey that included a scoring system designed to identify appropriate participants for the four cohorts. A score of 0 meant no exposure to simulation or surgery, and a score of 1 meant minimal exposure to diagnostic procedures and no surgical experience (Appendix 5, available online at https://links.lww.com/AOG/B925) (Fig. 2). The cohorts were defined both by level of training and self-described exposure to hysteroscopic and laparoscopic surgery and surgical simulation. Qualified candidates were provided a unique study number that allowed anonymous participation and storage and analysis of data. The key linking the study number to the participant's identification and contact information was stored securely in a web-based electronic database, with access limited to the Principal Investigator and selected study personnel.

F2
Fig. 2.:
Study flow. Potential participants were identified by the site principal investigator and offered the web-based qualification survey to determine qualification for the trial based on residency year, or postresidency status, as well as exposure to laparoscopic and hysteroscopic surgery and surgical simulation. Those who could be included as participants were provided a voucher containing a unique and anonymized identification number for scheduling and then presentation at the time of on-site testing. Testing was performed by an Essentials in Minimally Invasive Gynecology (EMIG) study team on site. Trained proctors supervised registration, orientation, and the acquisition of study data for each of the five laparoscopic and two hysteroscopic exercises.Munro. Essentials in Minimally Invasive Gynecology. Obstet Gynecol 2020.

The three study proctors were selected based on a spectrum of skills and knowledge as well as performance in the pilot study.6 Each was trained to be proficient at both assembling and troubleshooting the study systems that comprised mechanical and endoscopic equipment as well as computer hardware and software. Proctors were made responsible for maintaining the integrity of study data and protocols, including candidate orientation, conduct of study exercises and appropriate acquisition, labeling and storage of videos and test specimens. Each proctor was trained to use the data-entry systems, which comprised electronic tablets with touchscreens that allowed for secure web-based data entry. The study psychometrician observed the activities at the first study site and then participated in subsequent review and discussions, as appropriate.

In each of the locations or participating centers, the test environment comprised three dedicated areas: one for the structured orientation process and two for system orientation, participant testing, and data acquisition, all made free of potential disturbances.

All study data, including participant qualification questionnaires and testing data, were stored on a cloud-based server using Secure Socket Layer for data transmission, data storage encryption, and password-based access for data retrieval. The server was designed with data storage redundancy to protect against possible data loss. Access to the study data was strictly limited to the principal investigator, the co-principal investigator, the statistician, the psychometrician, and the selected AAGL staff who were directly assigned to the Essentials in Minimally Invasive Gynecology project. The deidentified data were provided to the statistician and the psychometrician during data analysis and reduction to provide the final data and statistical products to the investigator group led by the principal investigator and co-principal investigator.

The testing proctors were trained to measure and, when necessary, rate the performance exercises in the field, entering data that included exercise times, accuracy measurements and categorical variables into the appropriate electronic data forms. For participants who failed to complete a given exercise in the predetermined maximum time, the exercise was stopped, the maximum allowable time in seconds entered and the exercise designated as “did not complete.” Where applicable, study specimens were measured for both targeting accuracy and completeness, with data recorded in the electronic system. For later central review, study specimens were photographed and then sealed in containers labeled with the participant’s unique identification number. These included pattern-cut materials from L-2 and the Penrose targets for each suturing exercise; L-3, L-4, and L-5.

The central review was performed at the study center by a different trained proctor who evaluated both the video capture and the stored specimens. If there was a time discrepancy of more than 5 seconds or discrepancies in metrics related to accuracy, a third review was performed by two proctors designed to resolve differences. In such instances, a “final” score was obtained for each of the parameters by consensus.

The primary outcome was distinction between the novice and mid-level participants, and secondary outcomes included comparisons of the proficient and expert cohorts. For the pairwise primary and main secondary outcomes, the statistical plan was designed to determine whether any significant differences in scores and times could be observed between two levels of training.

The sample size calculation was based on the need to analyze data from at least 30 participants to meet the assumptions of a normal curve under the central limit theorem, thus allowing each cohort to stand as its own normal distribution. To allow for attrition and the possibility of incomplete data or data errors, the target for each group recruitment was increased to 40.

Captured simulation data were entered into an Excel spreadsheet. Using Excel, 95% CIs were constructed around the differences between the novice and mid-level means and the proficient and expert means for each scored simulation outcome. Each difference was subjected to hypothesis testing, where H0: x1−x2=0 and H1: x1−x2>0. Thus, any CI around a difference in means not including 0 would indicate a difference in cohort performance significant at the α=0.05 level.

RESULTS

A total of 227 participants (77 novice, 70 mid-level, 33 proficient, and 47 expert) were enrolled from two AAGL sites and from 13 academic centers representing all five ACOG regions (Table 1). Participants were divided into cohorts by training status, self-reported exposure to laparoscopic and hysteroscopic surgery and related surgical simulation. For the second, and, if necessary, the third and final analysis it was necessary to have complete video for the exercise as well as the study specimens. Full data were available on 67 novices, 61 mid-levels, 32 proficients, and 41 experts. These 201 participants comprised the evaluable study cohort, with the minimum sample size exceeded for each of the categories.

T1
Table 1.:
Enrolled Participants

Times for the five laparoscopic exercises (L1-5) by group are shown in Figure 3 and can also be seen in Table 2. In general, the mean times in seconds (±SD) for the mid-level participants for all exercises were significantly less than for the novices: L-1, 187 (±45) vs 256 (±59); L-2, 232 (±55) vs 274 (±38); L-3, 284 (±107) vs 344 (±101); L-4, 376 (±141) vs 481 (±126); and L-5, 420 (±100) vs 494 (±106). For those in the expert group, exercise times were less than for the proficient participants: L-1, 138 (±30) vs 199 (±50); L-2, 164 (±51) vs 245 (±51); L-3, 182 (±62) vs 294 (±104); L-4, 178 (±71) vs 402 (±156); and L-5, 239 (±76) vs 386 (±135). Many participants were not able to complete a given laparoscopic exercise in the allotted time and, in such instances, were assigned the maximum time allocated for the given exercise for the purpose of calculating the exercise completion times. These did-not-complete rates are shown in Figure 4. The novice cohort had high failure to complete rates in all but the extracorporeal knotting exercise (L-3), and for the mid-level and proficient groups, failure to complete rates were greater than 15% for the Circle Cut (L-2) and intracorporeal knot tying (L-4) exercises. Only the expert group had negligible failure to complete rates for all five laparoscopic exercises.

F3
Fig. 3.:
Laparoscopic exercise completion times. Mean exercise times in seconds for the four cohorts (standard error of the mean) for each of the L-1 to L-5 exercises. At the base of each bar is the number of participants included in the calculation for each exercise. Variable numbers of participants reflect the absence of recorded video for some participants because all required central verification. If a participant timed out, not completing the exercise, they were assigned the maximum allowable time. The error bars indicate standard error of the mean. PGY, postgraduate year; FMIGS, fellowship in minimally invasive gynecologic surgery.Munro. Essentials in Minimally Invasive Gynecology. Obstet Gynecol 2020.
T2
Table 2.:
Exercise Completion Times
F4
Fig. 4.:
Laparoscopic exercise did-not-complete rates. Percentage of participants not completing the task within the allotted time for each of the L-1 through L-5 exercises. For these, the maximum time allowed for the exercise was assigned for the time calculations shown in Figure 3 and in Table 2. PGY, postgraduate year; FMIGS, fellowship in minimally invasive gynecologic surgery.Munro. Essentials in Minimally Invasive Gynecology. Obstet Gynecol 2020.

Candidates were also assessed for precision and accuracy in each of the five laparoscopic exercises (Table 3). For these calculations, only “completed-the-task” elements were used for the denominator, which varies according to task. For example, for L-1, dropped or incorrect transfer of sleeves were recorded. In this case, the mid-level group averaged the fewest incorrect transfers, the expert group had the lowest frequency of dropped sleeves, and novices had the highest frequency.

T3
Table 3.:
Exercise Quality Metrics
table3-a
Table 3-A.:
Exercise Quality Metrics

Pattern cut (L-2) errors comprised crossing one or both lines and cutting the bottom layer. Performance here demonstrated a high degree of accuracy in the expert group and relatively frequent errors for the novices; the mid-level and proficient groups were similar. Few in any group erroneously cut the bottom layer (not shown). Extracorporeal (L-3) and intracorporeal (L-4) errors comprised accuracy (target entrance and exit errors); tissue handling (avulsion and tear through errors, not shown); and knot construction (“air knots” and square knot errors). These outcomes were similar to other groups where, in general, the mid-level and proficient participants were similar but superior to novices, and the expert group scored highest overall. For some outcomes, there were similarities between the experts and others, including errors in square knot formation and the creation of one or more “air knots.” Running suture (L-5) accuracy for the 10 targets is shown in Appendix 6 (available online at https://links.lww.com/AOG/B925) and Table 3. In this exercise members of the mid-level cohort were more accurate than the novice group, and the expert group had greater overall accuracy (sum of errors) than the proficient cohort.

The mean exercise completion times for the two hysteroscopic exercises chosen for this trial are shown in Figure 5, as well as Table 2. The hierarchy of completion times for H-1 and H-2 was similar to that demonstrated in the laparoscopic exercises, but the differences, although significant, were not as profound. The novice group had longer mean completion times in seconds (±SD) than the mid-level group: H-1, 176 (±56) vs 141 (±48); and H-2, 200 (±96) vs 150 (±37). The proficient cohort had longer times than the expert group: H-1, 141 (±52) vs 117 (±33); and H-2, 138 (±33) vs 120 (±31). The expert cohort performed better than the other three and the mid-level and proficient groups had similar times for each of the two exercises. Unlike the laparoscopic exercises, virtually all of the participants in each group completed each of the two hysteroscopic tasks, and targeting errors were similar (Table 3).

F5
Fig. 5.:
H-1 and H-2 completion times. Completion times in seconds (±standard error of the mean) for the two hysteroscopic exercises, by cohort. The denominators vary and are less than the overall cohort because a complete video was necessary to centrally validate the participants' time. PGY, postgraduate year; FMIGS, fellowship in minimally invasive gynecologic surgery.Munro. Essentials in Minimally Invasive Gynecology. Obstet Gynecol 2020.

DISCUSSION

The Essentials in Minimally Invasive Gynecology simulation-based assessment demonstrates multiple sources of evidence supporting construct validity. The unique nature of being gynecologic surgery-specific allowed the study to reflect the construct it was intended to measure—laparoscopic and hysteroscopic skills. Response process validity evidence is supported through a rigorous review process of the tasks incorporating quality control and an evaluation of rater discrepancies to align the scores with the intended construct. The availability of video capture and centrally stored specimens allowed for a structured remote review and analysis of all participants by more than one rater, a circumstance that largely removed observer error as a confounder.

The Essentials in Minimally Invasive Gynecology simulation-based assessment distinguished a spectrum of performance outcome variables between groups, thereby supporting the interpretation that performance is related to participants' training level. Differences were demonstrated for both the primary outcome comparisons of novice (PGY-1) compared with mid-level (PGY-3) residents and the principal secondary outcome comparison of ABOG-certified proficient gynecologic surgeons without subspecialty training compared with those expert ABOG-certified ob-gyns who had completed an accredited 2-year fellowship in minimally invasive gynecologic surgery. These outcomes are even more discrepant when considering the high frequency of “failure to complete” tasks in the novice cohort, a circumstance that was also present to a degree in the mid-level and proficient groups, at least when compared with the expert cohort. These differences were generally lessened because calculation of both time and accuracy could not be performed if an exercise was not completed, a circumstance that occurred more often with decreasing level of training. Interestingly, the differences in performance were more pronounced for the five laparoscopic than the two hysteroscopic exercises. The reasons for this observation are not clear but may be related to a relatively lower degree of difficulty of hysteroscopic exercises.

The similar performance of the mid-level residents compared with the proficient cohort could reflect some selection bias. The mid-level residents were all from centers where there was an interest and experience in simulation-based medical education, usually with an established fellowship in minimally invasive gynecologic surgery program, which may not be representative. Expert surgeons performed at a superior level in time and accuracy in almost all of the categories.

The strengths of this validity trial include the large sample size from multiple centers that comprised regionally representative training programs. The rigorous design included field and centrally performed analysis of both time and measures of accuracy. In addition, the study used a contemporary validity framework describing how assessment interpretations can support defensible decisions, whereas the majority of the studies described by Cook et al in a meta-analysis used “an outdated or incomplete framework to interpret validity data, if they used any framework at all.”11,12

There are limitations to the study. As stated, there could be selection bias, particularly for the mid-level cohort, for which training, including simulation-based medical education, may have been more robust than the average U.S. program. The proficient cohort, though meeting the sample size requirements, was nonetheless less regionally representative, and, consequently, performance characteristics might vary with a larger sample from a broader geographical spectrum. A larger study including community programs without a fellowship in minimally invasive gynecologic surgery could provide additional insight. It is also possible that the posttraining experience of the proficient and expert cohorts contributed to differences in performance. Although this variable was estimated in the survey, a more rigorous evaluation might provide evidence of a more granular nature.

The long-term significance and consequence evidence of the Essentials in Minimally Invasive Gynecology simulation-based assessment was neither evaluated nor defined at the outset of this study. Consequence validity evidence examines the intended and unintended implications of deciding to use an assessment on downstream outcomes (eg, health systems, surgical practice variation, surgical outcomes, and what is not taught in order to learn how to pass the assessment). Despite its importance, consequence evidence is rarely reported as part of health professions' education validity studies (5–20%) and is lacking for utilization of the Fundamentals of Laparoscopic Surgery examination system for obstetrics and gynecology trainees.8,13 As future studies are developed, consequence evidence will be an important factor when determining whether to continue to require Fundamentals of Laparoscopic Surgery or adopt modified or new high-stakes simulation-based assessments that have stronger content validity evidence such as Essentials in Minimally Invasive Gynecology.

Providing a context for the development and assessment of both laparoscopic and hysteroscopic skills is an important educational element, perhaps critical for safe and effective contemporary gynecologic surgery. However, and despite its adoption as a criterion for ABOG certification, there has been only one published assessment of the construct validity of the Fundamentals of Laparoscopic Surgery platform among trainees in gynecologic surgery.3 Investigators showed that comparative performance in standardized tasks using the Fundamentals of Laparoscopic Surgery box simulator could distinguish skilled from unskilled surgeons. There are still relatively few hysteroscopic simulation articles available in the literature published since the mid-1990s (Wallwiener D, Rimbach S, Aydeniz B, Pollmann D, Bastert G. Operative hysteroscopy: results, security aspects, in vitro simulation training (hysterotrainer) [abstract]. J Am Assoc Gynecol Laparoscopists 1994;1:S39; and Lefebvre Y, Cote J, Lefebvre L. Teaching surgical hysteroscopy with a computer [abstract]. J Am Assoc Gynecol Laparoscopists 1996;3(Suppl 4):S25).14 Only a limited number, and with variable quality, have evaluated construct validity,15–19 and none have demonstrated predictive validity.20,21 Those described in the literature are usually assessments of virtual reality devices that, because of cost, may limit access for most trainees and training programs.22 The Essentials in Minimally Invasive Gynecology Hysteroscopy Simulation System is a “low-fidelity” modular system that may provide an opportunity to expand this critically important aspect of gynecologic simulation because of its dramatically reduced cost compared with virtual reality systems.

REFERENCES

1. Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA. Linking simulation-based educational assessments and patient-related outcomes: a systematic review and meta-analysis. Acad Med 2015;90:246–56.
2. AlJamal YN, Zendejas B, Gas BL, Ali SM, Heller SF, Kendrick ML, et al. Annual surgeon volume and patient outcomes following laparoscopic totally extraperitoneal inguinal hernia repairs. J Laparoendosc Adv Surg Tech A 2016;26:92–8.
3. Hur HC, Arden D, Dodge LE, Zheng B, Ricciotti HA. Fundamentals of Laparoscopic Surgery: a surgical skills assessment tool in gynecology. JSLS 2011;15:21–6.
4. Dubina ED, Pham XD, Schwed AC, Wu H, McElroy I, Kaji AH, et al. Association of Fundamentals of Laparoscopic Surgery certification with outcomes of laparoscopic cholecystectomy performed by surgical residents. JAMA Surg 2018;153:1158–60.
5. Morozov V, Nezhat C. Proposal of a formal gynecologic endoscopy curriculum. J Minim Invasive Gynecol 2009;16:416–21.
6. Munro MG, Brown AN, Saadat S, Gomez N, Howard D, Kahn B, et al. Essentials in Minimally Invasive Gynecology manual skills pilot validation trial. J Minim Invasive Gynecol 2019;27:518–34.
7. Messick S. Validity. In: Linn R, editor. Educational measurement. 3rd ed. New York, NY: American Council on Education and Macmillan; 1989:13–103.
8. Zendejas B, Ruparel RK, Cook DA. Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review. Surg Endosc 2016;30:512–20.
9. National Council on Measurement in Education, American Educational Research Association, American Psychological Association. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 1999.
10. National Council on Measurements in Education, American Educational Research Association, American Psychological Association. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.
11. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theor Pract 2014;19:233–50.
12. Cook DA. Much ado about differences: why expert-novice comparisons add little to the validity argument. Adv Health Sci Educ Theor Pract 2015;20:829–34.
13. Cook DA, Lineberry M. Consequences validity evidence: evaluating the impact of educational assessments. Acad Med 2016;91:785–95.
14. Wallwiener D, Rimbach S, Bastert G. The HysteroTrainer, a simulator for diagnostic and operative hysteroscopy. J Am Assoc Gynecol Laparoscopists 1994;2:61–3.
15. Bajka M, Tuchschmid S, Fink D, Szekely G, Harders M. Establishing construct validity of a virtual-reality training simulator for hysteroscopy via a multimetric scoring system. Surg Endosc 2010;24:79–88.
16. Munro MG, Behling DP. Virtual reality uterine resectoscopic simulator: face and construct validation and comparative evaluation in an educational environment. JSLS 2011;15:142–6.
17. Elessawy M, Skrzipczyk M, Eckmann-Scholz C, Maass N, Mettler L, Guenther V, et al. Integration and validation of hysteroscopy simulation in the surgical training curriculum. J Surg Educ 2017;74:84–90.
18. Panel P, Neveu ME, Villain C, Debras F, Fernandez H, Debras E. Hysteroscopic resection on virtual reality simulator: what do we measure? J Gynecol Obstet Hum Reprod 2018;47:247–52.
19. Savran MM, Nielsen AB, Poulsen BB, Thorsen PB, Konge L. Using virtual-reality simulation to ensure basic competence in hysteroscopy. Surg Endosc 2019;33:2162–8.
20. Savran MM, Sorensen SM, Konge L, Tolsgaard MG, Bjerrum F. Training and assessment of hysteroscopic skills: a systematic review. J Surg Educ 2016;73:906–18.
21. Gambadauro P, Milenkovic M, Hadlaczky G. Simulation for training and assessment in hysteroscopy: a systematic review. J Minim Invasive Gynecol 2018;25:963–73.
22. Vitale SG, Caruso S, Vitagliano A, Vilos G, Di Gregorio LM, Zizolfi B, et al. The value of virtual reality simulators in hysteroscopy and training capacity: a systematic review. Minim Invasive Ther Allied Technol 2019;6:1–9.

Supplemental Digital Content

© 2020 The Author(s). Published by Wolters Kluwer Health, Inc.