Siddighi, Sam MD; Kleeman, Steven D. MD; Baggish, Michael S.; Rooney, Christopher M. MD; Pauls, Rachel N. MD; Karram, Mickey M. MD
From the Division of Urogynecology, Reconstructive Pelvic Surgery, Department of Obstetrics and Gynecology, Good Samaritan Hospital, Cincinnati, Ohio.
The authors thank David Dhanraj, MD, for his assistance with the pilot study that helped to establish interrater reliability, Shane Gamble, for his assistance with the figures, and Vicki Dryfhout, PhD, for her assistance with the statistics.
Corresponding author: Sam Siddighi, MD, 1138 Carney St. Unit #3, Cincinnati, OH 45202; e-mail: firstname.lastname@example.org.
OBJECTIVE: To develop a valid and reliable tool to objectively measure surgical skill necessary for repair of fourth-degree perineal lacerations and then to use this tool to measure improvement after a workshop.
METHODS: We measured baseline surgical ability and clinical knowledge of 26 residents (postgraduate year [PGY]-1 to PGY-4) using the Objective Structured Assessment of Technical Skills (OSATS) and a written examination. The OSATS consists of a global surgical skills assessment (OSATS-G), a procedure checklist (OSAT-C), and pass/fail grade. Five weeks after our baseline evaluation, a 1.5-hour workshop was administered to approximately half of the 26 residents (n=14). One week after this intervention, the residents were re-examined using the same assessment tools.
RESULTS: The OSATS demonstrated construct validity as scores on the examination increased on both the OSATS-G and the OSATS-C from PGY-1 through PGY-4 (P=.001 and P=.041, respectively). Reliability indices for the OSATS were high. Eighty-one percent of the residents failed the OSATS before intervention because of failure to identify and repair the internal anal sphincter. After educational intervention, senior residents improved on all assessments (OSATS-G, P=.041; OSATS-C, P=.004; written examination, P=.008), and all residents passed the OSATS.
CONCLUSION: A valid and reliable OSATS and written examination were developed to assess surgical skills, knowledge, and judgment necessary to properly manage fourth-degree perineal lacerations. Residents improved on the OSATS and the written examination after undergoing a structured educational workshop.
LEVEL OF EVIDENCE: II
Up to 50% of women experience some degree of fecal incontinence after anal sphincter repair.1–4 Three fourths of women who have been repaired after a third- or fourth-degree tear have persistent defect in the anal sphincter musculature 15 months after a repair.5 Sphincter injuries that are recognized at the time of delivery are repaired most times by a junior resident physician who is unfamiliar with the anatomy.6 Furthermore, with the decline in forceps deliveries and reduction of resident work hours, all graduating obstetrics and gynecology residents are acquiring less experience in the repair of these advanced perineal lacerations.7 Training and testing on commercial anatomic replicas may be helpful in enhancing the acquisition of technical skills for repair of these advanced perineal lacerations
In-training (eg, Council on Resident Education in Obstetrics and Gynecology) as well as written and oral board examinations assess cognitive knowledge and clinical judgment. Technical proficiency, however, is not easily evaluated. Case logs and faculty evaluations are imprecise and unreliable methods of evaluating physician competence. Unfortunately, most resident training programs rely on these subjective faculty assessments.8,9
The Objective Structured Assessment of Technical Skills (OSATS) is a valid and reliable method of measuring surgical skills, and it has been validated for different types of procedures in both general surgery and obstetrics and gynecology.10–15 Early in its development, it was demonstrated that OSATS measures using bench models is equivalent to those using live animals.11 Advantages of inanimate anatomic models include availability, lower cost, and portability, and use of the models obviates the ethical issues involved in the use of live animals.
The purpose of this study was to develop a valid and reliable OSATS to measure surgical skill necessary for proper repair of fourth-degree perineal laceration. We hypothesize that technical skills, knowledge, and judgment pertinent to these repairs will improve after a structured educational workshop.
MATERIALS AND METHODS
The OSATS consists of a global surgical skills assessment (OSATS-G), a procedure checklist (OSATS-C), and a pass/fail grade. Table 1 describes OSATS in detail. We used the original global assessment evaluation form developed by Reznick9 to assess general surgery residents performing inguinal hernia repairs. We developed our own procedure checklist for fourth-degree perineal laceration repair following the recommendations in Williams Obstetrics, 22nd edition.16 Finally, we gave a pass or fail grade based on criteria described in Table 1.
The residents arrived in the testing room, which contained the fourth-degree laceration anatomic replica, an adjustable light, and appropriate and multiple distracter sutures and instruments. The same clinical scenario was read to all of the residents, and then standardized instructions were given. The examiners acted as surgical assistants, did whatever was asked of them, but did not give input or assistance unless they were asked. The extent of the obstetric laceration was not stated. Four evaluators completed the OSATS score sheets for each resident.
Twenty-six obstetrics and gynecology residents from TriHealth Hospitals in Cincinnati, Ohio, participated in baseline assessment and reassessment using OSATS. Testing and retesting were separated by 6 weeks because each session lasted more than 6 hours (7 am to 1:30 pm) and interfered with clinical duties. Therefore, the chairman of the department permitted testing only during the monthly resident education day set aside for the Division of Urogynecology and Reconstructive Pelvic Surgery. In this study, we were interested in studying the short-term effects of a workshop. Thus, our educational intervention was scheduled exactly 1 week before the retesting date. Residents were re-examined with the same assessment tools. The intervention group, which consisted of about half (n=14) of the residents from each level of training, underwent a 1.5-hour workshop (Table 2). Residents at Trihealth residency program are evenly split between two hospitals that are geographically separated. To minimize interaction between the intervention and control groups, those residents doing their rotation at Good Samaritan Hospital during our study period were assigned to the intervention group.
Residents repaired fourth-degree laceration on a commercial anatomic replica, the Sultan Anal Sphincter (Limbs and Things Inc, Bristol, UK) (Fig. 1). This model comes with a replaceable perineal pad that allows approximately 24 repairs before replacement is necessary. Each model (skin, 1 perineal pad, base, and clamps) costs approximately $642.94. Each additional replacement perineal pad costs $109.22.
This study was reviewed and approved by the Good Samaritan Hospital Institutional Review Board and funded by the Hatton Institute for Research and Education. The four evaluators underwent a 30-minute examiner orientation before the study was begun. They also participated in a pilot study to demonstrate high interrater reliability on the OSATS-C and OSATS-G.
Statistical analyses were performed with SPSS 14.0 for Windows (SPSS Inc, Chicago, IL). The consistency of our tests was measured by test-retest, interrater, and internal consistency reliability (Cronbach α) indices. Construct validity is a proxy for validity or the extent to which a test measures what it is intended to measure. Because of ordinal measures and small group–specific sample sizes, Kruskal-Wallis and Dunn tests were used to establish construct validity. Fischer exact and t tests were used to establish similarity between the control group and the intervention group. Spearman ρ correlation coefficients were used to measure correlation between OSATS-G, OSATS-C, and written examination scores. P<.05 was considered statistically significant.
Table 3 demonstrates a rising trend in the scores from postgraduate year (PGY)-1 to PGY-4. In Table 4, construct validity is established as senior residents perform better than junior residents on the OSATS. Our assessment techniques were highly reliable (Table 5). The written examination did not demonstrate construct validity (P=.052), but its test-retest reliability was high (0.91). Eighty-one percent of all residents (35% of senior residents) failed the baseline assessment because they failed to recognize and repair the internal anal sphincter.
The control group and the intervention group did not differ in demographic characteristics. Both groups also had similar baseline composite scores on the OSATS (Table 6). The intervention group performed better than the control group on the OSATS-G, OSATS-C, and written examination (Table 7). Additionally, the PGY-2, PGY-3, PGY-4 residents improved in at least one of the assessments. The senior residents improved on all assessments after the intervention (Table 8). All residents passed the OSATS after intervention.
Table 9 shows the Spearman ρ correlation between the different assessment techniques. There was moderate correlation between OSATS-G and OSATS-C before intervention. After intervention, both OSATS-C versus with OSATS-G and OSATS-C versus with written examination were strongly correlated.
The OSATS is a valid and highly reliable assessment tool and has been used successfully with several types of surgical procedures to measure resident physician operating skill proficiency. The OSATS assessment using commercial anatomic replicas allows residency programs and board examiners to test technical skill. It also offers the advantages of availability, portability, and cost-savings over live animals and cadavers. In our study, we demonstrated that our OSATS for fourth-degree laceration repairs is both valid and highly reliable. A statistically significant rising trend in scores on both the checklist and the global assessment tools was observed as one progressed from junior to more senior level residents.
Correlations between the OSATS-G and OSATS-C and between the OSATS-C and written examination were high after our workshop. This suggests that a written examination by itself, which is easier to administer and less time-consuming, may suffice as a technical assessment tool if administered after a highly structured workshop.
Approximately 81% of all residents, including 35% of senior residents, failed the OSATS because they did not recognize the existence of internal anal sphincter and made no attempt to repair it. We doubt that this occurred because of the inadequacies of our commercial anatomic replica because the internal anal sphincter layer is obvious and prominent (Fig. 2). To highlight the gravity of this issue, our residents are above the national average for the number of vaginal deliveries, instrument deliveries, and third- and fourth-degree laceration repairs. In 2005, one of the two TriHealth Hospitals had 4,401 vaginal deliveries, and 11% of these were forceps deliveries while 6% were vacuum deliveries. Residents at this hospital were exposed to 461 third-degree and 159 fourth-degree lacerations. Since the internal anal sphincter makes up 70–80% of the resting tone of the anal sphincter, it is very important to recognize and fix this structure properly. Thus, a nationwide educational effort is needed to address this problem.
The limitations of this study are as follows. First, because of the need to minimize interaction between the two groups, the intervention was not assigned by an accepted randomized method such as computer generation or random digit table. However, we doubt that this was a source of bias since residents are not assigned to their rotations in any systematic fashion. Additionally, baseline OSATS scores for the two groups were highly comparable. Another limitation is that sample size was small and residents were trained and tested by familiar faculty.
We did not study the diminution of testing effects with time or the long-term effects of our intervention. We also did not study the effects of varying our educational intervention. For example, one may study the effects of different types of intervention (didactic versus video versus self-practice).
In conclusion, we developed a valid assessment tool that was able to reliably quantify surgical skill. We were able to objectively measure improvement in surgical ability after an educational intervention.
1. Nikiteas N, Korsgen S, Kumar D, Keighley MR. Audit of sphincter repair: factors associated with poor outcome. Dis Colon Rectum 1996;39:1164–70.
2. Morren Gl, Hallbook O, Nystrom PO, Baeten CG, Sjodahl R. Audit of anal-sphincter repair. Colorectal Dis 2001;3:17–22.
3. Walsh CJ, Mooney EF, Upton GJ, Motson RW. Incidence of third-degree perineal tears in labour and outcome after primary repair. Br J Surg 1996;83:218–21.
4. Karoui S, Leroi AM, Koning E, Menard JF, Michot F, Denis P. Results of sphincteroplasty in 86 patients with anal incontinence. Dis Colon Rectum 2000;43:813–20.
5. Pinta TM, Kylanpaa ML, Salmi TK, Teramo KA, Luukkonen PS. Primary sphincter repair: are the results of the operation good enough? Dis Colon Recum 2004;47:18–23.
6. Sultan AH, Hudson CN. Are junior doctors and midwives adequately prepared to repair the perineum? J Obstet Gynaecol 1993;13:484–5.
7. Blanchard MH, Amini SB, Frank TM. Impact of work hour restrictions on resident case experience in an obstetrics and gynecology residency program. Am J Obstet Gynecol 2004;191:1746–51.
8. Streiner Dl. Global rating scales. In: Neufeld VR, Norman GR, editors. Assessing clinical competence. New York (NY): Springer; 1985. p. 119–41.
9. Reznick RK. Teaching and testing technical skills. Am J Surg 1993;165:358–61.
10. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a structured technical skills assessment form. Am J Surg 1994;167:423–7.
11. Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, et al. Objective Structured Assessment of Technical Skills (OSATS) for surgical residents. Br J Surg 1997;84:273–8.
12. Mandel LS, Lentz GM, Goff BA. Teaching and evaluating surgical skills. Obstet Gynecol 2000;95:783–5.
13. Goff BA, Lentz GM, Lee D, Fenner D, Morris J, Mandel LS. Development of an objective structured assessment of technical skills for obstetrics and gynecology residents. Obstet Gynecol 2000;96:146–50.
14. Nielsen PE, Foglia LM, Mandel LS, Chow GE. Objective structured assessment of technical skills for episiotomy repairs. Am J Obstet Gynecol 2003;189:1257–60.
15. VanBlaricom AL, Gogg BA, Chinn M, Icasiano MM, Nielsen P, Mandel L. A new curriculum for hysteroscopy training as demonstrated by an objective structured assessment of technical skills (OSATS). Am J Obstet Gyencol 2005;193:1856–65.
16. Cunningham FG, Leveno KJ, Bloom SL, Hauth JC, Gilstrap III LC, Wenstrom KD. Williams obstetrics. 22nd ed. New York (NY): McGraw-Hill; 2005. p. 435–8.
Figure. No caption available.