Bowling, C. Bryce MD; Greer, W. Jerod MD; Bryant, Shannon A. MD; Gleason, Jonathan L. MD; Szychowski, Jeff M. PhD; Varner, R. Edward MD; Holley, Robert L. MD; Richter, Holly E. PhD, MD
Cystourethroscopy has long been used by gynecologists for diagnostic and operative indications.1 However, a unified system for training residents in cystourethroscopy and documenting competence has not been described by the American College of Obstetricians and Gynecologists (the College). Many residents in obstetrics and gynecology are not exposed to enough intraoperative cystoscopy to confidently identify lower urinary tract injury and differentiate normal from abnormal findings. Recently, the College released a Committee Opinion that stated that postgraduate education in obstetrics and gynecology should include education in the instrumentation, technique, and evaluation of findings of cystourethroscopy, as well as in the pathophysiology of diseases of the lower urinary tract.1
Bench models and surgical simulators have gained popularity as efficient methods of providing training outside the operating room and for providing a means of testing competence; however, cystoscopy models used in resident training can approach a cost of $60-5243 R000.00.2,3 Using easily obtainable materials, we designed a previously described low-cost cystoscopy model using a balloon, which resembles a normal bladder and urethra complete with ureteral orifices, vessels, and different pathologies (Fig. 1).4 The model was originally developed to provide an inexpensive and available method for surgeons in training to learn, perform, and practice cystourethroscopy in an effective environment before performing this technique in the operating room.
The objective of this study was to validate that model's ability to effectively train obstetrics and gynecology residents in performing cystourethroscopy.
MATERIALS AND METHODS
This was a randomized, controlled, and evaluator-blinded study, to validate the bladder model as an effective teaching tool. Study approval was obtained from the institutional review board of The University of Alabama, Birmingham. The institutional review board protocol number is X090707009. Afterward, 29 obstetric and gynecology residents from all training levels (PGY-1 through PGY-4) were recruited from The University of Alabama at Birmingham. With the assistance of professors from the university's Department of Anatomy, all resident participants had access to fresh-frozen, female cadavers with intact urethra, bladder, and ureteral orifices on which to perform a predetermined series of cystoscopy skills. The skills included proper assembly of the cystoscope, proper setup of additional cystoscopy equipment (distension medium, tubing, camera, and light source), as well as performing a series of predetermined bladder and urethra survey tasks, including identification of abnormalities. A single attending physician, with experience in cystourethroscopy (R.L.H.), was the blinded examiner, scoring the resident physician's baseline and follow-up cystoscopic evaluations. The residents wore gowns and gloves, and a series of drapes were used to ensure that the examiner remained blinded to the residents' identities. Drapes were placed between the examiner and the examinee at a specific height that allowed the examiner to see the gowned resident from the abdomen down, the end of the examination table, the residents' hand movements with the cystoscope, the cadaver, and the video monitor. Further, the examiner was also acoustically blinded using ear buds and an MP3 player.
Individual scores were assigned using the validated Objective Structured Assessment of Technical Skills (OSATS) checklist and global rating scale as originally described by Reznick and colleagues.5,6 The OSATS is made up of a series of elements scoring a participant's level of operative performance. Several OSATS scoring checklists have been created and validated over the years for basic surgical skills, such as knot-tying and laceration repair, and for more specialized skills, such as control of hemorrhage and cystoscopy.2,7 The cystoscopy-specific OSATS scoring elements used in this study have been previously evaluated and determined to be reliable and valid for scoring cystoscopy technique.2
The first scored evaluation is a task-specific checklist (Appendix 1) used to assess the surgeon's ability to perform a predetermined series of surgical skills. It consists of a series of yes/no choices and Likert scales to assign a specific score. The checklist assesses cystoscope assembly, bladder survey tasks, and overall technique and also provides spaces for examiner comments and assembly and survey times. Some of the skills the resident needed to demonstrate were proper assembly of the scope and using the correct lens and sheath size, choosing the correct distension media, observing the ureteral orifices, changing out the lens to properly survey the urethra, and observing abnormalities. Before the laboratory session, the authors placed a small amount of mid-urethral sling mesh through the dome of the bladder to create an abnormality.
Because fresh-frozen cadavers were used, certain skills of the validated scoring elements could not be directly assessed, such as observing ureteral efflux. A small quiz (Appendix 2) was incorporated at the end of each skills laboratory to assess the residents' knowledge of that portion of the process, and their answers were used to complete the OSATS checklist scoring.
The second OSATS-scored evaluation is a 5-point global rating scale (Appendix 3), which assigns a number from 1 to 5 to the surgeon's performance of the tested skills. The residents' proficiency in skills ranging from “respect for tissue” and “instrument handling” to “flow of operation” and “knowledge of procedure” were evaluated and scored, with a 1 being assigned to poor performance and a 5 assigned to excellent performance. Numbers 2, 3, and 4 are used as intermediate scores. In addition, the evaluator using the global checklist was asked to determine whether “Overall, on this task, should this candidate pass or fail?” Overall scores for both the task-specific checklist (maximum score 49) and the global rating scale (maximum score 35) were converted to a 0–100 scale.
After the initial cadaver laboratory and baseline scoring, the residents were randomized by resident year to one of two study arms. Randomization was performed by using a computerized, four-block randomization program. Randomization results were revealed to study participants 48 hours before the intervention. The first arm consisted of residents who were given a 2-hour didactic session on setup, instrumentation, and proper use of cystourethroscopy using both the described balloon teaching model and Olympus and ACMI cystoscopes (study arm). Skills taught using the cystoscopy model and didactic session included proper cystoscope assembly and proper setup of additional equipment (sterile water as correct fluid; fluid source set at correct height; correct setup of tubing, camera, and light source). Bladder surveillance was taught by having the trainee identify a random number of permanent “dots” drawn on the inside surface of the balloon model. The ability to find the correct number of dots within the bladder model would be seen as “satisfactory” for completing the didactic session. Identification of simulated ureteral orifices or simulated bladder pathologies or both was also performed. The technique of simulating ureteral orifices, the urethra, and other model features has been previously described.4 The second arm consisted of resident “controls” that did not undergo the 2-hour didactic session using the model.
All residents then repeated the cystoscopy evaluation using the cadavers within 1–2 weeks. The residents were asked to perform the same series of cystoscopic skills as performed at baseline. They were again scored using the validated OSATS scoring elements by the same blinded examiner. Additionally, the general quiz was repeated to assess any change in general cystoscopy knowledge.
Fisher exact test was used to assess the randomization balance of resident sex and years of training. Times and scores were reported as mean plus or minus standard deviation. The study was designed to compare improvement in scores with independent t tests. Based on previous studies,2 we estimated the within-group standard deviation to be as low as 5 points, and concluded that a sample size of 28 (14 in each group) would provide above 90% power to detect a 10-point improvement (equivalent to one letter grade) in test scores. Before final statistical analysis, the assumption of normality was tested and nonparametric tests were used as appropriate. Equality of all measures at the first evaluation was assessed with the Wilcoxon rank-sum test. To validate the balloon model didactic as an effective teaching tool, the changes in times and scores were also evaluated with the Wilcoxon rank-sum test. The signed-rank test assessed score improvements for both the study and control arms. In addition, construct validity, an assessment of the skills of each trainee and whether scores correlate with the trainee's PGY level, was evaluated with the Kruskal-Wallis test with the year of clinical training serving as the independent variable.
The residents were enrolled the month before the initiation of the baseline testing (July 1–31, 2009). A total of 28 of 29 residents completed both the baseline and postrandomization assessments in the trial, n=14 in each study arm. There was no significant difference in sex composition (P=.17) or PGY year (P=1.00) in either arm (Table 1). No difference was noted in assembly time, task-specific checklist scores, or global rating scale scores between the groups at baseline; specifically, checklist scores were 59.3 plus or minus 24.5 and 55.7 plus or minus 17.0 for the study arm and control arm respectively (P=.62); global rating scale scores were 61.0 plus or minus 20.7 and 64.4 plus or minus 17.8 for the study arm and control arm respectively (P=.47) (Table 2). There was a significant difference (P=.047) in cystourethroscopy survey time at baseline, with the study group having a baseline time averaging more than 50 seconds longer than that of the control group.
After the study arm underwent didactic training using the bladder model and both groups repeated the cadaver skills assessment, a statistically significant change in cystoscope assembly time was seen between arms (P=.031) and a significant decrease in assembly time was seen within the study arm (P=.005). There were no statistically significant changes in bladder survey time in either the study arm (P=.682) or the control arm (P=.221) after didactics. Further, there was no significant difference in change in cystourethroscopy bladder survey time (P=.694) between the two arms after didactics (Table 2).
The study arm demonstrated a significant improvement in the task-specific checklist (P<.001), global rating scale (P<.001), and knowledge quiz (P=.002) scores after the didactic session using the model. There were no differences in any outcome measure within the control group between the first and second evaluations. Further, there was a significant improvement in task-specific checklist (P<.001), global rating scale (P=.002), and knowledge quiz (P=.011) scores in the study arm compared with the control arm (Table 2).
An assessment of whether scores correlated with the trainee's PGY level (construct validity), demonstrated that third- and fourth-year residents scored higher than first- and second-year residents, in both arms, on both the task-specific checklist scores (P<.001) and the global rating scale scores (P<.001) at baseline. However, after the didactic session using the model, participants in the study arm showed no difference in task-specific checklist (P=.516) or global rating scale (P=.514) scores regardless of their PGY level of training. The control arm participants continued to demonstrate higher scores correlating with higher PGY level.
Our study showed that residents using the cystoscopy teaching model combined with a didactic session resulted in a significant improvement in overall cystoscopic abilities compared with their baseline evaluations and compared with the control group. In addition, we demonstrated that after the didactic session using the model, all participants in the study arm showed no difference in task-specific checklist or global rating scale scores regardless of their PGY level of training. That is, after the didactic session using the model, study arm PGY-4s' and PGY-3s' scores were not significantly different from study arm PGY-2s' or PGY-1s' on cystoscopic skills or cystoscopic knowledge. This demonstrates the model's ability to rapidly provide lower-level residents with the same cystoscopic skills as their senior counterparts.
Decreased operative time does not always translate to superior surgical skill; however, the cystoscopy specific OSATS scoring elements evaluate the participant's cystoscope assembly time and cystourethroscopy survey time. A statistically significant decrease in cystoscope assembly time was noted in the study arm after the didactic, whereas no significant difference was noted in the control group. This would suggest an increased knowledge of and ease associated with working with the instruments.
There was no significant difference in cystourethroscopy survey time in the study group after training; however, the study arm did have a higher cystourethroscopy survey time compared with the control arm. In this particular case, this finding appears to reflect essentially no change in the study arm survey time after training, with a trend toward a slight decrease in surveillance time in the control arm. Increased survey time after training could theoretically result from a more thorough bladder survey among the study arm participants, as well as reflect, for example, an increased time in switching lenses for proper survey of the urethra. However, the study group's survey time was increased at baseline compared with the controls despite randomization, which most likely happened by chance. We do not suggest that these findings necessarily correlate with increased cystoscopic skill. In fact, cystoscopic skill may have more of a direct correlation with cystoscope assembly time, rather than survey time, as our findings suggest.
Certification of cystoscopic abilities and a more thorough cystoscopic evaluation by gynecologists is important, as the ability to identify lower urinary tract injuries, complications, and pathology within the bladder will lead to earlier diagnosis, improved treatment modalities, and it is hoped a better understanding of pelvic floor anatomy. In addition, with the increasing placement of mid-urethral slings for stress incontinence and increasing surgical intervention for pelvic organ prolapse repair, the need for cystoscopy will continue to be an important component of residency training in the coming years.
Not all ob-gyn residency programs have formal cystoscopy teaching laboratories available to their residents, despite the College's recommendation for training all ob-gyn residents in cystoscopy. In addition to the new College recommendations, there is a growing body of evidence suggesting that not only should cystoscopy be performed at the time of routine hysterectomy,8,9 but that it is also cost-effective10 if ureteral injury rates exceed 1.5% for abdominal approaches and 2.0% for vaginal or laparoscopic approaches. Owing to new recommendations and an increasing call for routine cystoscopy in our field, the teaching of cystoscopy basics needs to be robustly addressed.
One way to address this need is with the use of surgical simulators and bench models; however, this is not always feasible because of their cost. We showed that a low-cost alternative can be used effectively. In our study, the residents' abilities to assemble the cystoscope and perform cystourethroscopy significantly improved after residents completed the bladder model didactic. It should be stressed that the cost to build each of these low-cost bladder models is less than $1,4 thus constituting an alternative for any training program.
The strengths of this study include its randomized, controlled design and its objective of answering an important question regarding cystoscopic training for ob-gyn residents. In addition, the study arm residents and control residents were from the same training site; this is different from previous controlled, OSATS projects in which branching out to other sites was performed but seen as a limitation.2 Additionally, we find strength in having a single, visually and acoustically blinded examiner. This allowed a fair and unbiased scoring system and further strengthens our results and conclusions. The study also lays the groundwork for follow-up studies, which may be used to determine the long-term effect of this training approach.
The main limitation of this study lies in its single-study nature with no long-term re-evaluation. The validity of an instrument shows the extent to which a test or series of tests can measure what the instrument was intended to do. As a measurement tool's validity is normally measured in part by reproducibility, it typically requires more than a single study. Because this was a single study, construct validity was substituted to demonstrate its validity, which is a practice we see in other studies assessing surgical skills.2 Despite this limitation, we still feel the balloon model is a valid instrument in resident training.
In conclusion, we showed a statistically significant increase in skill level of resident participants using the balloon model compared with to those who did not randomize to a didactic session using the model. This low-cost cystoscopy model is a valid and effective teaching tool and may improve clinical performance and knowledge of cystourethroscopy among other obstetrics and gynecology residency programs.
© 2010 by The American College of Obstetricians and Gynecologists.