Journal Logo

Article: Clinical Trial/Experimental Study

Virtual Reality Training Versus Blended Learning of Laparoscopic Cholecystectomy

A Randomized Controlled Trial With Laparoscopic Novices

Nickel, Felix MD; Brzoska, Julia A. MD; Gondan, Matthias PhD; Rangnick, Henriette M. MD; Chu, Jackson MD; Kenngott, Hannes G. MD, MSc; Linke, Georg R. MD; Kadmon, Martina MD, MME; Fischer, Lars MD; Müller-Stich, Beat P. MD

Section Editor(s): Lobo., Dileep N.

Author Information
doi: 10.1097/MD.0000000000000764
  • Open

Abstract

INTRODUCTION

Laparoscopic surgery provides advantages for the patients compared with open surgery.1 Learning to perform and manage laparoscopic surgical procedures, however, comes with additional difficulties for novice surgeons.2 The learning process is prolonged by the 2-dimensional view, challenging hand–eye coordination, difficult instrument handling, fulcrum effect, and restricted haptic feedback.3 Training of laparoscopic skills and operations outside the operating room is mandatory to reduce operative times and risks.4–8 Currently used training methods for laparoscopic surgery include live animals, cadavers, video and box trainers, and virtual reality (VR) trainers.8 Live animal training is the most realistic training option. but is expensive, ethically questionable, and not widely available.9 Training with cadaveric animal organs can provide excellent tissue feeling to simulate parts of operations but requires supervision by experienced teachers.10 Box trainers are known to be effective for the acquisition of laparoscopic basic skills but do not provide possibilities for training complete operations.11 VR offers training of basic skills, procedural skills, and complete operations in an animated setting with the possibility of monitoring training progress using automatic recording of training parameters. VR training has proven efficacy for not only acquisition of basic skills in laparoscopic surgery, but also for the improvement of clinical operative skills and patient outcome when compared with no training.12,13 VR training is independent of work hour restrictions and is ethically sound. VR training has become mandatory as preparation for surgical residents in some countries for example, Denmark.14 The disadvantages of currently available VR trainers include lack of realism, high costs, and, as a consequence, restricted availability.15

Box trainers offer high availability and low prices compared with current VR trainers.16 Box training usually depends on supervision by experienced trainers and does not offer procedural training without the use of cadaveric organs. E-learning has become increasingly popular over the last decades due to the possibilities offered by video recordings of laparoscopic operations in combination with the World Wide Web. It uses electronic media for the presentation of theoretical and practical knowledge. This includes detailed instructions and videos for operative procedures. E-learning is independent of teachers and is highly available at a low cost any time.17,18 Blended learning (BL) combines the advantages of box trainers with E-learning tools to provide basic skills training as well as procedural training with high availability at a low price.19–23 Studies comparing box trainers with VR training have not shown clear superiority of one method over the other. However, most of the existing studies had the trainees tested in one of the used training modalities.24–27 A recent meta-analysis by Larsen et al28 showed superior procedural performance for VR simulators offering training of complete operations over simulators offering only basic skills training. Box training offers basic skills training only and lacks procedural knowledge teaching, which can be compensated by E-learning in the BL concept.

The aim of this study was to compare VR training with low-cost BL for teaching the basic performance of laparoscopic cholecystectomy (LC) to laparoscopic novices in a standardized and structured training program within an adequately powered trial. Secondary questions included sex differences, influences of the participants’ personal characteristics on training outcome, and the participants’ opinions about the training modalities.

MATERIALS AND METHODS

Participants

Laparoscopy-naïve medical students in their clinical years of study were invited to participate in this study. Students with ≥2 hours of experience in laparoscopic surgery training were excluded. The participation was voluntary and the participants were allowed to leave the study at any time. The participants received information about the study and informed consent was obtained. The local ethics committee at Heidelberg University approved the study protocol before inclusion of the trainees (S-334/2011). The cadaveric porcine livers used for the simulation of operations were obtained as side products from the local food industry.

Setting and Study Design

The study was designed as a prospective monocentric, 2-arm, randomized trial with 2 active intervention groups (Flow diagram). The randomization to the VR group (n = 42) or the BL group (n = 42) was stratified for sex. The VR and BL groups were compared for the effectiveness of training a basic operation in laparoscopic surgery. Each group employed their training method (VR or BL) for gaining laparoscopic skills. The training consisted of 3 sessions of 4 hours, adding to a total training time of 12 hours per participant in each group. After completing the training, the students participated in a post-test consisting of a knowledge test and a simulated operation on a cadaver model. The study was carried out at the training center for laparoscopic surgery at the Department of General, Visceral and Transplantation Surgery at Heidelberg University.

Training Groups

Virtual Reality Training

The VR group received a total of 12 hours of laparoscopy training using the LAP Mentor II (Simbionix, Cleveland, OH). The simulator software enables training within 9 laparoscopic basic skills scenarios as well as procedural skills training in the form of partial or complete laparoscopic operations. For this study, LC was the chosen procedure. Aggarwal et al29 provided a detailed description of the simulator tasks and investigated how many repetitions of each task were generally needed to reach expert level. This information was used to develop a structured VR curriculum for the present study. The VR curriculum started with 9 basic skills tasks (9 repetitions each), followed by 4 procedural skills tasks, for example, dissection of Calot's triangle (3 repetitions each). Afterwards, the participants had to perform 6 virtual cases of LC with different patient histories and anatomical conditions (2 repetitions each). Additional repetitions of each task were performed in the remaining training time.

Blended Learning

The BL group received a total of 10 hours of box training and 2 hours of E-learning for LC training. The participants received practical basic skills training with a conventional box trainer (Karl Storz GmbH, Tuttlingen, Germany). A structured training curriculum with 9 basic skills exercises was used (shown in Table 1 of the supplemental material). For each task, characteristic errors were identified and described in the curriculum. The participants had to document the task completion time and error rate of each task. They repeated each task several times until a plateau phase in the learning curve was reached for task time and error rate. The procedural skills for LC were taught through E-learning using 2 web-based surgical education platforms with specific content concerning LC. The first 4 didactic chapters of the German website www.webop.de LC module included fundamental knowledge about the relevant anatomy, perioperative management, step-by-step approach to the operation, and management of operative complications.17,18,30 The English website www.WeBSurg.com module, “Laparoscopic cholecystectomy for symptomatic cholelithiasis with or without cholangiography” provided additional information about the procedure. The final component of E-learning was a video recording of a narrated LC on www.WeBSurg.com, “Laparoscopic cholecystectomy: a gold standard case for the dissection of Calot's triangle.”30 A tutor was present during the E-learning session for assistance and to answer questions.

Skill Testing and Recording of Data

The comparison of the investigated training methods was made with a post-test at the end of the last training session. At the end of the training, every participant had to take a multiple choice (MC) test consisting of 16 questions about LC with 1 correct answer each. MC test results were compared between the BL and the VR groups to assess the effect of E-learning. The surgical post-test was carried out on a Pulsating Organ Perfusion (POP) trainer, a mechanical laparoscopic training device for procedural laparoscopic training with explanted cadaveric animal organs that enables lifelike pulsatile perfusion of the organs with artificial blood.31–33 Participants had to perform an LC on an explanted porcine liver with preserved gallbladder and associated structures. Three specially trained raters, who were blinded to the groups of the participants, evaluated the operative performance. To ensure rater blinding, the participants were instructed to not disclose their training status to the raters. Each rater evaluated the same number of participants out of each group to avoid confounding due to rater-specific differences in evaluation.

The participants’ performances were judged using the Objective Structured Assessment of Technical Skills (OSATS), which is a standardized and validated tool for assessing surgical skills.34 OSATS consists of 2 subscores each including 7 different surgical criteria. The general technical skills (GTS) score gives an overall non-tas-specific evaluation of each participants’ performance, whereas the specific technical skills (STS) score evaluates the single steps of a specific procedure. Each GTS item is scored from 1 (worst) to 5 (best), whereas each STS item is scored in increments of 2 points from 2 (worst) to 10 (best). Consequently, a maximum of 35 points can be achieved in the GTS score and a maximum of 70 points in the STS score, yielding a maximum total score of up to 105 points. For the purpose of this study, the OSATS items used by Sarker et al35 were slightly modified. The GTS assessment was supplemented by a further criterion concerning the quantity and quality of required assistance. For the STS assessment, the criteria “access & port insertion” and “extraction of gallbladder” were replaced by “knowledge of procedure” and “quality of product.” The operation time was recorded, and the operation was discontinued when it could not be completed within 80 min; the operation time was then treated as a right-censored observation.

The participants evaluated the laparoscopic simulators and their individual training method by answering a standardized anonymous questionnaire after the post-test. Each training modality was evaluated for several training criteria on 5-point Likert scales (fully agree to fully disagree). Based on this questionnaire, differences in the acceptance of the training modalities were explored.

Outcome Measures

Primary Outcome

The OSATS total score was defined as the primary outcome measure. It was compared between the VR and BL groups.

Secondary Outcomes

Secondary outcomes were the results of the GTS and STS subscales, operation time, and rate of operations completed within 80 min. Furthermore, the results of the MC test were compared between the VR and BL groups as well as the participants’ opinions concerning the training modalities. OSATS scores of male and female participants were compared to investigate potential sex differences in laparoscopic performance. Possible correlations between participants’ baseline characteristics, questionnaire evaluations, and surgical performances (MC and OSATS results) were explored to identify factors that potentially influence surgical education, and can thus be taken into account for future research.

Randomization

The randomization process was conducted by using 2 computer-generated lists with variable block sizes, stratified for sex. The randomized assignment to the groups was implemented using closed envelopes. The envelopes were opened only after completion of all baseline assessments. The lists and envelopes were generated and handled by 2 persons who were not directly involved in the training, skills testing, and data collection.

Sample Size Calculation

Based on data reported by Sarker et al,35 a standard deviation of 7.86 for the mean OSATS total score among novices was anticipated (assuming a medium size positive correlation r = 0.5 between GTS and STS). A sample size of 80 participants (40 per group) was calculated to detect a standardized effect size of d = 0.63 at a 2-sided 5% significance level with a power of 80% using a t test for independent groups. This is equal to a mean difference of 5 points between the BL and VR groups, or slightly >1 criterion on the GTS subscale. Group differences of less than this margin of 5 points on the OSATS scale were considered to be irrelevant. A total of 84 participants were included in the study to allow for 5% dropout.

Statistical Analysis

To minimize the possibility of incorrect inputs, 2 independent persons entered the data in parallel. Data were analyzed using R version 3.1.0.36 Descriptive data are given as absolute frequency and as mean ± standard deviation. A normal distribution of OSATS mean scores was assumed,35 so that a 2 × 2 × 3 analysis of variance (Type III sums of squares, alpha 5% two-tailed) was applied to compare the groups for the primary outcome measure (OSATS total score) as well as GTS and STS scores, controlling for the stratification factor Gender and the influence of the 3 raters. The effect of the intervention is reported as difference in marginal means (“least-square means”) with the corresponding 95% confidence interval. The operation time was analyzed using Cox proportional hazards model, with hazard ratios >1 denoting shorter operation times than in the control condition. Group was again used as the main factor of interest and the analysis was stratified by Gender. Pearson correlation coefficient was applied for correlation analyses between participants’ baseline characteristics, questionnaire evaluations, and surgical performances. Differences were defined as statistically significant if the 2-tailed P value related to the group comparison was <0.05. Missing values were not imputed (available cases analysis). This decision was based on the consideration that in a real-life environment, the completion of the training and the assessment would be required for all candidates in surgery.

RESULTS

Demographics

In total, 84 medical students participated in the randomized study (42 per intervention group) and all of them completed the entire study protocol between September 15th 2012 and February 15th 2013. There were 21 female and 21 male participants in each group. All participants were novices with no laparoscopic experience and limited experience in open surgery (Table 1).

TABLE 1
TABLE 1:
Demographic Characteristics of the Participants

Surgical Test Performance

Virtual Reality Versus Blended Learning

Students in the VR training group and in the BL group showed equal performance of LC in the primary outcome OSATS (49.4 ± 10.5 vs 49.7 ± 12.0, mean difference = 0.3 OSATS units, 95% confidence interval −4.7 to 5.3, P = 0.92). Figure 1 shows box plots with the OSATS total and component scores per group. The VR and BL groups were equal for GTS (16.7 ± 5.0 vs 16.2 ± 4.9, P = 0.63), STS (32.7 ± 6.7 vs 33.5 ± 8.2, P = 0.89), and for the OSATS items (Table 2). No significant differences between the 3 raters were observed for the total and component scores of the OSATS.

FIGURE 1
FIGURE 1:
Operative performance of the participants: The performance of laparoscopic cholecystectomy (LC) was compared between the blended learning (BL) and virtual reality (VR) group. The Objective Structured Assessment of Technical Skills (OSATS) score was used to measure performance with the general technical skills (GTS) and specific technical skills (STS) subscores. Flow diagram. Participant flow diagram: Laparoscopcy-naïve medical students were randomly assigned to the VR and BL group. After participation in their individual groups’ training program, the operative performance of LC was measured on a cadaveric organ model with the OSATS by expert raters.
TABLE 2
TABLE 2:
Comparison of Objective Structured Assessment of Technical Skills Results Between the Virtual Reality Training and Blended Learning Groups on Expert Ratings of Trainee Performance

The average operation time was significantly shorter for the VR group with 75.8 ± 7.1 min compared with the BL group with 77.6 ± 7.0 min (hazard ratio = 2.39, P = 0.03). The BL group completed 9 operations compared with 19 in the VR group (21% vs 45% completed operations, P = 0.02). In contrast, the score of the knowledge test about LC was significantly better for the BL group than for the VR group (13.3 ± 1.3 vs 11.0 ± 1.7 out of 16 total points; P < 0.001).

The average operation time was significantly shorter for male students with 75.0 ± 1.3 min compared with the female students with 78.4 ± 0.1 min (hazard ratio = 2.46, P = 0.03). The female students completed 9 operations within the allowed 80 min compared with 19 completed operations for the male students (21% vs 45% proportion of completed operations, P = 0.02). Male and female participants did not show significant differences in mean OSATS scores (50.3 ± 11.3 vs 48.8 ± 11.2; P = 0.59). There was no significant difference between males and females for the STS score (32.8 ± 7.1 vs 33.4 ± 7.3; P = 0.79). A trend toward better performance of males in the GTS score was observed (17.5 ± 4.7 vs 15.5 ± 5.0; P = 0.09). This effect was <1 GTS item (4 units) and was due to slightly better scores in the 3 STS items “time and motion,” “flow of operation and forward planning,” and “extent of support needed,”

Evaluation of Training Methods by the Participants

Laparoscopic Simulators

The POP trainer used for the test received the best overall evaluation compared with the VR trainer and box trainer (Supplemental content Figure 1). Both the VR and box trainer were seen as generally helpful for laparoscopic training and helpful for training of hand–eye coordination. Concerning training for instrument coordination, the box trainer was evaluated better than the VR trainer (P = 0.04). However, the VR trainer was found to be more helpful than the box trainer for the more procedural issues: “Exact simulation of intraoperative situation” (P < 0.001), “Helpful for training tissue preparation” (P = 0.005), and “Helpful for training complete operations” (P < 0.001).

Individual Training Method

Each group evaluated their training method subjectively regarding training benefit, effects on training motivation, and changes in surgical interest, whereby no significant difference between the VR and BL groups was found. Both groups rated their individual training method as fun (Supplemental content Figure 2). Male participants felt significantly better prepared for the assistance (P = 0.008) and performance (P < 0.001) of basic laparoscopic operations than female participants. Further items concerning evaluation of training methods did not show any sex differences.

Correlation Analysis

There was a significant correlation between good OSATS results and seeing the course as good preparation for assistance and performance of laparoscopic surgery, respectively, as relevant for future choice of profession (Table 3). A positive relation between surgical interest and performance was observed for the GTS score but not for total OSATS and STS scores. There was a significant correlation between prior experience in open surgery and the GTS score, but not with the STS and total OSATS scores. Concerning the other questionnaire evaluation issues, no correlations with OSATS results were found. Concerning the MC test results, there was a significant correlation (r = 0.24; P = 0.03) between the STS item “gall bladder retraction and exposure of Calot triangle” and the MC question on the safe exposure and use of cautery near critical structures within Calot triangle.

TABLE 3
TABLE 3:
Correlation of Objective Structured Assessment of Technical Skills Scores, Prior Experience, and Questionnaire Evaluation of the Training

DISCUSSION

In the present study, the students in the VR group performed the operation faster and were more often able to complete the operation within the maximum allowed time of 80 minutes in comparison to the BL group. However, the BL group had better theoretical background knowledge about the operation as tested in a MC test. The VR and BL groups showed equal operative performance in the OSATS score, which was the primary outcome of this study. Male students were more often able to complete the operation within the maximum allowed time of 80 min, whereas the operative performance of LC in the OSATS score was similar for male and female trainees. The time limit of 80 min was set for feasibility reasons of the study. In preliminary tests, the time limit had proven sufficient for the students to succeed in performing the main steps of Calot triangle exposure, clipping and dividing of cystic duct and artery. The students who did not finish the LC in the time limit were all in the phase of dissecting the gall bladder from the liver bed. Due to the anatomy of the porcine livers that differs slightly from humans in that the gall bladder can be very encased in the liver bed, it would often have taken more time to finish the LC. However, the time was always sufficient to adequately rate the performance. The time differences between the groups and sexes were of statistical significance despite rather small differences. However, the rates of finishing the operation in the time limit were of relevant differences between the groups and sexes.

BL had not been compared with VR training in a randomized trial prior to the present study. Quite a number of studies have been performed comparing VR training and box training without showing clear superiority of one method over the other.37 Some authors promote VR trainers because of the possibility to simulate complete operations repeatedly.28 Box trainers require the insertion of cadaveric organs or artificial organ models that need to be replaced after each training session. Box training, however, is far cheaper than VR training and is more easily accessible. Prices for an equipped box trainer station range from 200 to 7600 US-$, whereas common VR trainers cost 45,000 to 120,000 US-$.16,25,26,38–40 E-learning is also easily accessible and cost-effective.41 The concept of BL uses box training for basic skills supplemented by E-learning to provide knowledge and cognitive training for the operation.42 With the same training outcome, BL thus provides higher training efficiency given the cost advantages over VR training.43

The BL group scored significantly higher in the knowledge test concerning LC in the present study. A correlation was found between theoretical knowledge of specific technical aspects of the operation and the corresponding parts of the specific technical score of the OSATS from the performed LCs in the present study. This shows the important role of cognitive training as suggested in previous studies.18,41 It also shows that the theoretical part of the operations might not be addressed sufficiently in the VR trainer. Hence, it seems important to incorporate cognitive training, for example, adding E-learning to multimodality training curriculums for laparoscopic surgery.12

Sex differences have been previously addressed in surgical education research studies.44 This has drawn particular attention to laparoscopic surgery education research, as there are suspected sex differences in the hereby-important 3-dimensional orientation and psychomotor skills.45,46 In the present study, males performed faster than females, with higher rates of completing the operation within the allowed 80 min, but without differences in the quality of the operation, as reflected by the OSATS score. This is in line with previous laparoscopy training studies showing men performing faster with equal quality of performance.47

Interestingly, in the present study, there was a positive correlation between the OSATS scores and the students’ experienced benefits from the course in terms of being able to participate in laparoscopic surgery in the OR. There was also a positive correlation between the GTS score and the students’ interests in surgery, but not with the total OSATS and STS scores. There was a correlation between experience in open surgery and the GTS. This presumably reflects a better general understanding of surgical principles by the trainees who participated in open surgeries before the present study and by those who are interested in surgery. These findings emphasize the need for surgical training courses for students, as there seem to be relations between positive experiences in surgical training and the expected ability to participate in surgery.

In the evaluation of the roles of the different training modalities, the POP trainer received the best overall evaluation by the trainees. The box trainer was seen as better for the training of basic skills, such as instrument coordination, whereas the VR trainer received better ratings for the training of procedural skills and operations. Multimodality training courses for laparoscopic surgery should therefore combine the use of these training modalities for the advantages of each, thus maximizing the benefit and efficiency of training. The present study was conducted with laparoscopy-naïve medical students in a controlled training laboratory setting. Therefore, the results cannot directly be transferred to more experienced trainees. The absence of baseline laparoscopic experience of the students, however, guaranteed good comparability of the results and minimized bias due to different baseline experiences, as is often the case in studies with surgical residents. Furthermore, we believe that the technical abilities of medical students in their clinical years of study resemble those of surgeons at the very beginning of residency. Important to note, that participation in this course was not an obligation and therefore the students in this course were all rather interested in surgery.

The operative performance tests in the present study were done with explanted cadaveric porcine organs that come close but are not identical to the anatomy found in human patients in the OR. However, the POP trainer with the porcine organs received very good evaluations by the trainees and provided realistic tissue feeling in contrast to the box and VR trainer. Training on live animals and human cadavers was not considered for this study with medical students due to financial and ethical reasons. OSATS was chosen as the measurement method and primary outcome for performance of LC on the POP trainer in the present study. OSATS has proven construct validity in previous studies.34,48,49 The raters in the present study were blinded to the training status of trainees and there was no influence of the 3 raters on the OSATS results, thus indicating no rater bias.

In conclusion, both VR training and BL were effective for the training of LC. Using the validated OSATS score as the primary outcome, the groups and sexes showed similar operative performance of LC. The VR group and males performed the operation faster and were more likely to complete the operation within the maximum given time of 80 min. The VR training was rated better for procedural training and the box trainer for basic skills training. The BL group had more knowledge about the operation and BL is the more cost-effective option. The POP trainer received the best overall rating. The superiority of any kind of training over no training for laparoscopic novices is no longer doubted in the literature.6,42,50 The best training options for different training levels, however, lack a clear consensus. In line with the results of the present study, both VR and BL can be applied for training novices the basics of LC with different advantages. Multimodality training programs for different trainee levels should combine the available modalities to their advantage.

Acknowledgments

The authors thank Jonathan D. Hendrie for proofreading of the manuscript.

REFERENCES

1. Greene FL, Kercher KW, Nelson H, et al. Minimal access cancer management. CA Cancer J Clin 2007; 57:130–146.
2. Harrysson IJ, Cook J, Sirimanna P, et al. Systematic review of learning curves for minimally invasive abdominal surgery: a review of the methodology of data collection, depiction of outcomes, and statistical analysis. Ann Surg 2014.
3. van Dongen KW, Ahlberg G, Bonavina L, et al. European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills. Surg Endosc 2011; 25:166–171.
4. Schijven MP, Jakimowicz JJ, Broeders IA, et al. The Eindhoven laparoscopic cholecystectomy training course—improving operating room performance using virtual reality training: results from the first E.A.E.S accredited virtual reality trainings curriculum. Surg Endosc 2005; 19:1220–1226.
5. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006; 244:310–314.
6. Aggarwal R, Ward J, Balasundaram I, et al. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg 2007; 246:771–779.
7. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002; 236:458–463.
8. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg 2006; 243:291–300.
9. van Velthoven RF, Hoffmann P. Methods for laparoscopic training using animal models. Curr Urol Rep 2006; 7:114–119.
10. Undre S, Darzi A. Laparoscopy simulators. J Endourol 2007; 21:274–279.
11. Nagendran M, Toon CD, Davidson BR, et al. Laparoscopic surgical box model training for surgical trainees with no prior laparoscopic experience. Cochrane Database Syst Rev 2014; 1:CD010479.
12. Palter VN, Grantcharov TP. Development and validation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg 2012; 256:25–32.
13. Palter VN, Orzech N, Reznick RK, et al. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg 2013; 257:224–230.
14. Strandbygaard J, Bjerrum F, Maagaard M, et al. A structured four-step curriculum in basic laparoscopy: development and validation. Acta Obstet Gynecol Scand 2014; 93:359–366.
15. Nickel F, Bintintan VV, Gehrig T, et al. Virtual reality does not meet expectations in a pilot study on multimodal laparoscopic surgery training. World J Surg 2013; 37:965–973.
16. Orzech N, Palter VN, Reznick RK, et al. A comparison of 2 ex vivo training curricula for advanced laparoscopic skills: a randomized controlled trial. Ann Surg 2012; 255:833–839.
17. Pape-Koehler C, Chmelik C, Aslund AM, et al. An interactive and multimedia-based manual of surgical procedures: Webop—an approach to improve surgical education. Zentralbl Chir 2010; 135:467–471.
18. Pape-Koehler C, Immenroth M, Sauerland S, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc 2013; 27:1737–1747.
19. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg 2012; 99:1322–1330.
20. Evgeniou E, Loizou P. The theoretical base of e-learning and its role in surgical education. J Surg Educ 2012; 69:665–669.
21. Funke K, Bonrath E, Mardin WA, et al. Blended learning in surgery using the Inmedea Simulator. Langenbecks Arch Surg 2013; 398:335–340.
22. Kadmon M, Busemann A, Euteneier A, et al. Modular postgraduate training in surgery—a national concept with future. Zentralbl Chir 2012; 137:138–143.
23. Rieger UM, Pierer K, Farhadi J, et al. Effective acquisition of basic surgical techniques through blended learning. Chirurg 2009; 80:537–543.
24. Gurusamy KS, Nagendran M, Toon CD, et al. Laparoscopic surgical box model training for surgical trainees with limited prior laparoscopic experience. Cochrane Database Syst Rev 2014; 3:CD010478.
25. Nagendran M, Gurusamy KS, Aggarwal R, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev 2013; 8:CD006575.
26. Gurusamy KS, Aggarwal R, Palanivelu L, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev 2009. CD006575.
27. Sturm LP, Windsor JA, Cosman PH, et al. A systematic review of skills transfer after surgical simulation training. Ann Surg 2008; 248:166–179.
28. Larsen CR, Oestergaard J, Ottesen BS, et al. The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials. Acta Obstet Gynecol Scand 2012; 91:1015–1028.
29. Aggarwal R, Crochet P, Dias A, et al. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 2009; 96:1086–1093.
30. Mutter D, Vix M, Dallemagne B, et al. WeBSurg: an innovative educational Web site in minimally invasive surgery—principles and results. Surg Innov 2011; 18:8–14.
31. Clerici T, Lange J, Zerz A, et al. Educational opportunities in minimally invasive surgery. Wien Klin Wochenschr 1995; 107:43–48.
32. Szinicz G, Beller S, Bodner W, et al. Simulated operations by pulsatile organ-perfusion in minimally invasive surgery. Surg Laparosc Endosc 1993; 3:315–317.
33. Szinicz G, Beller S, Zerz A. Role of the pulsatile organ perfusion surgical simulator in surgery education. Langenbecks Arch Chir Suppl Kongressbd 1997; 114:687–693.
34. van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg 2010; 97:972–987.
35. Sarker SK, Chang A, Vincent C, et al. Development of assessing generic and specific technical skills in laparoscopic surgery. Am J Surg 2006; 191:238–244.
36. Team RCR. A language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2014.
37. Diesen DL, Erhunmwunsee L, Bennett KM, et al. Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ 2011; 68:282–289.
38. Debes AJ, Aggarwal R, Balasundaram I, et al. A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. Am J Surg 2010; 199:840–845.
39. Aggarwal R, Moorthy K, Darzi A. Laparoscopic skills training and assessment. Br J Surg 2004; 91:1549–1558.
40. Aggarwal R, Darzi A. Training in laparoscopy—which model to use? Indian J Gastroenterol 2005; 24:95–96.
41. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ 2010; 10:12.
42. Immenroth M, Bürger T, Brenner J, Nagelschmidt M, Eberspächer H, Troidl H. Mental training in surgical education: a randomized controlled trial. Ann Surg 2007; 245:385–391.
43. Clevin L, Grantcharov TP. Does box model training improve surgical dexterity and economy of movement during virtual reality laparoscopy? A randomised trial. Acta Obstet Gynecol Scand 2008; 87:99–103.
44. Baxter N, Cohen R, McLeod R. The impact of gender on the choice of surgery as a career. Am J Surg 1996; 172:373–376.
45. Madan AK, Harper JL, Frantzides CT, et al. Nonsurgical skills do not predict baseline scores in inanimate box or virtual-reality trainers. Surg Endosc 2008; 22:1686–1689.
46. Thorson CM, Kelly JP, Forse RA, et al. Can we continue to ignore gender differences in performance on simulation trainers? J Laparoendosc Adv Surg Tech A 2011; 21:329–333.
47. Grantcharov TP, Bardram L, Funch-Jensen P, et al. Impact of hand dominance, gender, and experience with computer games on performance in virtual reality laparoscopy. Surg Endosc 17:1082–1085.
48. Hiemstra E, Kolkman W, Wolterbeek R, et al. Value of an objective assessment tool in the operating room. Can J Surg 2011; 54:116–122.
49. Kundhal PS, Grantcharov TP. Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc 2009; 23:645–649.
50. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg 2011; 254:502–509.
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.