Laparoscopic surgery provides advantages for the patients compared with open surgery.1 Learning to perform and manage laparoscopic surgical procedures, however, comes with additional difficulties for novice surgeons.2 The learning process is prolonged by the 2-dimensional view, challenging hand–eye coordination, difficult instrument handling, fulcrum effect, and restricted haptic feedback.3 Training of laparoscopic skills and operations outside the operating room is mandatory to reduce operative times and risks.4–8 Currently used training methods for laparoscopic surgery include live animals, cadavers, video and box trainers, and virtual reality (VR) trainers.8 Live animal training is the most realistic training option. but is expensive, ethically questionable, and not widely available.9 Training with cadaveric animal organs can provide excellent tissue feeling to simulate parts of operations but requires supervision by experienced teachers.10 Box trainers are known to be effective for the acquisition of laparoscopic basic skills but do not provide possibilities for training complete operations.11 VR offers training of basic skills, procedural skills, and complete operations in an animated setting with the possibility of monitoring training progress using automatic recording of training parameters. VR training has proven efficacy for not only acquisition of basic skills in laparoscopic surgery, but also for the improvement of clinical operative skills and patient outcome when compared with no training.12,13 VR training is independent of work hour restrictions and is ethically sound. VR training has become mandatory as preparation for surgical residents in some countries for example, Denmark.14 The disadvantages of currently available VR trainers include lack of realism, high costs, and, as a consequence, restricted availability.15
Box trainers offer high availability and low prices compared with current VR trainers.16 Box training usually depends on supervision by experienced trainers and does not offer procedural training without the use of cadaveric organs. E-learning has become increasingly popular over the last decades due to the possibilities offered by video recordings of laparoscopic operations in combination with the World Wide Web. It uses electronic media for the presentation of theoretical and practical knowledge. This includes detailed instructions and videos for operative procedures. E-learning is independent of teachers and is highly available at a low cost any time.17,18 Blended learning (BL) combines the advantages of box trainers with E-learning tools to provide basic skills training as well as procedural training with high availability at a low price.19–23 Studies comparing box trainers with VR training have not shown clear superiority of one method over the other. However, most of the existing studies had the trainees tested in one of the used training modalities.24–27 A recent meta-analysis by Larsen et al28 showed superior procedural performance for VR simulators offering training of complete operations over simulators offering only basic skills training. Box training offers basic skills training only and lacks procedural knowledge teaching, which can be compensated by E-learning in the BL concept.
The aim of this study was to compare VR training with low-cost BL for teaching the basic performance of laparoscopic cholecystectomy (LC) to laparoscopic novices in a standardized and structured training program within an adequately powered trial. Secondary questions included sex differences, influences of the participants’ personal characteristics on training outcome, and the participants’ opinions about the training modalities.
MATERIALS AND METHODS
Laparoscopy-naïve medical students in their clinical years of study were invited to participate in this study. Students with ≥2 hours of experience in laparoscopic surgery training were excluded. The participation was voluntary and the participants were allowed to leave the study at any time. The participants received information about the study and informed consent was obtained. The local ethics committee at Heidelberg University approved the study protocol before inclusion of the trainees (S-334/2011). The cadaveric porcine livers used for the simulation of operations were obtained as side products from the local food industry.
Setting and Study Design
The study was designed as a prospective monocentric, 2-arm, randomized trial with 2 active intervention groups (Flow diagram). The randomization to the VR group (n = 42) or the BL group (n = 42) was stratified for sex. The VR and BL groups were compared for the effectiveness of training a basic operation in laparoscopic surgery. Each group employed their training method (VR or BL) for gaining laparoscopic skills. The training consisted of 3 sessions of 4 hours, adding to a total training time of 12 hours per participant in each group. After completing the training, the students participated in a post-test consisting of a knowledge test and a simulated operation on a cadaver model. The study was carried out at the training center for laparoscopic surgery at the Department of General, Visceral and Transplantation Surgery at Heidelberg University.
Virtual Reality Training
The VR group received a total of 12 hours of laparoscopy training using the LAP Mentor II (Simbionix, Cleveland, OH). The simulator software enables training within 9 laparoscopic basic skills scenarios as well as procedural skills training in the form of partial or complete laparoscopic operations. For this study, LC was the chosen procedure. Aggarwal et al29 provided a detailed description of the simulator tasks and investigated how many repetitions of each task were generally needed to reach expert level. This information was used to develop a structured VR curriculum for the present study. The VR curriculum started with 9 basic skills tasks (9 repetitions each), followed by 4 procedural skills tasks, for example, dissection of Calot's triangle (3 repetitions each). Afterwards, the participants had to perform 6 virtual cases of LC with different patient histories and anatomical conditions (2 repetitions each). Additional repetitions of each task were performed in the remaining training time.
The BL group received a total of 10 hours of box training and 2 hours of E-learning for LC training. The participants received practical basic skills training with a conventional box trainer (Karl Storz GmbH, Tuttlingen, Germany). A structured training curriculum with 9 basic skills exercises was used (shown in Table 1 of the supplemental material). For each task, characteristic errors were identified and described in the curriculum. The participants had to document the task completion time and error rate of each task. They repeated each task several times until a plateau phase in the learning curve was reached for task time and error rate. The procedural skills for LC were taught through E-learning using 2 web-based surgical education platforms with specific content concerning LC. The first 4 didactic chapters of the German website www.webop.de LC module included fundamental knowledge about the relevant anatomy, perioperative management, step-by-step approach to the operation, and management of operative complications.17,18,30 The English website www.WeBSurg.com module, “Laparoscopic cholecystectomy for symptomatic cholelithiasis with or without cholangiography” provided additional information about the procedure. The final component of E-learning was a video recording of a narrated LC on www.WeBSurg.com, “Laparoscopic cholecystectomy: a gold standard case for the dissection of Calot's triangle.”30 A tutor was present during the E-learning session for assistance and to answer questions.
Skill Testing and Recording of Data
The comparison of the investigated training methods was made with a post-test at the end of the last training session. At the end of the training, every participant had to take a multiple choice (MC) test consisting of 16 questions about LC with 1 correct answer each. MC test results were compared between the BL and the VR groups to assess the effect of E-learning. The surgical post-test was carried out on a Pulsating Organ Perfusion (POP) trainer, a mechanical laparoscopic training device for procedural laparoscopic training with explanted cadaveric animal organs that enables lifelike pulsatile perfusion of the organs with artificial blood.31–33 Participants had to perform an LC on an explanted porcine liver with preserved gallbladder and associated structures. Three specially trained raters, who were blinded to the groups of the participants, evaluated the operative performance. To ensure rater blinding, the participants were instructed to not disclose their training status to the raters. Each rater evaluated the same number of participants out of each group to avoid confounding due to rater-specific differences in evaluation.
The participants’ performances were judged using the Objective Structured Assessment of Technical Skills (OSATS), which is a standardized and validated tool for assessing surgical skills.34 OSATS consists of 2 subscores each including 7 different surgical criteria. The general technical skills (GTS) score gives an overall non-tas-specific evaluation of each participants’ performance, whereas the specific technical skills (STS) score evaluates the single steps of a specific procedure. Each GTS item is scored from 1 (worst) to 5 (best), whereas each STS item is scored in increments of 2 points from 2 (worst) to 10 (best). Consequently, a maximum of 35 points can be achieved in the GTS score and a maximum of 70 points in the STS score, yielding a maximum total score of up to 105 points. For the purpose of this study, the OSATS items used by Sarker et al35 were slightly modified. The GTS assessment was supplemented by a further criterion concerning the quantity and quality of required assistance. For the STS assessment, the criteria “access & port insertion” and “extraction of gallbladder” were replaced by “knowledge of procedure” and “quality of product.” The operation time was recorded, and the operation was discontinued when it could not be completed within 80 min; the operation time was then treated as a right-censored observation.
The participants evaluated the laparoscopic simulators and their individual training method by answering a standardized anonymous questionnaire after the post-test. Each training modality was evaluated for several training criteria on 5-point Likert scales (fully agree to fully disagree). Based on this questionnaire, differences in the acceptance of the training modalities were explored.
The OSATS total score was defined as the primary outcome measure. It was compared between the VR and BL groups.
Secondary outcomes were the results of the GTS and STS subscales, operation time, and rate of operations completed within 80 min. Furthermore, the results of the MC test were compared between the VR and BL groups as well as the participants’ opinions concerning the training modalities. OSATS scores of male and female participants were compared to investigate potential sex differences in laparoscopic performance. Possible correlations between participants’ baseline characteristics, questionnaire evaluations, and surgical performances (MC and OSATS results) were explored to identify factors that potentially influence surgical education, and can thus be taken into account for future research.
The randomization process was conducted by using 2 computer-generated lists with variable block sizes, stratified for sex. The randomized assignment to the groups was implemented using closed envelopes. The envelopes were opened only after completion of all baseline assessments. The lists and envelopes were generated and handled by 2 persons who were not directly involved in the training, skills testing, and data collection.
Sample Size Calculation
Based on data reported by Sarker et al,35 a standard deviation of 7.86 for the mean OSATS total score among novices was anticipated (assuming a medium size positive correlation r = 0.5 between GTS and STS). A sample size of 80 participants (40 per group) was calculated to detect a standardized effect size of d = 0.63 at a 2-sided 5% significance level with a power of 80% using a t test for independent groups. This is equal to a mean difference of 5 points between the BL and VR groups, or slightly >1 criterion on the GTS subscale. Group differences of less than this margin of 5 points on the OSATS scale were considered to be irrelevant. A total of 84 participants were included in the study to allow for 5% dropout.
To minimize the possibility of incorrect inputs, 2 independent persons entered the data in parallel. Data were analyzed using R version 220.127.116.11 Descriptive data are given as absolute frequency and as mean ± standard deviation. A normal distribution of OSATS mean scores was assumed,35 so that a 2 × 2 × 3 analysis of variance (Type III sums of squares, alpha 5% two-tailed) was applied to compare the groups for the primary outcome measure (OSATS total score) as well as GTS and STS scores, controlling for the stratification factor Gender and the influence of the 3 raters. The effect of the intervention is reported as difference in marginal means (“least-square means”) with the corresponding 95% confidence interval. The operation time was analyzed using Cox proportional hazards model, with hazard ratios >1 denoting shorter operation times than in the control condition. Group was again used as the main factor of interest and the analysis was stratified by Gender. Pearson correlation coefficient was applied for correlation analyses between participants’ baseline characteristics, questionnaire evaluations, and surgical performances. Differences were defined as statistically significant if the 2-tailed P value related to the group comparison was <0.05. Missing values were not imputed (available cases analysis). This decision was based on the consideration that in a real-life environment, the completion of the training and the assessment would be required for all candidates in surgery.
In total, 84 medical students participated in the randomized study (42 per intervention group) and all of them completed the entire study protocol between September 15th 2012 and February 15th 2013. There were 21 female and 21 male participants in each group. All participants were novices with no laparoscopic experience and limited experience in open surgery (Table 1).
Surgical Test Performance
Virtual Reality Versus Blended Learning
Students in the VR training group and in the BL group showed equal performance of LC in the primary outcome OSATS (49.4 ± 10.5 vs 49.7 ± 12.0, mean difference = 0.3 OSATS units, 95% confidence interval −4.7 to 5.3, P = 0.92). Figure 1 shows box plots with the OSATS total and component scores per group. The VR and BL groups were equal for GTS (16.7 ± 5.0 vs 16.2 ± 4.9, P = 0.63), STS (32.7 ± 6.7 vs 33.5 ± 8.2, P = 0.89), and for the OSATS items (Table 2). No significant differences between the 3 raters were observed for the total and component scores of the OSATS.
The average operation time was significantly shorter for the VR group with 75.8 ± 7.1 min compared with the BL group with 77.6 ± 7.0 min (hazard ratio = 2.39, P = 0.03). The BL group completed 9 operations compared with 19 in the VR group (21% vs 45% completed operations, P = 0.02). In contrast, the score of the knowledge test about LC was significantly better for the BL group than for the VR group (13.3 ± 1.3 vs 11.0 ± 1.7 out of 16 total points; P < 0.001).
The average operation time was significantly shorter for male students with 75.0 ± 1.3 min compared with the female students with 78.4 ± 0.1 min (hazard ratio = 2.46, P = 0.03). The female students completed 9 operations within the allowed 80 min compared with 19 completed operations for the male students (21% vs 45% proportion of completed operations, P = 0.02). Male and female participants did not show significant differences in mean OSATS scores (50.3 ± 11.3 vs 48.8 ± 11.2; P = 0.59). There was no significant difference between males and females for the STS score (32.8 ± 7.1 vs 33.4 ± 7.3; P = 0.79). A trend toward better performance of males in the GTS score was observed (17.5 ± 4.7 vs 15.5 ± 5.0; P = 0.09). This effect was <1 GTS item (4 units) and was due to slightly better scores in the 3 STS items “time and motion,” “flow of operation and forward planning,” and “extent of support needed,”
Evaluation of Training Methods by the Participants
The POP trainer used for the test received the best overall evaluation compared with the VR trainer and box trainer (Supplemental content Figure 1). Both the VR and box trainer were seen as generally helpful for laparoscopic training and helpful for training of hand–eye coordination. Concerning training for instrument coordination, the box trainer was evaluated better than the VR trainer (P = 0.04). However, the VR trainer was found to be more helpful than the box trainer for the more procedural issues: “Exact simulation of intraoperative situation” (P < 0.001), “Helpful for training tissue preparation” (P = 0.005), and “Helpful for training complete operations” (P < 0.001).
Individual Training Method
Each group evaluated their training method subjectively regarding training benefit, effects on training motivation, and changes in surgical interest, whereby no significant difference between the VR and BL groups was found. Both groups rated their individual training method as fun (Supplemental content Figure 2). Male participants felt significantly better prepared for the assistance (P = 0.008) and performance (P < 0.001) of basic laparoscopic operations than female participants. Further items concerning evaluation of training methods did not show any sex differences.
There was a significant correlation between good OSATS results and seeing the course as good preparation for assistance and performance of laparoscopic surgery, respectively, as relevant for future choice of profession (Table 3). A positive relation between surgical interest and performance was observed for the GTS score but not for total OSATS and STS scores. There was a significant correlation between prior experience in open surgery and the GTS score, but not with the STS and total OSATS scores. Concerning the other questionnaire evaluation issues, no correlations with OSATS results were found. Concerning the MC test results, there was a significant correlation (r = 0.24; P = 0.03) between the STS item “gall bladder retraction and exposure of Calot triangle” and the MC question on the safe exposure and use of cautery near critical structures within Calot triangle.
In the present study, the students in the VR group performed the operation faster and were more often able to complete the operation within the maximum allowed time of 80 minutes in comparison to the BL group. However, the BL group had better theoretical background knowledge about the operation as tested in a MC test. The VR and BL groups showed equal operative performance in the OSATS score, which was the primary outcome of this study. Male students were more often able to complete the operation within the maximum allowed time of 80 min, whereas the operative performance of LC in the OSATS score was similar for male and female trainees. The time limit of 80 min was set for feasibility reasons of the study. In preliminary tests, the time limit had proven sufficient for the students to succeed in performing the main steps of Calot triangle exposure, clipping and dividing of cystic duct and artery. The students who did not finish the LC in the time limit were all in the phase of dissecting the gall bladder from the liver bed. Due to the anatomy of the porcine livers that differs slightly from humans in that the gall bladder can be very encased in the liver bed, it would often have taken more time to finish the LC. However, the time was always sufficient to adequately rate the performance. The time differences between the groups and sexes were of statistical significance despite rather small differences. However, the rates of finishing the operation in the time limit were of relevant differences between the groups and sexes.
BL had not been compared with VR training in a randomized trial prior to the present study. Quite a number of studies have been performed comparing VR training and box training without showing clear superiority of one method over the other.37 Some authors promote VR trainers because of the possibility to simulate complete operations repeatedly.28 Box trainers require the insertion of cadaveric organs or artificial organ models that need to be replaced after each training session. Box training, however, is far cheaper than VR training and is more easily accessible. Prices for an equipped box trainer station range from 200 to 7600 US-$, whereas common VR trainers cost 45,000 to 120,000 US-$.16,25,26,38–40 E-learning is also easily accessible and cost-effective.41 The concept of BL uses box training for basic skills supplemented by E-learning to provide knowledge and cognitive training for the operation.42 With the same training outcome, BL thus provides higher training efficiency given the cost advantages over VR training.43
The BL group scored significantly higher in the knowledge test concerning LC in the present study. A correlation was found between theoretical knowledge of specific technical aspects of the operation and the corresponding parts of the specific technical score of the OSATS from the performed LCs in the present study. This shows the important role of cognitive training as suggested in previous studies.18,41 It also shows that the theoretical part of the operations might not be addressed sufficiently in the VR trainer. Hence, it seems important to incorporate cognitive training, for example, adding E-learning to multimodality training curriculums for laparoscopic surgery.12
Sex differences have been previously addressed in surgical education research studies.44 This has drawn particular attention to laparoscopic surgery education research, as there are suspected sex differences in the hereby-important 3-dimensional orientation and psychomotor skills.45,46 In the present study, males performed faster than females, with higher rates of completing the operation within the allowed 80 min, but without differences in the quality of the operation, as reflected by the OSATS score. This is in line with previous laparoscopy training studies showing men performing faster with equal quality of performance.47
Interestingly, in the present study, there was a positive correlation between the OSATS scores and the students’ experienced benefits from the course in terms of being able to participate in laparoscopic surgery in the OR. There was also a positive correlation between the GTS score and the students’ interests in surgery, but not with the total OSATS and STS scores. There was a correlation between experience in open surgery and the GTS. This presumably reflects a better general understanding of surgical principles by the trainees who participated in open surgeries before the present study and by those who are interested in surgery. These findings emphasize the need for surgical training courses for students, as there seem to be relations between positive experiences in surgical training and the expected ability to participate in surgery.
In the evaluation of the roles of the different training modalities, the POP trainer received the best overall evaluation by the trainees. The box trainer was seen as better for the training of basic skills, such as instrument coordination, whereas the VR trainer received better ratings for the training of procedural skills and operations. Multimodality training courses for laparoscopic surgery should therefore combine the use of these training modalities for the advantages of each, thus maximizing the benefit and efficiency of training. The present study was conducted with laparoscopy-naïve medical students in a controlled training laboratory setting. Therefore, the results cannot directly be transferred to more experienced trainees. The absence of baseline laparoscopic experience of the students, however, guaranteed good comparability of the results and minimized bias due to different baseline experiences, as is often the case in studies with surgical residents. Furthermore, we believe that the technical abilities of medical students in their clinical years of study resemble those of surgeons at the very beginning of residency. Important to note, that participation in this course was not an obligation and therefore the students in this course were all rather interested in surgery.
The operative performance tests in the present study were done with explanted cadaveric porcine organs that come close but are not identical to the anatomy found in human patients in the OR. However, the POP trainer with the porcine organs received very good evaluations by the trainees and provided realistic tissue feeling in contrast to the box and VR trainer. Training on live animals and human cadavers was not considered for this study with medical students due to financial and ethical reasons. OSATS was chosen as the measurement method and primary outcome for performance of LC on the POP trainer in the present study. OSATS has proven construct validity in previous studies.34,48,49 The raters in the present study were blinded to the training status of trainees and there was no influence of the 3 raters on the OSATS results, thus indicating no rater bias.
In conclusion, both VR training and BL were effective for the training of LC. Using the validated OSATS score as the primary outcome, the groups and sexes showed similar operative performance of LC. The VR group and males performed the operation faster and were more likely to complete the operation within the maximum given time of 80 min. The VR training was rated better for procedural training and the box trainer for basic skills training. The BL group had more knowledge about the operation and BL is the more cost-effective option. The POP trainer received the best overall rating. The superiority of any kind of training over no training for laparoscopic novices is no longer doubted in the literature.6,42,50 The best training options for different training levels, however, lack a clear consensus. In line with the results of the present study, both VR and BL can be applied for training novices the basics of LC with different advantages. Multimodality training programs for different trainee levels should combine the available modalities to their advantage.
The authors thank Jonathan D. Hendrie for proofreading of the manuscript.
1. Greene FL, Kercher KW, Nelson H, et al. Minimal access cancer management. CA Cancer J Clin
2. Harrysson IJ, Cook J, Sirimanna P, et al. Systematic review of learning curves for minimally invasive abdominal surgery: a review of the methodology of data collection, depiction of outcomes, and statistical analysis. Ann Surg
3. van Dongen KW, Ahlberg G, Bonavina L, et al. European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills. Surg Endosc
4. Schijven MP, Jakimowicz JJ, Broeders IA, et al. The Eindhoven laparoscopic cholecystectomy training course—improving operating room performance using virtual reality training: results from the first E.A.E.S accredited virtual reality trainings curriculum. Surg Endosc
5. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg
6. Aggarwal R, Ward J, Balasundaram I, et al. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg
7. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg
8. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg
9. van Velthoven RF, Hoffmann P. Methods for laparoscopic training using animal models. Curr Urol Rep
10. Undre S, Darzi A. Laparoscopy simulators. J Endourol
11. Nagendran M, Toon CD, Davidson BR, et al. Laparoscopic surgical box model training for surgical trainees with no prior laparoscopic experience. Cochrane Database Syst Rev
12. Palter VN, Grantcharov TP. Development and validation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg
13. Palter VN, Orzech N, Reznick RK, et al. Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg
14. Strandbygaard J, Bjerrum F, Maagaard M, et al. A structured four-step curriculum in basic laparoscopy: development and validation. Acta Obstet Gynecol Scand
15. Nickel F, Bintintan VV, Gehrig T, et al. Virtual reality does not meet expectations in a pilot study on multimodal laparoscopic surgery training. World J Surg
16. Orzech N, Palter VN, Reznick RK, et al. A comparison of 2 ex vivo training curricula for advanced laparoscopic skills: a randomized controlled trial. Ann Surg
17. Pape-Koehler C, Chmelik C, Aslund AM, et al. An interactive and multimedia-based manual of surgical procedures: Webop—an approach to improve surgical education. Zentralbl Chir
18. Pape-Koehler C, Immenroth M, Sauerland S, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc
19. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg
20. Evgeniou E, Loizou P. The theoretical base of e-learning and its role in surgical education. J Surg Educ
21. Funke K, Bonrath E, Mardin WA, et al. Blended learning in surgery using the Inmedea Simulator. Langenbecks Arch Surg
22. Kadmon M, Busemann A, Euteneier A, et al. Modular postgraduate training in surgery—a national concept with future. Zentralbl Chir
23. Rieger UM, Pierer K, Farhadi J, et al. Effective acquisition of basic surgical techniques through blended learning. Chirurg
24. Gurusamy KS, Nagendran M, Toon CD, et al. Laparoscopic surgical box model training for surgical trainees with limited prior laparoscopic experience. Cochrane Database Syst Rev
25. Nagendran M, Gurusamy KS, Aggarwal R, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev
26. Gurusamy KS, Aggarwal R, Palanivelu L, et al. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev
27. Sturm LP, Windsor JA, Cosman PH, et al. A systematic review of skills transfer after surgical simulation training. Ann Surg
28. Larsen CR, Oestergaard J, Ottesen BS, et al. The efficacy of virtual reality simulation training in laparoscopy: a systematic review of randomized trials. Acta Obstet Gynecol Scand
29. Aggarwal R, Crochet P, Dias A, et al. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg
30. Mutter D, Vix M, Dallemagne B, et al. WeBSurg: an innovative educational Web site in minimally invasive surgery—principles and results. Surg Innov
31. Clerici T, Lange J, Zerz A, et al. Educational opportunities in minimally invasive surgery. Wien Klin Wochenschr
32. Szinicz G, Beller S, Bodner W, et al. Simulated operations by pulsatile organ-perfusion in minimally invasive surgery. Surg Laparosc Endosc
33. Szinicz G, Beller S, Zerz A. Role of the pulsatile organ perfusion surgical simulator in surgery education. Langenbecks Arch Chir Suppl Kongressbd
34. van Hove PD, Tuijthof GJ, Verdaasdonk EG, et al. Objective assessment of technical surgical skills. Br J Surg
35. Sarker SK, Chang A, Vincent C, et al. Development of assessing generic and specific technical skills in laparoscopic surgery. Am J Surg
36. Team RCR. A language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2014.
37. Diesen DL, Erhunmwunsee L, Bennett KM, et al. Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ
38. Debes AJ, Aggarwal R, Balasundaram I, et al. A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. Am J Surg
39. Aggarwal R, Moorthy K, Darzi A. Laparoscopic skills training and assessment. Br J Surg
40. Aggarwal R, Darzi A. Training in laparoscopy—which model to use? Indian J Gastroenterol
41. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ
42. Immenroth M, Bürger T, Brenner J, Nagelschmidt M, Eberspächer H, Troidl H. Mental training in surgical education: a randomized controlled trial. Ann Surg
43. Clevin L, Grantcharov TP. Does box model training improve surgical dexterity and economy of movement during virtual reality laparoscopy? A randomised trial. Acta Obstet Gynecol Scand
44. Baxter N, Cohen R, McLeod R. The impact of gender on the choice of surgery as a career. Am J Surg
45. Madan AK, Harper JL, Frantzides CT, et al. Nonsurgical skills do not predict baseline scores in inanimate box or virtual-reality trainers. Surg Endosc
46. Thorson CM, Kelly JP, Forse RA, et al. Can we continue to ignore gender differences in performance on simulation trainers? J Laparoendosc Adv Surg Tech A
47. Grantcharov TP, Bardram L, Funch-Jensen P, et al. Impact of hand dominance, gender, and experience with computer games on performance in virtual reality laparoscopy. Surg Endosc
48. Hiemstra E, Kolkman W, Wolterbeek R, et al. Value of an objective assessment tool in the operating room. Can J Surg
49. Kundhal PS, Grantcharov TP. Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc
50. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg