Journal Logo

Original Scientific Articles

Framework for Training in Minimally Invasive Pancreatic Surgery: An International Delphi Consensus Study

Korrel, Maarten MD,; Lof, Sanne MD, PhD,; Alseidi, Adnan A MD, PhD, FACS; Asbun, Horacio J MD, PhD, FACS; Boggi, Ugo MD, PhD; Hogg, Melissa E MD, PhD, FACS; Jang, Jin-young MD, PhD; Nakamura, Masafumi MD, PhD, FACS; Besselink, Marc G MD, MSC, PhD; Abu Hilal, Mohammad MD, PhD, FACS,;  for the International Consortium on Minimally Invasive Pancreatic Surgery (I-MIPS)

Collaborators

Björnsson, Bergthor; Conrad, Claudius; Dokmak, Safi; Edwin, Bjorn; Esposito, Alessandro; Fuks, David; Ghanem, Maher; Hackert, Thilo; Han, Ho-Seong; He, Jin; Jeyarajah, Rohan; Keck, Tobias; Kendrick, Michael; Khatkov, Igor; Kim, Song Cheol; Kooby, David; Machado, Marcel; Martinie, John; Molenaar, Quintus; Mou, Yiping; Miyasaka, Yoshihiro; Palanivelu, Chinnusamy; Pessaux, Patrick; Rosso, Edoardo; Salti, George; Senthilnathan, Palanisamy; Shrikhande, Shailesh; Talamonti, Mark; Wakabayashi, Go; Zeh, Herbert; Zureikat, Amer

Author Information
Journal of the American College of Surgeons: September 2022 - Volume 235 - Issue 3 - p 383-390
doi: 10.1097/XCS.0000000000000278
  • Free
  • SDC
  • Infographic

The first laparoscopic pancreatic resections were described in 1994, the first robotic distal pancreatectomy (DP) in 2001, and robotic pancreatoduodenectomy (PD) in 2003.1-4 Since then, the worldwide implementation of minimally invasive pancreatic surgery (MIPS) has been relatively slow compared with the development of minimally invasive surgery for other abdominal procedures.5,6

In a worldwide survey, the majority of responding pancreatic surgeons stated that specific standardized training in MIPS was lacking.7 Moreover, 75% of European pancreatic surgeons felt that they would benefit from specific standardized training in MIPS.8 In the modern era where both patient expectation and surgical standards are increasing, the concept of “See one, do one, teach one” is no longer acceptable and sufficient.6 Patients expect an adequately trained surgeon to prevent increased risks during the relatively long “learning curve” phase of complex procedures such as MIPS.6

Training in MIPS is increasing worldwide, provided mainly by proctoring experts in MIPS or through dedicated MIPS training sessions organized by associations such as the European Consortium on Minimally Invasive Surgery (E-MIPS, www.e-mips.com) and MIPS fellowships. To our knowledge, only a few MIPS training programs, 3 laparoscopic9-11 and 4 robotic,12-14 have been reported. Of these, the 3 Dutch Longitudinal Assessment and rEalization of minimaLly invAsive Pancreatic Surgery (LAELAPS) programs are the only ones reporting on a nationwide approach to training in MIPS.9,10,15 These training programs were associated with an increased use of MIPS with satisfactory outcomes but without a negative impact of the learning curve.9-15 To expand MIPS safely and reduce the learning curve, standardized training methods are needed as called for by the Miami International Guidelines on Minimally Invasive Pancreatic Surgery.16 However, on this specific point, a clear international consensus on the framework for such training methods is lacking.

This modified Delphi study builds on the evidence gathered in the Miami guidelines16 and evaluated items of importance for training in MIPS. The goal was to achieve an international consensus on a framework for training programs in MIPS. This framework is intended to improve training, proctoring, and safe worldwide dissemination of MIPS.

METHODS

Study design and recruitment of experts

This study was performed according to the American Association for Public Opinion Research (AAPOR) reporting guidelines.17 This Delphi consensus study was performed as an international collaborative project by the International Consortium on Minimally Invasive Pancreatic Surgery (I-MIPS, www.i-mips.com). Experts were invited based on their participation in the Miami International Guidelines on Minimally Invasive Pancreatic Resection, their publications on the subject of MIPS, and their involvement as founding members of the I-MIPS consortium. This group of participants is further referred to as the “experts” (Appendix). The study coordinators (MK and SL) in combination with co-investigators (AA, HA, UB, MH, JJ, MN) and principal investigators (MGB and MAH) are further referred to as the “study committee.”

Table 1. - Characteristics of the 31 Participating Experts
Characteristic Data
Academic center, n (%) 28 (90)
Robotic system available, n (%) 28 (90)
Years of experience, y, median (IQR)
 General laparoscopic surgery 20 (15–25)
 General robotic surgery 5 (2–7.5)
 Minimally invasive distal pancreatectomy 10 (9.5–15)
 Minimally invasive pancreatoduodenectomy 8.5 (3.5–10.5)
Annual case volume, median (IQR)
 Minimally invasive distal pancreatectomy 20 (13.5–31.5)
 Minimally invasive pancreatoduodenectomy 20 (4–40)
Total number of cases per expert, median (IQR)
 Minimally invasive distal pancreatectomy 200 (100–275)
 Minimally invasive pancreatoduodenectomy 70 (27.5–200)
Experience as proctor in MIPS, n (%) 17 (55)
IQR, interquartile range; MIPS, minimally invasive pancreatic surgery.

Considering the international design of this study, a modified Delphi consensus study method was chosen to allow for eliminating physical meetings during the ongoing COVID pandemic. Several surgical curricula have already been developed using the Delphi study method.18-20 The Delphi study method is a process wherein each participant’s responses are systematically collected during multiple survey rounds. Based on the answers returned by the participants, questions are either deleted or altered for reassessment and voting in a next survey round until group consensus is ultimately achieved.21,22

Study process

Step 1: Principal investigator meeting

In November 2019, the principal investigators and study coordinators developed the study set-up and consensus process. Experts in the field were identified, and an invitation letter was sent out 4 weeks before start of the first the Delphi round. The study committee agreed on 3 essential domains of training in MIPS: (1) training framework items, (2) items for the eligibility of centers or surgeons to receive training in MIPS, and (3) items for the eligibility of surgeons to proctor during training in MIPS.

Step 2: Delphi consensus process

The first Delphi round was created by the study coordinators and agreed by the study committee. This round included questions on demographics and experience in MIPS and training and 23 questions regarding the 3 proposed domains. The latter consisted of free-text questions and 5-point scale Likert questions (from 1 [not important] to 5 [very important]). This survey was sent to all the participating experts using web-based survey software (SurveyMonkey, Palo Alto, CA). Reminders were sent weekly, and the first Delphi round was closed 4 weeks after the initial start of this round. The replies of the experts were analyzed by the study coordinators and used for the second Delphi round. Items from the first round which were rated “important” or “very important” by at least 80% of the experts were altered and entered in the second Delphi round. The study committee created statements with further details. Items in this second Delphi round were scored on a different 5-point scale Likert (from 1 [fully disagree] to 5 [fully agree]). Again, reminders were sent weekly until closure 4 weeks after the initial invitation. Both survey rounds can be found as Supplemental Digital Content.

Step 3: Compiling the framework for training in MIPS

Based on the results from both Delphi rounds, a proposal for a framework for training in MIPS over the 3 domains was created by the study coordinators. The final framework was sent to the study committee for their approval.

Statistical analyses

Data were analyzed using IBM SPSS Statistics for Windows version 25.0 (IBM Corp, Armonk NY). Internal consistency of the experts’ replies per domain was calculated by the Cronbach’s α test. A Cronbach’s α of 0.70 or greater was predefined as the cut-off value to achieve consensus. Each item that achieved 80% or greater of expert votes was considered as fundamental for a training program in MIPS.

RESULTS

Experts and demographics

A total of 31 experts in MIPS were identified and invited for the first Delphi round. Both Delphi rounds had a 100% response rate. Expert characteristics are shown in Table 1. The experts included surgeons from 31 centers in 13 countries across 4 continents (see the Appendix) with a median experience in laparoscopic and robotic pancreatic surgery of 20 years (interquartile range 15 to 25) and 5 (interquartile range 2 to 7.5) years, respectively. Most experts (90%) were employed at an academic center and 90% had access to a robotic system. The median personal experience in minimally invasive DP (MIDP) was 200 (interquartile range 100 to 275) and for minimally invasive pancreatoduodenectomy (MIPD) 70 (interquartile range 27.5 to 200) procedures. Approximately half (55%) of the included experts had experience as proctor in MIPS. Most experts (82%) stated that they were aware of MIPS training programs. The University of Pittsburgh Medical Center (UPMC) program in the US and the 3 LAELAPS programs in The Netherlands were most frequently mentioned.9,10,12

Domain 1: Training framework

Consensus among experts was achieved on the items needed for a framework of a training program for MIPS (Table 2 and Supplemental Figure 1, https://links.lww.com/JACS/A101; Cronbach’s α = 0.72). Experts agreed that international or national societies should be in charge or involved in organizing the MIPS training programs, whereas funding and support for training should be organized by individual hospitals that receive training. It was agreed that a dry-run on biotissue and cadaver/simulation workshops should be included in a training program. Training should always be performed in the physical presence of a proctor.

Table 2. - Items Included for the Framework and Eligibility of Centers and Proctors for MIPS Training
Domain Item
Round 1 Round 2
Framework Dry-run during cadaver workshop and biotissue training
Start with MIDP and early-stage malignant tumors
Societies in charge, either alone or in combination with governments
Exclude vascular & multivisceral involvement, borderline resectable tumors
Take the learning curve into account
First cases performed with physical presence of a proctor
Perform ≥5 MIDP and biotissue training before MIPD training
Start with MIDP, either laparoscopic or robotic
Design and program by international societies/study groups
Funding by individual hospitals
Vascular and multivisceral resections after ≥50–100 procedures
Difficulty scores should be used, but are to be developed
Completed when surgeon and proctor agree, checklist to be developed
Center eligibility Availability to 24/7 infrastructure
Availability to previous pancreatic surgeries and MIS procedures
Follow and document outcomes during the training process
Willing to receive external audit for validation of organization and results
Follow outcomes of MIPS procedures in dedicated (inter)national registries
Formalized support from their institutional Board of Directors
24/7 ICU, interventional and emergency departments
Availability to a robotic system
Surgeon eligibility Minimum annual volume of PD per surgeon
Minimum annual volume of DP per surgeon
Passed learning curve for open PD and DP
Experience in advanced minimally invasive organ resection
≥50 PDs to achieve ≥20 MIPDs annually
≥10 DPs annually
Experience in DPs, ≥60 open PDs
Proctor eligibility At least a certain number of MIPS resections performed ≥50 MIPDs and 50 MIDPs performed throughout career
DP, distal pancreatectomy; ICU, intensive care unit; MIDP, minimally invasive distal pancreatectomy; MIPD, minimally invasive pancreatoduodenectomy; MIPS, minimally invasive pancreatic surgery; MIS, minimally invasive surgery; PD, pancreatoduodenectomy.

Different procedures and different tumors were considered eligible for training in MIPS. Experts agreed that a training program should start with either laparoscopic or robotic distal pancreatectomies in patients with early-stage tumors (i.e. stage T1 to T2). The use of difficulty scores23,24 for MIDP and MIPD is encouraged when selecting patients for MIPS. Tumors requiring vascular resections or multivisceral resections, and borderline resectable tumors, should be excluded from the initial MIPS training. These advanced procedures should only be performed once a surgeon has performed at least 50 to 100 MIDP and 50 to 100 MIPD procedures.

Items regarding quality control of a training program in MIPS focused on documentation of outcomes. It was found to be fundamental that centers eligible for training should document outcomes of their MIPS procedures in national or international registries, including MIPS. Furthermore, experts agreed that centers participating in a formal MIPS training program should be willing to receive an external audit to validate results and have formalized support from their institutional Board of Directors.

Domain 2: Eligibility of centers and surgeons for training in MIPS

Consensus among experts was achieved regarding the eligibility criteria for centers and surgeons to receive training in MIPS (Table 2 and Supplemental Figure 2, https://links.lww.com/JACS/A102 Cronbach’s α = 0.87).

Centers receiving training in MIPS should have sufficient infrastructure and equipment for training (ie 24/7 availability for the following services: intensive care unit, emergency department, interventional radiology, and endoscopy). It was found to be fundamental that centers should have a robotic system in place while starting with training to prevent long delays between training and performing the actual first robotic MIPS procedures.

Eligibility criteria for centers and surgeons to receive training in MIPS focused on total and annual case volumes. It was agreed that trainees should have passed the learning curve for both open DP and open PD (60 or more pancreatoduodenectomies) and should have experience in advanced minimally invasive organ resections (ie beyond cholecystectomy, appendectomy, hernia repair). Centers receiving training in MIPS should perform at least 50 PDs and 10 DPs annually in total to achieve the minimum annual volume of 20 MIPDs, as described by the Miami guidelines, when training has been completed.

Domain 3: Eligibility of proctors for training in MIPS

Experts agreed that surgeons eligible as proctor for training in MIPS should have performed at least 50 laparoscopic or robotic DPs and at least 50 laparoscopic or robotic PDs before proctoring MIPS cases (Table 2 and Supplemental Figure 3, https://links.lww.com/JACS/A103; Cronbach’s α = 0.89).

A summary of formulated recommendations is shown in Figure 1.

F1
Figure 1.:
Overview of recommendations. DP, distal pancreatectomy; MIDP, minimally invasive distal pancreatectomy; MIPD, minimally invasive pancreatoduodenectomy; MIPS, minimally invasive pancreatic surgery; ODP, open distal pancreatectomy; OPD, open pancreatoduodenectomy; PD, pancreatoduodenectomy.

DISCUSSION

This international modified Delphi consensus study among 31 experts in MIPS from 13 countries across 4 continents reports on a consensus-based training framework in MIPS. Consensus, defined as at least 80% agreement, was reached during 2 rounds across 3 domains: (1) the framework, (2) eligibility for centers and surgeons to be proctored, and (3) eligibility for surgeons as proctors in MIPS training.

Structured training programs in MIPS remain scarce, as only 3 center-specific11,12,14 and 3 nationwide9,10,13 programs for laparoscopic DP, laparoscopic PD, and robotic PD have been published. The present study recommends that MIPS training programs should be facilitated or endorsed by (inter)national societies and funded by the individual hospitals receiving the training program. It was agreed that following dry-lab practices, training should start with MIDP procedures in early-stage tumors.25 An important element of the previously published training programs was the inclusion of biotissue-based suturing training.12,13 Considering the different anastomoses required during MIPD, advanced training in the specific anastomoses could minimize the risk of postoperative anastomotic leakage, which remains the most common complication after pancreatic surgery.26 Tumors involving vascular and multivisceral structures should be excluded as training cases, because insufficient data on outcomes after vascular resections are available as stated by the Miami guidelines.16,25 It was suggested to perform these procedures only when surgeons have performed a minimum of 50 to 100 MIDPs and 50 to 100 MIPDs. It was agreed that the training was completed when both the surgeon and proctor concur that sufficient proficiency has been gained during the proctoring sessions. However, after completing the training, surgical outcomes should be registered.

Registration of outcomes during and after completion of training is fundamental for monitoring the outcomes of MIPS. Previously published training programs have reported their outcomes. For instance, the nationwide LAELAPS-1 program reported a 7-fold increase in MIDP use after the start of the training program.9 Similarly, the LAELAPS-3 program reported a 25% increase in the use of robotic MIPD.15 Furthermore, the UPMC training program reported improvement on overall skill scores after the completion of the simulation exercises among all trainees.12 The present study strongly recommends documenting outcomes of all MIPS procedures performed during and after the training program, which should be registered in national (eg NSQIP and Dutch Pancreatic Cancer Audit) and international registries (eg E-MIPS and I-MIPS). Moreover, experts in the present study agreed that centers participating in such registries should be willing to receive external auditing for validation of team organization and results. One example of such a registry is the ongoing E-MIPS registry (www.e-mips.com/registry), which was developed with considerable support from European pancreatic surgeons and is endorsed by the European-African Hepato-Pancreato-Biliary Association (E-AHPBA).27

Since the introduction of MIPS, proficiency-based learning curves and annual case volumes are a recurrent topic of debate.1,2 Adequate and sufficient training is crucial for the safe implementation of MIPS into clinical practice and subsequent patient outcomes.16 Considering the highly complex nature of MIPS, it is therefore recommended to only start training when a surgeon has passed the learning curve for open DP and PD and has experience in advanced minimally invasive abdominal procedures. The present study recommends that centers receiving training in MIPS should perform at least 50 PDs annually in total to achieve the minimum annual volume of 20 MIPDs after training. Both morbidity and mortality seem to be worse in centers performing fewer than the minimum annual volume of 20 MIPDs per center, as also recommended by the Miami guidelines.16,28 Such case volumes would also be required to achieve the currently suggested feasibility learning curve of 10 to 40 procedures for MIDP and 10 to 50 procedures for MIPD.16 A much lower volume advice was given for distal pancreatectomy (and MIDP) given the less complex nature of this procedure; moreover, the Miami guidelines provide no specific volume advice for these procedures.16

Important training efforts in MIPS have been made by constructing multiple institutional and nationwide training programs and MIPS training courses organized by experienced MIPS surgeons.9-15 Recommendations of this consensus study were based on the Miami guidelines16 and could form the basis of a structured training program for MIPS on an international level. Currently, the only international training program known to us is the European LEARNBOT program for robotic MIPD which is organized by E-MIPS and endorsed by the E-AHPBA (http://www.e-mips.com/learnbot). However, the fundamental items in this study remain a subject of debate, and whether these are integrated in such programs is up to the organizers’ discretion. Further optimization of the training program could be achieved by either adding or removing items from its curriculum. Future research should focus on the inclusion of these items in such programs, and after the construction of a definitive, comprehensive training program, results should be evaluated and reported.

The results of the present study should be interpreted in light of several limitations. First, some experts in the field of MIPS might not have been identified by our review process, which could have influenced the results.29 However, it was agreed that the current inclusion process would harvest most experts in the field and would therefore give a good representation of the full range of MIPS experts. This was confirmed by the inclusion of 31 experts from 13 countries across 4 continents. Second, the range of MIPD procedures included 0 to 60 procedures annually, indicating that some surgeons were not performing MIPD at the time of this survey. This is explained by the inclusion of experts based on the Miami guidelines, which includes some experts in open pancreatic surgery and some with only extensive experience in MIDP.16 Third, some of the experts encountered some delay in response owing to the COVID-19 pandemic, which was, however, considered to have had low impact on outcomes and recommendations of this study. Fourth, opinions and recommendations were formulated by expert surgeons only, without the inclusion of other multidisciplinary team members (eg anesthesiologists), less experienced surgeons, and patients. Although the multidisciplinary approach is crucial for outcomes after MIPS, the present study intended to identify fundamental items for surgical training in MIPS rather than training of the entire multidisciplinary team. Future studies could focus on this topic of broader involvement of stakeholders. Fifth, geographical differences and different health care systems might have influenced the choice whether items were felt to be fundamental or not. However, all items were scored rather consistently between experts in both rounds and would therefore appear to have had little impact on outcomes of the present study. Strengths of the present study are represented by the 100% response rates in both study rounds among 31 experts from 13 countries, which enhances the generalizability of the findings.30

CONCLUSIONS

In conclusion, this Delphi consensus study is the first to determine fundamental items for a training program in MIPS by consensus among international experts. These items may serve as guidelines for constructing a comprehensive training program and may eventually help increase the worldwide implementation of MIPS, leading to a decreased proficiency gain curve and thereby reducing complications after MIPS.

Author Contributions

Study conception and design: Korrel, Lof, Besselink, Abu Hilal

Acquisition of data: Alseidi, Asbun, Boggi, Hogg, Jang, Nakamura, Besselink, Abu Hilal

Analysis and interpretation of data: Korrel, Besselink, Abu Hilal

Drafting of manuscript: Korrel, Lof, Besselink, Abu Hilal

Critical revision: Alseidi, Asbun, Boggi, Hogg, Jang, Nakamura

    APPENDIX

    Members of the International Consortium on Minimally Invasive Pancreatic Surgery (I-MIPS): Bergthor Björnsson (University Hospital Linköping, Sweden), Claudius Conrad (St Elizabeth’s Medical Center, Boston, MA), Safi Dokmak (Beaujon Hospital, Clichy, France), Bjorn Edwin (Oslo University Hospital, Oslo, Norway), Alessandro Esposito (University Hospital Verona, Verona, Italy), David Fuks (Institut Mutualiste Montsouris, University of Paris Descartes, Paris, France), Maher Ghanem (Central Michigan University of Medicine, Mt Pleasant, MI), Thilo Hackert (University Hospital Heidelberg, Heidelberg, Germany), Ho-Seong Han (Seoul National University Hospital, Seoul, South Korea), Jin He (Johns Hopkins University School of Medicine, Baltimore, MD), Rohan Jeyarajah (Methodist Richardson Medical Center, Richardson, TX), Tobias Keck (University Hospital Lübeck, Lübeck, Germany), Michael Kendrick (Mayo Clinic, Rochester, NY), Igor Khatkov (Moscow Clinical Scientific Center, Moscow, Russia), Song Cheol Kim (Asan Medical Center, Seoul, South Korea), David Kooby (Emory University, Atlanta, GA), Marcel Machado (University of Sao Paulo, Sao Paulo, Brazil), John Martinie (Carolina’s Health Care Hospital, Charlotte, NC), Quintus Molenaar (University Hospital Utrecht, Utrecht, The Netherlands), Yiping Mou (People’s Hospital of Hangzhou Medical College, Zhenjiang, China), Yoshihiro Miyasaka (Fukuoka University Chikushi Hospital, Chikushino, Japan), Chinnusamy Palanivelu (GEM Hospital and Research Center, Coimbatore, India), Patrick Pessaux (University Hospital Strasbourg, Strasbourg, France), Edoardo Rosso (Pôle Santé Sud, Le Mans, France), George Salti (University of Illinois Hospital, Chicago, IL), Palanisamy Senthilnathan (GEM Hospital and Research Center, Coimbatore, India), Shailesh Shrikhande (Tata Memorial Center, Mumbai, India), Mark Talamonti (NorthShore University Health System, Evanston, IN), Go Wakabayashi (Ageo Central General Hospital, Saitama, Japan), Herbert Zeh (University of Pittsburgh School of Medicine, Pittsburgh, PA), and Amer Zureikat (University of Pittsburgh School of Medicine, Pittsburgh, PA).

    REFERENCES

    1. Cuschieri A, Jakimowicz JJ, van Spreeuwel J. Laparoscopic distal 70% pancreatectomy and splenectomy for chronic pancreatitis. Ann Surg 1996;223:280–285.
    2. Gagner M, Pomp A. Laparoscopic pylorus-preserving pancreatoduodenectomy. Surg Endosc 1994;8:408–410.
    3. Giulianotti PC, Coratti A, Angelini M, et al. Robotics in general surgery: Personal experience in a large community hospital. Arch Surg 2003;138:777–784.
    4. Melvin WS, Needleman BJ, Krause KR, Ellison EC. Robotic resection of pancreatic neuroendocrine tumor. J Laparoendosc Adv Surg Tech A 2003;13:33–36.
    5. Sood A, Meyer CP, Abdollah F, et al. Minimally invasive surgery and its impact on 30-day postoperative complications, unplanned readmissions and mortality. Br J Surg 2017;104:1372–1381.
    6. Hogg ME, Besselink MG, Clavien PA, et al.; Minimally Invasive Pancreatic Resection Organizing Committee. Training in minimally invasive pancreatic resections: A paradigm shift away from “See one, Do one, Teach one.” HPB (Oxford) 2017;19:234–245.
    7. van Hilst J, de Rooij T, Abu Hilal M, et al. Worldwide survey on opinions and use of minimally invasive pancreatic resection. HPB (Oxford) 2017;19:190–204.
    8. de Rooij T, Besselink MG, Shamali A, et al.; DIPLOMA Trial Group. Pan-European survey on the implementation of minimally invasive pancreatic surgery with emphasis on cancer. HPB (Oxford) 2016;18:170–176.
    9. de Rooij T, van Hilst J, Boerma D, et al.; Dutch Pancreatic Cancer Group. Impact of a nationwide training program in minimally invasive distal pancreatectomy (LAELAPS). Ann Surg 2016;264:754–762.
    10. de Rooij T, van Hilst J, Topal B, et al.; Dutch Pancreatic Cancer Group. Outcomes of a multicenter training program in laparoscopic pancreatoduodenectomy (LAELAPS-2). Ann Surg 2019;269:344–350.
    11. Nakamura Y, Matsushita A, Katsuno A, et al. Laparoscopic distal pancreatectomy: Educating surgeons about advanced laparoscopic surgery. Asian J Endosc Surg 2014;7:295–300.
    12. Hogg ME, Tam V, Zenati M, et al. Mastery-based virtual reality robotic simulation curriculum: The first step toward operative robotic proficiency. J Surg Educ 2017;74:477–485.
    13. Nota CL, Zwart MJ, Fong Y, et al.; Dutch Pancreatic Cancer Group. Developing a robotic pancreas program: The Dutch experience. J Vis Surg 2017;3:106.
    14. Klompmaker S, van der Vliet WJ, Thoolen SJ, et al. Procedure-specific training for robot-assisted distal pancreatectomy. Ann Surg 2021;274:e18–e27.
    15. Zwart MJW, Nota CLM, de Rooij T, et al. Outcomes of a multicenter training program in robotic pancreatoduodenectomy (LAELAPS-3). Ann Surg 2021.
    16. Asbun HJ, Moekotte AL, Vissers FL, et al. The Miami International Evidence-Based Guidelines on Minimally Invasive Pancreas Resection. Ann Surg 2020;271:1–14.
    17. Pitt SC, Schwartz TA, Chu D. AAPOR reporting guidelines for survey studies. JAMA Surg 2021;156:785–786.
    18. Palter VN, Graafland M, Schijven MP, et al. Designing a proficiency-based, content validated virtual reality curriculum for laparoscopic colorectal surgery: A Delphi approach. Surgery 2012;151:391–397.
    19. Singh SS, Marcoux V, Cheung V, et al. Core competencies for gynecologic endoscopy in residency training: A national consensus project. J Minim Invasive Gynecol 2009;16:1–7.
    20. Visser E, van Rossum PSN, van Veer H, et al. A structured training program for minimally invasive esophagectomy for esophageal cancer- A Delphi consensus study in Europe. Dis Esophagus 2018;31.
    21. Zevin B, Levy JS, Satava RM, Grantcharov TP. A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery. J Am Coll Surg 2012;215:580–586.e3.
    22. Graham B, Regehr G, Wright JG. Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol 2003;56:1150–1156.
    23. Nagakawa Y, Nakamura Y, Honda G, et al. Learning curve and surgical factors influencing the surgical outcomes during the initial experience with laparoscopic pancreaticoduodenectomy. J Hepatobiliary Pancreat Sci 2018;25:498–507.
    24. Ohtsuka T, Ban D, Nakamura Y, et al. Difficulty scoring system in laparoscopic distal pancreatectomy. J Hepatobiliary Pancreat Sci 2018;25:489–497.
    25. Klompmaker S, van Zoggel DM, Watkins AA, et al. Nationwide evaluation of patient selection for minimally invasive distal pancreatectomy using American College of Surgeons’ National Quality Improvement Program. Ann Surg 2017;266:1055–1061.
    26. Bassi C, Marchegiani G, Dervenis C, et al.; International Study Group on Pancreatic Surgery (ISGPS). The 2016 update of the International Study Group (ISGPS) definition and grading of postoperative pancreatic fistula: 11 years after. Surgery 2017;161:584–591.
    27. van der Heijde N, Vissers FL, Boggi U, et al.; European consortium of Minimally Invasive Pancreatic Surgery (E-MIPS). Designing the European registry on minimally invasive pancreatic surgery: A pan-European survey. HPB (Oxford) 2021;23:566–574.
    28. Adam MA, Thomas S, Youngwirth L, et al. Defining a hospital volume threshold for minimally invasive pancreaticoduodenectomy in the United States. JAMA Surg 2017;152:336–342.
    29. Campbell SM, Hann M, Roland MO, et al. The effect of panel membership and feedback on ratings in a two-round Delphi survey: Results of a randomized controlled trial. Med Care 1999;37:964–968.
    30. Mealing NM, Banks E, Jorm LR, et al. Investigation of relative risk estimates from studies of the same population with contrasting response rates and designs. BMC Med Res Methodol 2010;10:26.

    Supplemental Digital Content

    © 2022 by the American College of Surgeons. Published by Wolters Kluwer Health, Inc. All rights reserved.