Evaluation and usability study of low-cost laparoscopic box trainer “Lap-Pack”: a 2-stage multicenter cohort study

Supplemental Digital Content is available in the text. Introduction: Laparoscopic training is restricted in low resource settings due to limited access to specialist training equipment and financial constraints. This study aimed to evaluate simulation skills and usability of an original low-cost laparoscopic trainer, the “Lap-Pack,” developed at the University of Leeds, UK. Methods: Stage I evaluation was conducted in Kolkata (India) between March, 12 and 14, 2019. Laparoscopic simulation training was based on the 5 domains of fundamentals of laparoscopic surgery (FLS), which assessed skill acquisition across 7 rural surgeons from North-East India. The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) criteria was used to statistically analyze trainee performance between pretraining and posttraining sessions. Also, Lap-Pack was qualitatively compared with a commercial box trainer, Inovus Pyxus HD (IPHD). Stage II involved a multi-center usability study in 2 centers of India and the United Kingdom (2019). Seventy-eight participants performed 2 FLS tasks using Lap-Pack and provided scores on a 25-point questionnaire, including a preestablished Face-Validity Criteria and 4 evaluation categories—Usability, Camera, View, and, Material. Results: In stage I, the total posttraining MISTELS score for Lap-Pack was higher, that is 773.37 (SD: 183.67) than pretraining score, that is 351.2 (SD: 471.5). The posttraining scores showed laparoscopic skill acquisition with statistically significant (P<0.05) difference for precision cutting, intracorporeal and extracorporeal knot. In stage II, Lap-Pack scored highly in Face-Validity with a combined mean score of 4.81 [95% confidence interval (CI): 4.52–5.09, P<0.05] out of a possible 6. It scored highest (scale: 1=low to 7=high) in Usability 6.14 (95% CI: 6.05–6.22, P<0.05) and Camera 6.14 (95% CI: 6.01–6.27, P<0.05). The “Lightweight” (6.46, 95% CI: 6.32–6.60, P<0.05) and “Portability” (6.35, 95% CI: 6.18–6.51, P<0.05) features of Lap-Pack were appreciated. Conclusion: The Lap-Pack is a suitable low fidelity simulator for laparoscopic training in a low-resource setting.


Introduction
The Lancet Commission on Global Surgery estimates that around 5 billion people globally do not have timely access to affordable surgical care [1,2] .The provision of surgical care in rural settings is particularly disadvantaged with few trained surgeons, limited resources, financial and time pressures, and long distances to access health care facilities.Surgery to save lives or prevent disability is inaccessible for many single-income households in low and middle-income countries (LMICs) [3][4][5] .It is estimated that 143 million additional surgical procedures are needed each year to address the clinical need in LMICs, of which around 18 million might be suitable for a laparoscopic approach [6] .
In contrast, laparoscopic surgery has been widely adopted in high-income countries (HIC) with documented benefits in reducing postoperative pain and complications, shortening hospital stay, and increasing efficiency and economics for health care providers [2] .Laparoscopic surgery is feasible in LMICs with various training programs having used cadaveric or animal models [7,8] , training workshops [9][10][11][12] , didactic lectures [13] , and tele-monitored programs [14] .But, most of these programs are not accredited [15] and laparoscopic training is often unstructured and opportunistic, requiring self-motivated learning and maintenance of skills [16] .Lowfidelity box trainers are potentially effective in addressing financial constraints in laparoscopic training, facilitating its wider adoption with a minimal surgical care gap in LMICs [17] .
One such low-cost, low-fidelity trainer was developed by Storm Lab at the School of Electronics and Electrical Engineering at the University of Leeds, in conjunction with the NIHR Global Health Research Group in Surgical Technologies (NIHR GHRG-ST) at St James' University Hospital.This study aims to evaluate this low-cost laparoscopic trainer, "Lap-Pack," through a 2-stage approach: (I) a structured training program in a low resource environment (Kolkata Medical College, India) to test laparoscopic skills acquisition, utility and acceptability [27] , and (II) an international, multi-center usability study in Maulana Azad Medical College (MAMC, India), Association of Rural Surgeons of India [28] conference (ARSICON, Bagalkot, India) and St. James University Hospital (SJUH), Leeds, UK.

Materials and methods
Stage I evaluation involved skill transfer assessment of 7 rural surgeons of North East India (Assam, Nagaland, Manipur, and Arunachal Pradesh) under a structured training program (TARGET) developed by the NIHR GHRG-ST [29] and based on fundamentals of laparoscopic surgery (FLS) principles [30] .All trainees were qualified and experienced ("SDC  Medical College from March 12 to 14, 2019, where Lap-Pack was provided along with a commercial box-trainer, i.e.Inovus Pyxus HD (IPHD) [27] for conducting a qualitative comparison.Figure 1A (version 1) shows the features of Lap-Pack: a lightweight design built from corrugated plastic.The collapsible feature makes it easy to disassemble as a flat-pack into a portable system.The visualization was provided through an endoscopic camera (ie, Pancellent LLC, USB endoscope camera introduced via camera port), which had an integrated light suitable for illuminating and monitoring the operative field.This camera can slide back-forth, rotate, twist in an angular direction and be fixed in the desired position.It is compatible with tablet, phone, and computer screens through its USB interface and operable with free android applications, such as "USB Camera" (by ShenYao China) or "Camera Fi" (by Vault Micro, Korea.).The camera had a narrow field of view of 66 degrees with a resolution of 640 × 480p and a focal distance between 4 cm-infinity.Figure 1B shows the design of IPHD [27] , which has a portable plexiglass structure, an internal cavity illuminated through an LED light source.A 1080p HD camera is fixed on the inner surface of the box and is compatible with Mac and PC.The user interface comprises 2 predefined port sites on an elastic wall fixed to the surface.The trainees were assessed for their laparoscopic knowledge through a precourse assessment and online didactic lectures, including (i) preoperative considerations, (ii) intraoperative considerations, (iii) basic laparoscopic procedures, (iv) postoperative care and complications, and (v) manual skills training (with permission from FLS [31,32] ).The manual skills training provided information about setting up both the training boxes and instructions on 5 skill domains; (i) peg transfer, (ii) ligating loop, (iii) precision cutting, (iv) suture with an extracorporeal knot, and (v) suture with intracorporeal knot (Fig. 1C).Trainee skill acquisition was assessed through practical sessions, which included a pretraining and posttraining assessment.During the pretraining assessment, the initial laparoscopic skills of the trainees were assessed using the FLS manual skill completion criteria [31,32] and MISTELS proficiency score [32] .All trainees were required to undertake a series of FLS laparoscopic exercises on both Lap-Pack and IPHD.Thereafter, individual training from the faculty was provided to the trainees to improve trainees' skills and performance.In the posttraining session, the trainees were asked to reperform each of the FLS tasks on Lap-Pack and IPHD.For each task performed, the trainee was scored by 2 experienced laparoscopic surgeons (assessors) using the MISTELS proficiency criteria.
Trainees were later asked to scores on the structure, task view, camera and task-specific skills of Lap-Pack, and IPHD on a 7-point Likert scale ("SDC, Table 2," Supplemental Digital Content 2, http://links.lww.com/IJSGH/A11), with "7" indicating strong approval and "1" indicating strong disapproval.The qualitative nature of this comparative feedback (discussed later) was then used to build an improved version of Lap-Pack, (Fig. 1A, version 2).The design material remained the same corrugated plastic, allowing dispersed ambient brightness to act as the light source for the box.Additions included an elastic abdominal wall moulded from silicone (Dragon Skin 30, Smooth-On Inc., USA) for simulated placement of laparoscopic ports, a stronger reinforced structure, and a 1080P webcam USB wide angle 2.1 mm lens (CMOS AR0330 Mini H.264 camera).
Stage II usability evaluation was conducted with the redesigned Lap-Pack in multiple centers, 1 rural health care center (Bagalkot, at ARSICON [28] , India, November 2019) and 2 urban teaching hospitals in India (MAMC, Delhi, August 2019), and a teaching hospital in the United Kingdom (SJUH, Leeds, September 2019).This study involved 78 participants comprising of 27 senior surgeons, 29 junior trainees and 22 medical students.
Participants were specifically required to complete 2 laparoscopic training tasks, peg transfer and precision cutting [6,33,34] , to provide a standardized experience with the equipment; their proficiency in performing the skills was not assessed.They were then asked to evaluate Lap-Pack in a 25-point questionnaire, comprising of a previously described [35] Face-Validity Criteria score ("SDC Table 2," Supplemental Digital Content 2, http://links.lww.com/IJSGH/A11), a sum of the criteria listed under "A" and "B," (scale 1, low to 6, high) [18] .This was followed by 4 major evaluation categories-Usability, Camera, View, and Material ("SDC Table 3," Supplemental Digital Content 3, http://links.lww.com/IJSGH/A12refined from "SDC Table 2," Supplemental Digital Content 2, http:// links.lww.com/IJSGH/A11)-whereinparticipants ranked subcriteria in each category on a 7-point Likert scale.All studies were conducted in compliance with the STROCSS criteria [36] and was registered with unique identification number "6938" [37] .

Stage I
Trainee 4 completed the highest number of FLS tasks in both pretraining and posttraining sessions (most experienced-"SDC Table 1," Supplemental Digital Content 1, http://links.lww.com/IJSGH/A10).Hence, his scores were chosen to normalize the results of other trainees, by dividing the total score of all trainees (for a particular task) by the score achieved by trainee 4 (in the pretraining and posttraining session) and multiplying that number by 100.Table 1a shows the normalized MISTEL scores for Lap-Pack.The overall total score (along with mean, SD, and P-value) was calculated by adding the individual trainee score for all 5 tasks.Total scores are compared for Lap-Pack in (Table 1

b).
The pretraining and posttraining assessment of normalized MISTELS scores (individual/overall) for each of the 5 FLS tasks on Lap-Pack are shown in Figures 2A-F, respectively.
Statistically significant increased MISTELS scores were observed for all trainees for precision cutting, intracorporeal and extracorporeal knot-tying (Fig. 2B, D, E) in comparison to the pretraining session (Table 1a).Some trainees showed reduction in performance scores for ligating loop and peg transfer (Figs.2A, C).The total normalized MISTELS scores (Fig. 2F) for all trainees increased in the posttraining session (except trainee 8).It can be observed in (Table 1b) that the mean scores for Lap-Pack increased in the posttraining session (773.37,SD: 183.67) in comparison to the pretraining session (351.20,SD: 471.55), although this was not statistically significant (P = 0.15).
Table 2 shows the mean scores obtained from trainees using the questionnaire to compare the features of IPHD and Lap-Pack.Structural durability and usability: IPHD scored higher for the material used to construct the trainer (plexiglass vs. corrugated plastic), but Lap-Pack scored higher for the ease of assembly and portability.Task view and Camera: Lap-Pack scored lower for the easy view of the task as a function of ambient light, background color, angle of view, image quality, and depth perception etc., which was attributed to the superior quality of the IPHD camera.Task-specific skills: the scores for both box trainers were relatively similar for all the 5 FLS tasks and task-specific skills.
The overall mean scores for IPHD and Lap-Pack were 5.42 (SD: 0.37) and 4.56 (SD: 0.71), respectively.This user feedback was considered to create version 2 of the Lap-Pack (Fig. 1A), which was further evaluated in stage II.

Stage II
Findings across India and the UK cohorts were similar throughout, suggesting the universal application of Lap-Pack as a training tool across economic settings.Score "0" implied that the trainee could not complete the task and exceeded the time limit.Quantitative Face validity scores for urban settings in India (MAMC) and United Kingdom (SJUH) were observed to be the same with a score of 4.63 (Table 3), while the same criterion scored higher (5.39) in a rural setting (ARSICON).Regarding the 4 evaluation categories, Usability and Camera were consistently the highest-scoring categories across all cohorts and sub-groups, with combined mean scores of 6.13 [95% confidence interval (CI): 6.05-6.22,P < 0.05) and 6.14 (95% CI: 6.01-6.27,P < 0.05) out of a possible 7, respectively.In addition, Lap-Pack's lightweight, mobility, quality of image, and color of image features were among the highest scoring subcriteria within these categories in all cohorts ("SDC Table 4," Supplemental Digital Content 4, http://links.lww.com/IJSGH/A13).Views scored slightly lower than the other categories with a combined mean score of 5.60 (95% CI: 5.49-5.71,P < 0.05), primarily due to difficulty attaining a good view for task completion and "background color contrast."Materials scored the lowest with a combined mean score of 5.21 (95% CI: 5.00-5.42,P < 0.05).Overall, the lightweight (6.46, 95% CI: 6.32-6.60,P < 0.05) and portability (6.35, 95% CI: 6.18-6.51,P < 0.05) of Lap-Pack were highly appreciated.

Discussion
Laparoscopic surgery requires surgical task-specific eye-hand coordination skills, which involve managing long instruments tremors, 3-dimensional depth perception, instrument targeting along fulcrum effect [38,39] .Such skills acquisition turn out to be expensive in "see one, do one, teach one" apprenticeship model [40] or animal training due to limited availability, ethical issues, etc.In LMICs, surgical trainees have limited options outside of participating in expensive courses with minimal chances of regular hospital practice.The unstructured and opportunistic nature of such training programs [15,16] has created an unmet need for low-fidelity, cheap box trainers.Various studies confirm skill acquisition on low-cost box trainers but lack rigorous validation [41,42] through a standardized evaluation process [22,[43][44][45][46] , such as MISTELS.Some studies use the FLS tasks for evaluation but are limited by a small number of participants comprising of medical students [47,48] , or resident surgeons [6,49] .According to a survey [50] , the introduction of novel simulation and continuing medical education can address the barriers to the adoption of laparoscopic surgery.This study addresses the above with limitations with multi-center skill acquisition and usability evaluation, with 85 participants (stage I and II combined), in LMIC and HIC settings, through the FLS authorized training program.
This study was not without its limitations.For example, "Views" scored slightly lower than the other categories, primarily due to difficulty attaining a wide-angle view and "background color contrast."This arises from one of the general limitations of low-cost trainers in missing the ability to zoom in and focus on tasks with ease as would be routinely done in the operating room.While there is potential to add this feature to the device, this would have to be carefully balanced with manufacturing costs in order to ensure the trainer continues to be affordable for LMICs.Alternatively, background color contrast can be easily improved by changing the background color of the fixation plates used for tasks.
The low score for "Materials" is explained by participant testimonials that they could not confidently answer some of the questions given the short amount of time they had to utilize the Lap-Pack.For instance, without being able to test it first-hand, participants questioned the durability of Lap-Pack's corrugated plastic and its waterproof capability.One potential way of improving trainees' assessment would be to study Lap-Pack's usability as a personally affordable at-home device-a role many participants suggested.This would allow participants ample time to self-assemble, utilize, and test Lap-Pack's material thoroughly.
This study has shown that Lap-Pack is a promising laparoscopic box trainer that allows skill acquisition and performance improvement.Acquiring surgical skills [17,21] has been shown to be is a function of perceptual awareness, comprehension, speed, efficiency, and precision.This can improve with increased repetition on box trainers.Lap-Pack has potential to be employed as a home or officebased low-cost box trainer which allows its users to update selfpaced skill improvement over a prolonged duration.
Where cost is concerned, self-assembly trainers require sourcing of construction material for abdominal wall simulation, laparoscopic ports, light-source, camera, and visualization screen [18] .The uniqueness of the Lap-Pack system is that it incorporates an adjustable camera, an in-built abdominal wall for port placement, and smart device enabled visualization, all of which can be easily sourced even in LMIC settings.The lab prototyping cost of Lap-Pack was £100 and £130 for versions 1 and 2, respectively, which included the structural material, endoscopic camera, abdominal wall with laparoscopic ports and manufacturing cost.For both the studies, the Lap-Pack with consumables (peg transfer board, circular gauge, ligating loop, pen rose drain) and instruments (Needle holder, Maryland grasper, knot pusher) were provided at no cost to participants through NIHR Global Surgery funding.These instruments and consumables would have an added cost of £110.Thus, the overall cost of using the Lap-Pack (version 1 and 2) in a training course was £210 and £285, which was still cheaper than the IPHD (£460).Future work will involve manufacturing the Lap-Pack in LMICs to make it available to surgical trainees in country: it is now being considered for commercialization via a local manufacturer in India [51] .Another way of widening access to a larger number of trainees is providing them with the capability of building the Lap-Pack via do-it-yourself instructions (with the provision of low-cost, elemental building blocks).A digital platform or web application-based feedback system is planned for monitoring trainee skill acquisition.These steps would help to improvement in the design features of Lap-Pack and make it accessible to a larger number of surgical trainees.

Figure 1 .
Figure 1.Design features of Lap-Pack; (A) version 1 and version 2. B, Design features of Inovus box trainer.C, Consumable task for laparoscopic training.

Table 1 ,
" Supplemental Digital Content 1, http://links.lww.com/IJSGH/A10)doctors practising in rural hospitals and performing open surgical procedures.The training program was conducted at the Kolkata

Table 2
Comparison of mean usability scores for Lap-Pack and Inovus Pyxus HD.
The above scores are mean values of scores obtained for each category (on a 7-point Likert scale).