Surgical innovation is recognized as having unique methodological and practical barriers that make rigorous evaluation and quality reporting inherently complex.1 The Idea, Development, Exploration, Assessment, Long Term Study (IDEAL) Framework and Recommendations were introduced in 2009 to clarify and address the key challenges presented in surgical innovation.2 The IDEAL integrated evaluation pathway guides the conduct and reporting of clinical studies of the outcomes of surgery and complex therapeutic interventions. The IDEAL Framework has evolved over time, and now includes a model for medical devices evaluation, IDEAL-D, and a modified framework for physical therapy and rehabilitation, IDEAL-Physio.3–6
Although widely accepted in principle, the use of the IDEAL recommendations in practice to date has been limited and in many cases suboptimal. Surveys of publications using IDEAL have shown that reporting of the stages was often inaccurate and the key features which define each stage were frequently omitted.7–9 The IDEAL Recommendations introduce several novel ideas into clinical study design, and following other published guidelines alone is therefore unlikely to enable authors to report them accurately. The lack of clear and detailed instructions on reporting has been considered to be a barrier to wider adoption of IDEAL, and there have been calls for the development of reporting guidelines specific to IDEAL.10,11
The EQUATOR Network (http://www.equator-network.org) defines a reporting guideline as a simple, structured tool for health researchers to use while writing manuscripts to guide the authors in reporting a specific type of research.12 Reporting guidelines are developed using an explicit methodology and provide a minimum list of information needed to ensure a manuscript can be understood and assessed by a reader, replicated by a researcher, used to make a clinical decision, and included in a systematic review. There are now >400 checklists on the EQUATOR Network's website, including 42 that are surgery related (http://www.equator-network.org/reporting-guidelines/). Among these existing reporting guidelines applicable to surgery SCARE,13 PROCESS,14 STROCSS,15 and CONSORT NPT16 aim to improve the reporting of specific, well-recognized study types (case reports, case series, cohort studies, and randomized controlled trials). Additional guidance exists for specific aspects of surgical evaluative study designs. For example TIDIER 17 has been developed as an extension of CONSORT to improve reporting of intervention details, and the COHESIVE18 guidelines are being developed for a generic core outcome set in studies of early phase surgery and medical devices. A CONSORT extension for RCTs using cohorts and routinely collected health data is also underway.19 Additionally, the National Evaluation System for Health Technology Coordinating Center (NESTcc) Methods Framework has been developed to define the key components of study design, protocols, and evidence gathering for the evaluation of medical devices20
All these guidelines draw upon principles of good research methodology, as does the IDEAL framework. However the IDEAL reporting guidelines have a unique and specific focus on stepwise evaluation of surgical innovation through all stages of its development, from pre-clinical through first in human evidence, feasibility and randomized trials to long-term follow-up including use of real-world data. Throughout these stages, IDEAL prescribes specific novel study formats, focusing on clarity about key elements of surgical innovation, with which many researchers may be unfamiliar. This likely explains the inaccuracy in reporting noted in studies attempting to follow the IDEAL recommendations and suggests a pressing need for IDEAL reporting guidelines.
The development of such IDEAL-specific guidelines would serve to improve the usability and impact of IDEAL. As Altman and Moher state, “reporting problems affect journal articles in 2 main ways. First, the study methods are frequently not described in adequate detail. Second, the study findings are presented ambiguously, incompletely, or selectively. The cumulative effect of these problems is to render many reports of research unusable or even harmful; at the very least, such articles certainly represent a waste of resources”.21 Chalmers and Glasziou have highlighted the detrimental effects of waste in the production and reporting of research evidence and call for increased use of reporting guidelines as part of the solution.22
This article describes the development of a set of IDEAL reporting guidelines which aim to improve the quality, transparency, consistency, and utility of the surgical innovation evidence base. The protocol for the project was published in 2018.23
We developed the IDEAL reporting guidelines through a 5-phase process between October 2018 and May 2019. This comprised identifying existing relevant reporting guidelines and those in development registered on the EQUATOR Network website (http://www.equator-network.org), a survey to establish which stages of IDEAL, if any, participants considered in need of new reporting guidelines, the development of draft checklists for Stages 1, 2a, 2b, and 4 following establishment of need based on consensus, 2 rounds of a Delphi consensus survey to refine each of the checklists, and a consensus meeting via teleconference with the working group to finalize the checklists.
Using our MailChimp e-newsletter, we contacted 445 individuals who met at least 1 of 4 criteria: member of the original IDEAL steering group; a wider member of the IDEAL Collaboration signed up to receive newsletters; author or co-author of any key core IDEAL publication; attended ≥1 of the 2016, 2017, and 2018 IDEAL conferences. From this group, 119 expressed interest in participating, and those individuals were invited to complete an online survey about the need for reporting guidelines for any stage of IDEAL, as defined by the EQUATOR Network's suggested criteria.24 The 61 respondents recommended that new reporting guidelines should be developed for Stage 1 (69%), Stage 2a (78%), Stage 2b (75%), and Stage 4 (66%) (Supplementary Figure 1, http://links.lww.com/SLA/C301). IDEAL Stage 3 was considered adequately covered by existing guidelines (CONSORT, CONSORT NPT and other relevant CONSORT extensions) by 68% of respondents. Characteristics of responding participants are shown in Table 1. Fourteen volunteers from groups (1) and (3) formed a working group to agree the final checklists during the final consensus call.
TABLE 1 -
Characteristics of Survey Participants (% of group—N = 61)
|No. of years qualified
||Rest of Europe
∗(May have multiple roles).
†Including: medical ethicist, program manager, scientist, statistician, trial methodologist, private practice.
Development of Checklists
The initial drafts of checklists for the 4 stages were developed using a combination of approaches to identify what key items may be needed. These included ongoing monitoring of the literature using or commenting on the usability of IDEAL, identified from Google Scholar and Web of Science Alerts of articles citing the 2009 and 2019 IDEAL Statement articles, assessing gaps in existing reporting guidelines, and accumulated refinements of IDEAL principles developed from issues repeatedly raised in recent years during educational seminars and lectures on IDEAL. These checklists were evaluated in 2 rounds of a Delphi consensus survey using a 9-point Likert scale, (as recommended by GRADE), to rate agreement on the importance of each element and elicit free-text comments on the wording of each item.25,26 In the Likert scale, a score of 1 to 3 signified limited importance of the item, 4 to 6 important but not critical, and 7 to 9 critical. If ≥70% of respondents scored an item 7 to 9 and >15% scored it 1 to 3, then that item was included in the reporting guideline. Conversely, consensus that an item should not be included was defined as ≥70% scoring it 1 to 3 and ≤15% scoring it 7 to 9. Fifty-four participants completed Round 1 of the Delphi. Two reviewers (N.B., A.H.) reviewed the responses and modified the checklist drafts based on the participants scoring and feedback, editing phrasing and wording of checklist items, and merging redundant items, but ensuring meaning was not significantly changed. Round 2 of the Delphi survey presented the results of round one to participants, addressed clarifications, and served to refine the checklist items where consensus was not yet reached. Participants were again asked to score the Round 2 checklist items on a 1 to 9 Likert Scale. Forty-seven participants completed Round 2 and their responses were analyzed by 2 researchers (N.B., A.H.) and presented to the working group during the final teleconference consensus meeting.
A consensus call held in May 2019 was attended by 11 of the 14 members of the working group. Three participants who were unable to attend submitted written contributions via email. A draft of the methods, checklist drafts for each stage, and a provisional list of items where the Delphi process had not yielded a definitive result were prepared in advance and sent to the call participants. The items that remained controversial were discussed and modifications were made once the group reached agreement. All checklist items for each stage were then reviewed and revised as needed to ensure clarity. The group also decided on a strategy to disseminate the checklists. The revisions were then sent out to the participants for final comment and approval.
The citation alerts and search of the EQUATOR Network list of reporting guidelines revealed no specific guidelines dealing with the IDEAL Recommendations as an entity either as the sole focus of study or as part of a larger ambit, and no guidelines which specifically described how to report IDEAL format studies. Numerous guidelines contained generic recommendations also included by IDEAL, for example, use of standard outcome definitions, publication of protocols, and full informed consent, but none referenced IDEAL as a special case.14–19
Supplementary Tables 1, http://links.lww.com/SLA/C302 and 2, http://links.lww.com/SLA/C303 show the distribution of scores for each checklist item in Rounds 1 and 2 respectively of the Delphi process. The 4 unresolved issues following Round 2, discussed during the final consensus call were: Inclusion of IDEAL and the stage number in titles; inclusion of “first-in-human” in the title of Stage 1 studies; mandatory publication of a previous protocol for all studies; and how to resolve tensions between commercial confidentiality and ethical demands for transparency in early-stage studies involving devices. The solutions to each are incorporated in the final full checklists presented here.
IDEAL is based on ethical and methodological principles which apply to all stages and types of medical research, and many of which are held in common with other methodological paradigms and checklists. Supplementary Table 3, http://links.lww.com/SLA/C820 presents the full checklists for IDEAL Stages 1, 2a, 2b, and 4 and highlights key or unique IDEAL items with an example from the literature, which illustrates appropriate use. We have not provided examples for more familiar items common to other reporting checklists. The article quoted does not necessarily demonstrate good reporting for other IDEAL recommendations.
The IDEAL Recommendations include several innovations in study design for clinical research, such as reporting of iterative modifications and monitoring of learning curves. Because of this, the diligent use of other guidelines cannot help investigators to report IDEAL studies appropriately, and an IDEAL-specific guideline is required. For example comparing the SCARE guidelines for surgical case reports13 and the IDEAL Stage 1 recommendations, IDEAL specifies requirements absent from SCARE (Applicability only to first-in-human studies, justify need and refer to pre-clinical development, detailed account of patient selection, requirement for fully informed consent for treatment, requirement for explanation of procedure detailed enough to allow reproduction).
The IDEAL checklists are intended to provide a minimum list of concepts authors should include in a report of an IDEAL format study. They have been created for wide-spread adoption across all interventional specialties, including therapeutic devices, for which IDEAL has published a modified framework, IDEAL-D.5 The checklists are equally usable for IDEAL and IDEAL-D studies, most items pertaining to both surgical and device innovation. Although not designed expressly for these purposes, the guidelines can also be used both prospectively to help plan a study and retrospectively to assist in appraisal. They can also be used alongside other reporting guidelines to provide additional useful detail (eg, COHESIVE for core outcomes reporting, CONSORT extension for routinely collected health data for Stage 4).18,19
Dissemination and Evaluation of IDEAL Checklists
Checklists are not effective unless used, so dissemination efforts are important. The EQUATOR network27 recommends 10 specific actions to aid dissemination: making the checklist freely accessible in editable format; publishing and presenting editorials, blogs, website pages, conference papers and webinars, co-operating with EQUATOR, seeking journal endorsements and running training courses. We intend to follow this guidance utilizing our website and Twitter accounts (www.ideal-collaboration.net and @IDEALCollab). Some specific elements of our strategy will be the use of social media to establish a feedback dialogue with users, presentation of the guidelines at conferences and scientific meetings, editorials and blogs, and translation into Chinese and Spanish to maximize international readership. In line with EQUATOR guidance, we will pilot the checklist in ongoing IDEAL format studies, for which we will shortly be introducing an advisory service, and will publish an updated version with any modifications resulting from both feedback and piloting. A pre-post guideline publication survey of reporting of IDEAL format studies to evaluate the effectiveness of these efforts will be developed at a future date. Endorsement by authoritative bodies and efforts by editors to encourage or enforce compliance were features of the development and spread of the CONSORT guidelines, which appear to be the best complied with to date, so we will continue to seek support in these sectors.
Like all consensus processes, our Delphi study was dependent on the representativeness of participants. We invited participants with a good understanding of IDEAL, as a naïve group would be unlikely to be able to appreciate some of the issues or achieve consensus. We attempted to include a wide range of stakeholders but were not universally successful. The number of journal editors, for example, was small. A different group might have come up with different recommendations, but no major differences in responses could be detected according to background in our sample. Although consensus was achieved by GRADE criteria on the need for a guideline in all stages except Stage 3, the minority which disagreed was substantial (around 25%) in each case. Informal feedback indicates that this may have been because participants familiar with IDEAL had come to see reporting of IDEAL stage items as straightforward, and not needing guidance. Some participants also mentioned the proliferation of reporting guidelines as a reason for caution.
The consensus-based checklists developed through this Delphi process have the potential to improve the standards of reporting of early-stage innovation and surgical trials and thereby advance the quality of research on surgery and complex interventions. Surgical innovation and research that follows a transparent, sequential framework such as IDEAL and is reported with these guidelines can also aid reproducibility, reduce research waste, and improve evidence-based practice for the benefit of patients.
IDEAL Collaboration Reporting Guidelines Working Group: Jane Blazeby, Maroeska Rovers, Christopher Pennell, Joel Horovitz, Angelos Kolias, Janet Martin, Tom Lewis, Riaz Agha, Josh Feinberg, in addition to the authors.
1. Ergina PL, Cook JA, Blazeby JM, et al. Challenges in evaluating surgical innovation. Lancet
2. McCulloch P, Altman DG, Campbell WB, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet
3. Pennell CP, Hirst AD, Campbell WB, et al. Practical guide to the idea, development and exploration stages of the IDEAL framework and recommendations. Br J Surg
4. Hirst A, Philippou Y, Blazeby J, et al. No surgical innovation without evaluation: evolution and further development of the IDEAL framework and recommendations. Ann Surg
5. Sedrakyan A, Campbell B, Merino JG, et al. IDEAL-D: A rational framework for evaluating and regulating the use of medical devices. BMJ
6. Hamilton D, Beard D, McCulloch P, et al. Evidence-based evaluation of practice and innovation in rehabilitation using the IDEAL-physio framework. Phys Ther
7. Khachane A, Philippou Y, Hirst A, et al. Appraising the uptake and use of the IDEAL framework and recommendations: a review of the literature. Int J Surg
8. Tradewell MB, Albersheim J, Dahm P. Use of the IDEAL framework in the urological literature: where are we in 2018? BJU Int
9. Currie A, Brigic A, Blencowe NS, et al. Systematic review of surgical innovation reporting in laparoendoscopic colonic polyp resection. Br J Surg
10. Barkun JS, Dimick JB, Clavien PA. Surgical research
in patients: ideal time for an IDEAL checklist. Ann Surg
11. Kirkham EN, Main BG, Jones KJB, et al. Systematic review of the introduction and evaluation of magnetic augmentation of the lower oesophageal sphincter for gastro-oesophageal reflux disease. Br J Surg
12. EQUATOR Network website. What is a Reporting Guideline? Available at: https://www.equator-network.org/about-us/what-is-a-reporting-guideline/
(Accessed March 30, 2020)
13. Agha RA, Borrelli MR, Farwana R, et al. For the SCARE Group. The SCARE 2018 Statement: Updating Consensus Surgical CAse REport (SCARE) Guidelines. Int J Surg
14. Agha RA, Borrelli MR, Farwana R, et al. For the PROCESS Group. The PROCESS 2018 Statement: Updating Consensus Preferred Reporting of CasE Series in Surgery (PROCESS) Guidelines. Intl J Surg
15. Agha R, Abdall-Razak A, Crossley E, et al. STROCSS 2019 Guideline: strengthening the reporting of cohort studies in surgery. Int J Surg
16. Boutron I, Altman DG, Moher D, et al. CONSORT NPT Group. CONSORT statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts. Ann Intern Med
17. Hoffmann T, Glasziou P, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ
18. Avery K, Blazeby J, Wilson N, et al. Development of reporting guidance and core outcome sets for seamless, standardised evaluation of innovative surgical procedures and devices: a study protocol for content generation and a Delphi consensus process (COHESIVE study). BMJ Open
19. Kwakkenbos L, Juszczak E, Hemkens LG, et al. Protocol for the development of a CONSORT extension for RCTs using cohorts and routinely collected health data. Res Integr Peer Rev
20. National Evaluation System for health Technology Coordinating Center (NESTcc) Frameworks Feb 2020. Available at: https://mdic.org/resource/nestcc-data-quality-framework/
(Accessed March 30, 2020).
21. Altman DG, Moher D. Guidelines for Reporting Health Research: A User's Manual
. In: Moher, D., A: https://onlinelibrary.wiley.com/doi/book/10.1002/9781118715598
22. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet
23. Agha RA, Hirst A, Khachane A, et al. A protocol for the development of reporting guidelines for IDEAL stage studies. Int J Surg Protoc
25. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ
2008; 336: 924 LP-926.
26. Jaeschke R, Guyatt GH, Dellinger P, et al. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ
27. EQUATOR Network – Developing and disseminating your reporting guideline. Available at: https://www.equator-network.org/toolkits/developing-a-reporting-guideline/disseminating-your-reporting-guideline/
(Accessed March 30, 2020).