The IDEAL Reporting Guidelines

A Delphi Consensus Statement Stage Specific Recommendations for Reporting the Evaluation of Surgical Innovation

Abstract

Objective: 

The aim of this study was to define reporting standards for IDEAL format studies.

Background: 

The IDEAL Framework and Recommendations establish an integrated pathway for evaluation of new surgical techniques and complex therapeutic technologies. However guidance on implementation has been incomplete, and incorrect use is commonly seen. We describe the consensus development of reporting guidelines for the IDEAL stages, and plans for their dissemination and evaluation.

Methods: 

Using the EQUATOR Network recommendations, participants with knowledge of IDEAL were surveyed to determine which IDEAL stages needed reporting guidelines. Draft checklists for stages 1, 2a, 2b, and 4 were subsequently developed by 3 researchers (N.B., A.H., P.M.), and revised through a 2-round Delphi consensus process. A final consensus teleconference resolved outstanding disagreements and clarified wording for checklist items.

Results: 

Sixty-one participants completed the initial survey, a clear majority indicating that new reporting guidelines were needed for IDEAL Stage 1 (69.5%), Stage 2a (78%), Stage 2b (74.6%), and Stage 4 (66%). A proposed set of checklists was modified by survey participants in 2 online Delphi rounds (n = 54 and n = 47, respectively), resulting in a penultimate checklist for each stage. Fourteen expert working group members finalized the checklist items and successfully resolved any outstanding areas without agreement on a consensus call.

Conclusions: 

Participants familiar with IDEAL called for reporting guidelines for studies in all IDEAL stages except stage 3. The checklists developed have the potential to improve standards of reporting and thereby advance the quality of research on surgery and complex interventions and technologies, but require further evaluation in use.

Surgical innovation is recognized as having unique methodological and practical barriers that make rigorous evaluation and quality reporting inherently complex.¹ The Idea, Development, Exploration, Assessment, Long Term Study (IDEAL) Framework and Recommendations were introduced in 2009 to clarify and address the key challenges presented in surgical innovation.² The IDEAL integrated evaluation pathway guides the conduct and reporting of clinical studies of the outcomes of surgery and complex therapeutic interventions. The IDEAL Framework has evolved over time, and now includes a model for medical devices evaluation, IDEAL-D, and a modified framework for physical therapy and rehabilitation, IDEAL-Physio.^3–6

Although widely accepted in principle, the use of the IDEAL recommendations in practice to date has been limited and in many cases suboptimal. Surveys of publications using IDEAL have shown that reporting of the stages was often inaccurate and the key features which define each stage were frequently omitted.^7–9 The IDEAL Recommendations introduce several novel ideas into clinical study design, and following other published guidelines alone is therefore unlikely to enable authors to report them accurately. The lack of clear and detailed instructions on reporting has been considered to be a barrier to wider adoption of IDEAL, and there have been calls for the development of reporting guidelines specific to IDEAL.^10,11

Reporting Guidelines

The EQUATOR Network (http://www.equator-network.org) defines a reporting guideline as a simple, structured tool for health researchers to use while writing manuscripts to guide the authors in reporting a specific type of research.¹² Reporting guidelines are developed using an explicit methodology and provide a minimum list of information needed to ensure a manuscript can be understood and assessed by a reader, replicated by a researcher, used to make a clinical decision, and included in a systematic review. There are now >400 checklists on the EQUATOR Network's website, including 42 that are surgery related (http://www.equator-network.org/reporting-guidelines/). Among these existing reporting guidelines applicable to surgery SCARE,¹³ PROCESS,¹⁴ STROCSS,¹⁵ and CONSORT NPT¹⁶ aim to improve the reporting of specific, well-recognized study types (case reports, case series, cohort studies, and randomized controlled trials). Additional guidance exists for specific aspects of surgical evaluative study designs. For example TIDIER ¹⁷ has been developed as an extension of CONSORT to improve reporting of intervention details, and the COHESIVE¹⁸ guidelines are being developed for a generic core outcome set in studies of early phase surgery and medical devices. A CONSORT extension for RCTs using cohorts and routinely collected health data is also underway.¹⁹ Additionally, the National Evaluation System for Health Technology Coordinating Center (NESTcc) Methods Framework has been developed to define the key components of study design, protocols, and evidence gathering for the evaluation of medical devices²⁰

All these guidelines draw upon principles of good research methodology, as does the IDEAL framework. However the IDEAL reporting guidelines have a unique and specific focus on stepwise evaluation of surgical innovation through all stages of its development, from pre-clinical through first in human evidence, feasibility and randomized trials to long-term follow-up including use of real-world data. Throughout these stages, IDEAL prescribes specific novel study formats, focusing on clarity about key elements of surgical innovation, with which many researchers may be unfamiliar. This likely explains the inaccuracy in reporting noted in studies attempting to follow the IDEAL recommendations and suggests a pressing need for IDEAL reporting guidelines.

The development of such IDEAL-specific guidelines would serve to improve the usability and impact of IDEAL. As Altman and Moher state, “reporting problems affect journal articles in 2 main ways. First, the study methods are frequently not described in adequate detail. Second, the study findings are presented ambiguously, incompletely, or selectively. The cumulative effect of these problems is to render many reports of research unusable or even harmful; at the very least, such articles certainly represent a waste of resources”.²¹ Chalmers and Glasziou have highlighted the detrimental effects of waste in the production and reporting of research evidence and call for increased use of reporting guidelines as part of the solution.²²

This article describes the development of a set of IDEAL reporting guidelines which aim to improve the quality, transparency, consistency, and utility of the surgical innovation evidence base. The protocol for the project was published in 2018.²³

METHODS

Research Design

We developed the IDEAL reporting guidelines through a 5-phase process between October 2018 and May 2019. This comprised identifying existing relevant reporting guidelines and those in development registered on the EQUATOR Network website (http://www.equator-network.org), a survey to establish which stages of IDEAL, if any, participants considered in need of new reporting guidelines, the development of draft checklists for Stages 1, 2a, 2b, and 4 following establishment of need based on consensus, 2 rounds of a Delphi consensus survey to refine each of the checklists, and a consensus meeting via teleconference with the working group to finalize the checklists.

Participants

Using our MailChimp e-newsletter, we contacted 445 individuals who met at least 1 of 4 criteria: member of the original IDEAL steering group; a wider member of the IDEAL Collaboration signed up to receive newsletters; author or co-author of any key core IDEAL publication; attended ≥1 of the 2016, 2017, and 2018 IDEAL conferences. From this group, 119 expressed interest in participating, and those individuals were invited to complete an online survey about the need for reporting guidelines for any stage of IDEAL, as defined by the EQUATOR Network's suggested criteria.²⁴ The 61 respondents recommended that new reporting guidelines should be developed for Stage 1 (69%), Stage 2a (78%), Stage 2b (75%), and Stage 4 (66%) (Supplementary Figure 1, https://links.lww.com/SLA/C301). IDEAL Stage 3 was considered adequately covered by existing guidelines (CONSORT, CONSORT NPT and other relevant CONSORT extensions) by 68% of respondents. Characteristics of responding participants are shown in Table 1. Fourteen volunteers from groups (1) and (3) formed a working group to agree the final checklists during the final consensus call.

TABLE 1 - Characteristics of Survey Participants (% of group—N = 61)

Professional role∗	Clinical	Academic	Educator	Editor/Publisher	Industry	Other†
	66	74	21	17	2	15
No. of years qualified	0–4	5–9	10–14	15–19	20+
	9	17	23	19	32	—
Location	UK	Rest of Europe	N. America	Australia	S. America	Asia
	47	34	10	5	3	2
Sex	Male	Female
	75	25	—	—	—	–

^∗(May have multiple roles).

^†Including: medical ethicist, program manager, scientist, statistician, trial methodologist, private practice.

Development of Checklists

The initial drafts of checklists for the 4 stages were developed using a combination of approaches to identify what key items may be needed. These included ongoing monitoring of the literature using or commenting on the usability of IDEAL, identified from Google Scholar and Web of Science Alerts of articles citing the 2009 and 2019 IDEAL Statement articles, assessing gaps in existing reporting guidelines, and accumulated refinements of IDEAL principles developed from issues repeatedly raised in recent years during educational seminars and lectures on IDEAL. These checklists were evaluated in 2 rounds of a Delphi consensus survey using a 9-point Likert scale, (as recommended by GRADE), to rate agreement on the importance of each element and elicit free-text comments on the wording of each item.^25,26 In the Likert scale, a score of 1 to 3 signified limited importance of the item, 4 to 6 important but not critical, and 7 to 9 critical. If ≥70% of respondents scored an item 7 to 9 and >15% scored it 1 to 3, then that item was included in the reporting guideline. Conversely, consensus that an item should not be included was defined as ≥70% scoring it 1 to 3 and ≤15% scoring it 7 to 9. Fifty-four participants completed Round 1 of the Delphi. Two reviewers (N.B., A.H.) reviewed the responses and modified the checklist drafts based on the participants scoring and feedback, editing phrasing and wording of checklist items, and merging redundant items, but ensuring meaning was not significantly changed. Round 2 of the Delphi survey presented the results of round one to participants, addressed clarifications, and served to refine the checklist items where consensus was not yet reached. Participants were again asked to score the Round 2 checklist items on a 1 to 9 Likert Scale. Forty-seven participants completed Round 2 and their responses were analyzed by 2 researchers (N.B., A.H.) and presented to the working group during the final teleconference consensus meeting.

Consensus Teleconference

A consensus call held in May 2019 was attended by 11 of the 14 members of the working group. Three participants who were unable to attend submitted written contributions via email. A draft of the methods, checklist drafts for each stage, and a provisional list of items where the Delphi process had not yielded a definitive result were prepared in advance and sent to the call participants. The items that remained controversial were discussed and modifications were made once the group reached agreement. All checklist items for each stage were then reviewed and revised as needed to ensure clarity. The group also decided on a strategy to disseminate the checklists. The revisions were then sent out to the participants for final comment and approval.

RESULTS

The citation alerts and search of the EQUATOR Network list of reporting guidelines revealed no specific guidelines dealing with the IDEAL Recommendations as an entity either as the sole focus of study or as part of a larger ambit, and no guidelines which specifically described how to report IDEAL format studies. Numerous guidelines contained generic recommendations also included by IDEAL, for example, use of standard outcome definitions, publication of protocols, and full informed consent, but none referenced IDEAL as a special case.^14–19

Supplementary Tables 1, https://links.lww.com/SLA/C302 and 2, https://links.lww.com/SLA/C303 show the distribution of scores for each checklist item in Rounds 1 and 2 respectively of the Delphi process. The 4 unresolved issues following Round 2, discussed during the final consensus call were: Inclusion of IDEAL and the stage number in titles; inclusion of “first-in-human” in the title of Stage 1 studies; mandatory publication of a previous protocol for all studies; and how to resolve tensions between commercial confidentiality and ethical demands for transparency in early-stage studies involving devices. The solutions to each are incorporated in the final full checklists presented here.

IDEAL Checklists

IDEAL is based on ethical and methodological principles which apply to all stages and types of medical research, and many of which are held in common with other methodological paradigms and checklists. Supplementary Table 3, https://links.lww.com/SLA/C820 presents the full checklists for IDEAL Stages 1, 2a, 2b, and 4 and highlights key or unique IDEAL items with an example from the literature, which illustrates appropriate use. We have not provided examples for more familiar items common to other reporting checklists. The article quoted does not necessarily demonstrate good reporting for other IDEAL recommendations.

DISCUSSION

The IDEAL Recommendations include several innovations in study design for clinical research, such as reporting of iterative modifications and monitoring of learning curves. Because of this, the diligent use of other guidelines cannot help investigators to report IDEAL studies appropriately, and an IDEAL-specific guideline is required. For example comparing the SCARE guidelines for surgical case reports¹³ and the IDEAL Stage 1 recommendations, IDEAL specifies requirements absent from SCARE (Applicability only to first-in-human studies, justify need and refer to pre-clinical development, detailed account of patient selection, requirement for fully informed consent for treatment, requirement for explanation of procedure detailed enough to allow reproduction).

The IDEAL checklists are intended to provide a minimum list of concepts authors should include in a report of an IDEAL format study. They have been created for wide-spread adoption across all interventional specialties, including therapeutic devices, for which IDEAL has published a modified framework, IDEAL-D.⁵ The checklists are equally usable for IDEAL and IDEAL-D studies, most items pertaining to both surgical and device innovation. Although not designed expressly for these purposes, the guidelines can also be used both prospectively to help plan a study and retrospectively to assist in appraisal. They can also be used alongside other reporting guidelines to provide additional useful detail (eg, COHESIVE for core outcomes reporting, CONSORT extension for routinely collected health data for Stage 4).^18,19

Dissemination and Evaluation of IDEAL Checklists

Checklists are not effective unless used, so dissemination efforts are important. The EQUATOR network²⁷ recommends 10 specific actions to aid dissemination: making the checklist freely accessible in editable format; publishing and presenting editorials, blogs, website pages, conference papers and webinars, co-operating with EQUATOR, seeking journal endorsements and running training courses. We intend to follow this guidance utilizing our website and Twitter accounts (www.ideal-collaboration.net and @IDEALCollab). Some specific elements of our strategy will be the use of social media to establish a feedback dialogue with users, presentation of the guidelines at conferences and scientific meetings, editorials and blogs, and translation into Chinese and Spanish to maximize international readership. In line with EQUATOR guidance, we will pilot the checklist in ongoing IDEAL format studies, for which we will shortly be introducing an advisory service, and will publish an updated version with any modifications resulting from both feedback and piloting. A pre-post guideline publication survey of reporting of IDEAL format studies to evaluate the effectiveness of these efforts will be developed at a future date. Endorsement by authoritative bodies and efforts by editors to encourage or enforce compliance were features of the development and spread of the CONSORT guidelines, which appear to be the best complied with to date, so we will continue to seek support in these sectors.

Limitations

Like all consensus processes, our Delphi study was dependent on the representativeness of participants. We invited participants with a good understanding of IDEAL, as a naïve group would be unlikely to be able to appreciate some of the issues or achieve consensus. We attempted to include a wide range of stakeholders but were not universally successful. The number of journal editors, for example, was small. A different group might have come up with different recommendations, but no major differences in responses could be detected according to background in our sample. Although consensus was achieved by GRADE criteria on the need for a guideline in all stages except Stage 3, the minority which disagreed was substantial (around 25%) in each case. Informal feedback indicates that this may have been because participants familiar with IDEAL had come to see reporting of IDEAL stage items as straightforward, and not needing guidance. Some participants also mentioned the proliferation of reporting guidelines as a reason for caution.

CONCLUSIONS

The consensus-based checklists developed through this Delphi process have the potential to improve the standards of reporting of early-stage innovation and surgical trials and thereby advance the quality of research on surgery and complex interventions. Surgical innovation and research that follows a transparent, sequential framework such as IDEAL and is reported with these guidelines can also aid reproducibility, reduce research waste, and improve evidence-based practice for the benefit of patients.

Contributors

IDEAL Collaboration Reporting Guidelines Working Group: Jane Blazeby, Maroeska Rovers, Christopher Pennell, Joel Horovitz, Angelos Kolias, Janet Martin, Tom Lewis, Riaz Agha, Josh Feinberg, in addition to the authors.