Investigating different typologies for the synthesis of evidence: a scoping review protocol

Abstract

Objective: 

The objective of this scoping review is to identify evidence synthesis types and previously proposed classification systems, typologies, or taxonomies that have guided evidence synthesis.

Introduction: 

Evidence synthesis is a constantly evolving field. There is now a plethora of evidence synthesis approaches used across many different disciplines. Historically, there have been numerous attempts to organize the types and methods of evidence synthesis in the form of classification systems, typologies, or taxonomies. This scoping review will seek to identify all the available classification systems, typologies, or taxonomies; how they were developed; their characteristics; and the types of evidence syntheses included within them.

Inclusion criteria: 

This scoping review will include discussion papers, commentaries, books, editorials, manuals, handbooks, and guidance from major organizations that describe multiple approaches to evidence synthesis in any discipline.

Methods: 

The Evidence Synthesis Taxonomy Initiative will support this scoping review. The search strategy will aim to locate both published and unpublished documents utilizing a three-step search strategy. An exploratory search of MEDLINE has identified keywords and MeSH terms. A second search of MEDLINE, Embase, CINAHL with Full Text, ERIC, Scopus, Compendex, and JSTOR will be conducted. The websites of relevant evidence synthesis organizations will be searched. Identified documents will be independently screened, selected, and extracted by two researchers, and the data will be presented in tables and summarized descriptively.

Details of this review project are available at: 

Open Science Framework https://osf.io/qwc27

Introduction

Robust and trustworthy evidence synthesis products are essential to inform policy, legislation, and clinical decision-making. Evidence synthesis (eg, systematic reviews, scoping reviews, realist reviews) as a field is undergoing radical transformation and expansion. This change is due to the increased demand by policy-makers, practitioners, and the community for access to summarized and trustworthy evidence to guide decision-making. Historically, evidence synthesis strategies focused on the effectiveness of an approach or treatment. However, there now exists a multitude of evidence synthesis types and methods conducted for multiple purposes across diverse fields of inquiry, evidence types, and questions.¹ As such, enabling researchers to ensure that they are undertaking the most appropriate review approach to respond to a clinical or policy question (evidence synthesis) and enabling decision-makers to ensure that they are receiving the most appropriate output for their needs (knowledge translation) are both critical.

There has been substantial growth in the different types of evidence synthesis, with systematic reviews having the highest profile. These reviews are considered the pillar of evidence-based health care² and are widely used to inform the development of trustworthy clinical guidelines.^3-5 Organizations and networks such as JBI, Cochrane, Environmental Evidence, and the Campbell Collaboration provide guidance on the conduct and reporting of different types of evidence synthesis, including systematic reviews. Although guidance from these groups may differ regarding the approaches to conducting evidence synthesis, all characterize systematic reviews as a predefined process that is comprehensive, unbiased, structured, and transparent. Systematic reviews should be conducted by review groups with specialized skills, who set out to identify and retrieve all of the international evidence that is relevant to a particular question (or questions) and to appraise and synthesize the results of this search to inform practice, policy, and, in some cases, further research.^3,6,7 Ultimately, the findings of systematic reviews should be reliable and meaningful to those who use them.⁸

As the methods to conduct systematic reviews have evolved and advanced, so too has the thinking around the types of questions that need to be answered in order to provide the best possible, evidence-based information.^3,9 Systematic review approaches are now able to address many different types of questions, from investigating the effectiveness of a particular clinical treatment or intervention to questions of feasibility, appropriateness, and meaningfulness.¹⁰ There are now many different types of systematic reviews, such as reviews of effectiveness, prevalence and incidence, etiology and risk, economic evidence, measurement properties, qualitative approaches, and more.

It is now acknowledged that different questions require different evidence synthesis types. For example, scoping reviews (which were previously referred to as scoping studies)¹¹ have emerged as a valid approach to answering broader questions not suited to a typical systematic review. In addition, other types of evidence synthesis have emerged, including realist reviews, mixed methods reviews, concept analyses, and others.^1,12–15 At least 15 different methods have been proposed for the synthesis of qualitative research alone.^16–19 These include thematic synthesis, narrative synthesis, realist synthesis, content analysis, meta-ethnography, and meta-aggregation.^16–18 Some preliminary typologies of these diverse review types have been developed and can be used as a reference for researchers, policy-makers, and funders to assist them when deciding on a review approach.^12-14

Typologies and taxonomies are types of classification systems, and are approaches to structuring, categorizing, or classifying items, ideas, approaches, or organisms (among others). Although various definitions exist, a typology can be thought of as based on “theoretical or conceptual distinctions,”^20(p.7) while a taxonomy “uses empirical observations to classify items into categories.”^20(p.7) The approaches of these typologies, taxonomies, and other classification systems vary, from those that focus on overall review methods (eg, systematic, scoping, or rapid reviews)¹³ to typologies that have identified the different types of a systematic review (eg, qualitative, effectiveness, or diagnostic test accuracy systematic reviews).¹

The development of typologies, taxonomies, and classification systems of evidence synthesis has not as yet been conducted through a systematic process, nor with extensive stakeholder engagement from members of the academic community. These typologies have also been criticized for being incomplete and lacking consistency in the terminology.²⁰ Therefore, there is a need to identify the current typologies, taxonomies, and classification systems, and to review their characteristics and the methodological approaches used in their development, as well as to map the evidence synthesis types addressed within each one. This scoping review will identify contemporary evidence synthesis types and previously proposed typologies, classification schemes, or taxonomies that have guided evidence synthesis to date through extensive database searching and hand-searching, and extensive stakeholder engagement. The results of this review will be an inventory of evidence synthesis classification schemes detailing distinct evidence synthesis types and their methods, which will then inform the development of a comprehensive online and publicly available data bank of evidence synthesis methods and methodologies.

Review questions

What typologies, taxonomies, and classification schemes or compendia have been proposed for evidence synthesis methods?

1. What evidence synthesis types/approaches have been described in typologies, taxonomies, and classification schemes or compendia discussing multiple evidence synthesis approaches?
2. What are the characteristics of these typologies, taxonomies, and classification systems or compendia approaches?
3. What are the facets against which evidence syntheses have been classified or distinguished from one another?
4. What are the methodological approaches to the development of these typologies, taxonomies, and classification systems?

Inclusion criteria

Participants

This criterion is not relevant to this review, as we will be investigating typologies, taxonomies, and classification schemes for evidence synthesis.

Concept

The concepts of interest are classification schemes, taxonomies, typologies, compendia, groupings, or overviews of evidence synthesis methods and approaches, particularly structured resources that describe and distinguish evidence synthesis approaches from each other. This could include manuals and guidance for conducting evidence synthesis projects that have explored multiple evidence synthesis approaches, such as the JBI Manual for Evidence Synthesis or the Cochrane Handbook. Guidance papers or manuals that do not include multiple types (two or more) will be excluded. Papers that discuss variations of the same type of review (ie, differences in scoping reviews) will also be included, as will empirical studies (such as methodological studies or scoping reviews that investigate methodology) evaluating differences in review approaches. For inclusion, a report should describe at least two types of evidence synthesis and include a definition and/or distinguishing characteristics between the types.

Context

This review will not be limited by any context, setting, discipline, or field. All evidence synthesis classification schemes will be included across diverse fields, including, but not limited to, clinical sciences, public health, social sciences, environmental sciences, biosciences, engineering, policy, law, and education.

Types of sources

This scoping review will consider documents, such as discussion papers, commentaries, books, editorials, manuals, handbooks, formal guidance from major organizations, and other resources that describe different approaches to evidence synthesis and potentially provide a classification scheme of these different approaches, as well as approaches to categorization.

Methods

The proposed scoping review will be conducted in accordance with the JBI methodology for scoping reviews^21,22 and reported in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR).²³

This scoping review aligns with the guidance for conducting a scoping review when needing to examine key characteristics or factors related to a concept.²⁴ This project constitutes part of the Evidence Synthesis Taxonomy Initiative, which endeavors to develop an evidence synthesis taxonomy developed through collaboration with key stakeholders to establish an online, interactive, and scalable evidence synthesis taxonomy. The Evidence Synthesis Taxonomy Initiative is not formally associated with Evidence Synthesis International or the Global Evidence Synthesis Initiative (or other groups/organizations in this field) but aims to collaborate with all researchers, end users, and organizations in this space. This work is linked with the Unity Health Toronto Knowledge Translation Program’s Right Review Tool and will assist in updating the tool.²⁵

The details of this review project are available on Open Science Framework (https://osf.io/qwc27).

Engagement

The authors of this review are all working in the field of evidence synthesis and are invested in improving synthesis methodology to better support evidence-based health care and other fields. This scoping review will include engagement with other evidence synthesis researchers through an advisory panel overseen by an executive team. The advisory panel includes over 80 researchers, guideline developers, methodologists, health professionals, policy-makers, patient partners, and information scientists with an interest in methodology and evidence synthesis across various fields and disciplines. They have had the opportunity to review this protocol and offer feedback on the research questions, search strategies, information sources, and the data extraction form. The advisory and expert panel will continue to support the scoping review in the following ways:

• The advisory panel and executive team will continue to be updated about the project every month via email, and every quarter via a Zoom meeting. Members of the advisory and executive panel will be asked to provide feedback at each stage of the scoping review.
• The advisory panel and executive team will provide feedback on which databases or websites should be searched.
• The advisory and executive panels will be asked to assist in the identification of available typologies, taxonomies, and classification systems.
• The advisory panel and executive team will be presented with all findings that are identified from the scoping review.

Individuals who form part of the advisory panel and executive team, and who have made a substantial commitment that meets the International Committee of Medical Journal Editors (ICMJE) authorship requirements, will be eligible to be an author on the final report.

Search strategy

The search strategy will aim to locate both published and unpublished documents. The search strategy was developed with the help of an expert health librarian (CP) and will be peer-reviewed using the Peer Review of Electronic Search Strategies (PRESS) guideline statement.²⁶ An initial exploratory search of MEDLINE (Ovid) was undertaken to identify key articles on the topic. The terminology used within the articles was analyzed and used to develop a full search strategy for MEDLINE via Ovid (see Appendix I) and other selected sources. This search was tested to ensure it picked up key articles. The search strategy, including all identified keywords and index terms, will be adapted for each included database and/or information source using manual translation.²⁷ The reference lists of all included documents will be screened for additional studies using citationchaser (Zenodo, Geneva, Switzerland).²⁸ Onwards citation screening will not be performed due to the likelihood of high citation counts of seminal discussions and guidance papers. However, to supplement the search, we will also investigate the co-citations of seminal papers^1,12–14,29 on the topic and screen co-citations of these papers where there are two or more co-citations. The advisory panel will be asked whether they know of any papers that are relevant to this review topic or others in the field who may know of relevant papers.

There will be no exclusions based on language or publication status (ie, published, unpublished, in press, in progress, preprint). There are no date limitations. For documents in languages other than English, DeepL (DeepL, Cologne Germany) will be used to determine whether they meet the inclusion criteria. Where documents are published in a language other than English and meet inclusion criteria, DeepL translations will be reviewed by a person fluent in the language.

The databases to be searched include MEDLINE (Ovid), Embase, CINAHL with Full Text (EBSCO), ERIC (EBSCO), Scopus, Compendex, and JSTOR. Sources of unpublished studies/gray literature to be searched include Google, the Lens, and Dimensions. Hand-searching the websites of Cochrane, JBI, Agency for Healthcare Research and Quality, Collaboration for Environmental Evidence, Campbell, EPPI, and York University Centre for Reviews and Dissemination (CRD) will be implemented to identify any suitable resources. All search strategies will be submitted with the scoping review report as supplementary material.

Study selection

Following the search, all identified citations will be collated and uploaded into EndNote v.X20 (Clarivate Analytics, PA, USA) and duplicates removed. Data will then be imported into Covidence (Veritas Health Innovation, Melbourne, Australia). Prior to title and abstract screening, a pilot screening of 50 documents will occur between the reviewers, who will review each title and abstract independently. More piloting, discussions, and clarification to the screening guidance will occur if >70% agreement has not been observed. The title and abstracts will be screened independently by pairs of reviewers (so that each record is screened by at least two people) for assessment against the inclusion criteria for the review.

Potentially relevant documents will be retrieved in full and subjected to another pilot test of 10 full-text articles, and if >70% agreement is achieved, full-test screening will commence. The full text of selected citations will be assessed in detail against the inclusion criteria by two or more independent reviewers (so that each record is screened by at least two people) in Covidence. Reasons for exclusion of full-text documents that do not meet the inclusion criteria will be recorded and reported in the final review. Any disagreements that arise between the reviewers at each stage of the selection process will be resolved through discussion or with an additional reviewer. The results of the search and the study inclusion process will be reported in full in the final review and presented in a PRISMA flow diagram.

Data extraction

Using a standardized extraction form specifically designed for this review, we will conduct a pilot exercise using the same five to 10 full-text reports for the data extractors to calibrate and test the review form. Each document will be extracted by pairs of reviewers working independently (so that each document is extracted by at least two people) using a data extraction tool developed by the reviewers (Appendix II). Data will be extracted at two levels: i) the typology/classification system itself, and ii) the evidence synthesis approaches detailed in the taxonomy.

For the first level, where data are extracted about the classification system itself, the extraction will include the specific details about the document’s aim; field of research; type of evidence source (eg, commentary, manual/guidance); type of classification system (eg, typology, taxonomy); development approach for the classification system; type of evidence synthesis approaches included; and the facets through which different approaches have been classified.

For extraction at the individual synthesis approach level, where possible, the question framework and the indication and/or purpose of the review (such as their utility to decision-making) will be extracted and tabulated. Any definitions for review types will be extracted. Details regarding particular methods (such as risk of bias assessment approaches and data synthesis techniques) will also be extracted, along with links to formal guidance; whether there is a reporting standard; perceived strengths and weaknesses; and intended influence on policy, practice, and future research. Additionally, if any tools exist to appraise/assess the risk of bias of the synthesis approach itself or any approaches to establish certainty/confidence or the evidential quality are mentioned, these will be extracted.

The draft data extraction form may require modification following piloting on retrieved documents and during the process of extracting data from each included document. Modifications will be detailed in the final scoping review. Any disagreements that arise between the reviewers will be resolved through discussion or with an additional reviewer. If appropriate, authors of papers will be contacted to request missing or additional data.

Data analysis and presentation

All identified typologies/classification systems will be listed and described in detail. Elements included in each classification system will be presented in tables as per data extracted at the classification level. Additionally, the individual evidence synthesis approaches detailed within each classification system will be presented. These will then undergo descriptive analysis and be reported as frequencies. Evidence synthesis approaches that are similar (ie, effectiveness or intervention review) will not be combined, but rather reported and tabulated as individual approaches (thereby removing any subjectivity in coding and classifying evidence synthesis approaches at this stage). After listing all of the individual synthesis approaches, these will be grouped into clusters based on similarity in approach and purpose. Network diagrams will be developed where appropriate. Definitive selection of the review types and terminology will take place at a subsequent stage of the broader Evidence Synthesis Taxonomy Initiative and is beyond the scope of this review.

Acknowledgments

Members of the Evidence Synthesis Taxonomy Initiative for their review of this work.

Funding

ZM is supported by an NHMRC Investigator Grant, APP1195676.

ACT is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis.

BC is funded by a Health Research Board (HRB) Emerging Investigator Award (EIA-2019-09). The funders will have no role in the development or approval of content.

Author contributions

ZM conceived the topic of research, wrote the first draft, incorporated feedback, and finalized the manuscript. All other authors contributed to discussions and initial ideas for the manuscript, reviewed drafts, provided feedback, and approved the final draft.

Availability of data and materials

We will endeavor to make all data and code used within this project available. This will be available in the final published manuscript, the supplementary files, or on the Open Science Framework project space (https://osf.io/kep8c/). We will endeavor to answer any queries from readers and supply the forms they need, most likely in the form of MS Excel (Redmond, Washington, USA) spreadsheets.