Introduction
Evidence-based health care (EBHC) is the use of scientific evidence to guide clinical guidelines and health policy.1 First appearing in the 1990s, EBHC has since grown in recognition and importance. It is now championed by organizations such as JBI,2 Cochrane,3 and the World Health Organization, and has become the backbone of international health policy agendas.4,5 EBHC is reliant on high-quality scientific evidence; it is, therefore, critical that the ever-growing breadth of medical literature be assessed and synthesized for consumption by health workers and policy-makers.
A systematic review (SR) is a structured research approach that collates, critiques, and synthesizes all available evidence specific to a clearly defined research topic.6 SRs are a key pillar of EBHC and are essential to developing modern clinical guidelines and health policy. While clearly important for summarizing research and identifying bias, they also curtail unnecessary duplication of research and identify new areas for study.2
Quantitative evidence relies on research designs that generate numerical data.7 Effectiveness reviews synthesize quantitative evidence by exploring how well a treatment or intervention achieves its intended outcome.6,7 Effectiveness reviews can synthesize evidence from experimental, quasi-experimental, or observational studies, but ideally summarize results from randomized controlled trials (RCTs).7 The RCT is the gold standard for quantitative scientific study design; however, because RCT quality can be variable and new RCTs often contradict the results of old ones, no single RCT can or should be judged as sufficient evidence to inform decision-making alone.8 Effectiveness reviews circumvent these limitations by collating and synthesizing the quantitative evidence from a series of studies that focus on the same outcome.7
JBI is an international research organization dedicated to the promotion and development of the best available evidence to support health policy and practice.9 JBI has developed a transparent methodological approach to appropriately conduct SRs of effectiveness that comprise the universal components of a high-quality SR.10 However, as health professionals around the world become more reliant on the robust evidence provided by SRs, it is important that SRs themselves are evaluated and methodologically assessed to be high quality and free from bias.
The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist is a guideline for authors of SRs that outlines the minimum reporting standards for SRs and meta-analyses with the aim of improving the quality of reporting.11 An update to PRISMA 2009 was published in 2020, which aimed to reflect recent developments within SR methodology.11 Consequently, 15 of the 27 PRISMA 2009 checklist items were updated to include requirements for more thorough reporting.12 The PRISMA 2020 update will enable greater adherence to PRISMA guidelines and, therefore, improve the standard of reporting within SRs.12 Risk of bias within SRs is assessed using the Risk of Bias in Systematic Reviews (ROBIS) tool, which comprises 3 stages of assessment that include evaluating the relevance of the target question, addressing potential introduction of bias throughout the review process, and assessing the potential impact of bias on the review’s findings and conclusions.13 ROBIS is one of the best tools for assessing risk of bias, particularly for reviews of interventions, and its design was partly aimed at authors undertaking overviews of SRs.13 Reducing risk of bias is essential for maintaining standards of quality and reporting within SRs.14
Previous methodological reviews have examined the adherence of SRs to guidelines and tools to analyze SR reporting quality,5,15,16 risk of bias in SRs,17 and certainty of findings in SRs.18 Numerous studies have utilized ROBIS and referenced PRISMA 2009 to determine the quality of SRs.16,19 This review will follow a similar approach in order to highlight areas of poor reporting and methodological quality that can be targeted for future improvement. To date there have not been any studies assessing the adherence of SRs of effectiveness published in JBI Evidence Synthesis to the PRISMA guidelines. SRs are critical tools for informing clinical decisions and health policy; therefore, poorly constructed and reported SRs may negatively impact these areas.4 The purpose of this review is to assess JBI SRs of effectiveness for adherence to the PRISMA 2020 guidelines and for risk of bias using ROBIS in order to reflect on SR quality, identify shortfalls, and improve the robustness of future SRs.
Review objective
The objective of this review is to perform a methodological assessment of recent JBI reviews of effectiveness by assessing their quality against the PRISMA checklist and using the ROBIS tool.
Inclusion criteria
SRs of effectiveness published in JBI Evidence Synthesis will be eligible for inclusion in this methodological review. The 20 most recent SRs of effectiveness comprise an adequate sample size to evaluate JBI reviews’ adherence to PRISMA 2020. All eligible SRs of effectiveness published in JBI Evidence Synthesis since 2018 will be retrieved, and the 20 most recent from the date of the search will be included in this review. These reviews may have any number of studies included.
Methods
Search strategy
We will search MEDLINE (PubMed) for all studies published in JBI Evidence Synthesis (formerly, the JBI Database of Systematic Reviews and Implementation Reports) from 2018 to the present. MEDLINE was chosen for the search rather than a direct search of the journal due to MEDLINE’s robust searching and information collection capabilities. The search string used will be (“JBI Evidence Synthesis”[Journal]) OR (“JBI database of systematic reviews and implementation reports”[Journal]), and the search will be limited to studies published since 2018.
Study selection
All records retrieved will be imported into Covidence (Veritas Health Information, Melbourne, Australia). Two independent reviewers will screen all records by title and abstract to select SRs of effectiveness. If the record’s title or abstract is indicative of an effectiveness review, it will be included for screening at full text. If the review type is unclear from title and abstract alone, the record will still be included for screening at full text. All records included at the full-text stage will be screened by 2 independent reviewers. Any reasons for exclusion at the full-text stage will be documented in the final review. Any disagreements between the 2 reviewers about included studies will be resolved through discussion or with a third reviewer.
Data extraction
The data of interest in this study are those related to the quality of the review and the conduct of reporting, specifically those items outlined in the PRISMA 2020 checklist and the ROBIS tool. An MS Excel spreadsheet (Redmond, Washington, USA) has been designed for data extraction using questions from the PRISMA 2020 checklist and ROBIS tool (see Appendix I). Two reviewers will extract data from all studies independently and add them into separate spreadsheets. Any discrepancies will be discussed or referred to a third reviewer for a decision. A pilot data extraction on 3 initial studies will be performed to assess the robustness of the data extraction tool.
Assessment of reporting quality (PRISMA)
All included reviews will be assessed against each item on the PRISMA checklist. Assessors will use a “yes,” “no,” or “partial” response to indicate whether the review has adhered to the checklist. Reviewers will include comments explaining their decisions, where necessary.
Assessment of risk of bias (ROBIS)
For items in phase 1 of the ROBIS guideline, assessors will use a “yes,” “no,” or “partial” response. For items in phases 2 and 3, assessors will use a “yes,” “probably yes,” “probably no,” “no,” or “no information” response. At the end of each phase and domain, assessors will give an overall summary rating of concern (high, low, unclear), with comments to describe the rating. Assessors will utilize the official ROBIS guidance document to make an accurate judgment of the risk of bias.13
Additional information
Some descriptive details, such as year, field, and topic, will also be collected. Some additional questions relevant to the conduct of JBI reviews that are not included in PRISMA or ROBIS will be included in the spreadsheet; for example, whether a Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach was followed, and whether a GRADE Summary of Findings was produced. These questions will give us additional information on whether authors are following JBI-recommended practices that are not specifically mentioned in PRISMA of ROBIS.
Data synthesis
Data will be presented descriptively with tables and figures and synthesized narratively.
Funding
ZM is supported by an NHMRC Investigator Grant, APP1195676.
Appendix I: Data extraction tool
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)
The following items will be answered with yes, no, or partial response.
• Does the title identify the report as a systematic review?
• Does the study’s abstract conform to the PRISMA abstract guidelines?
Does the study:
1. Describe the rationale for the review in the context of existing knowledge?
2. Provide an explicit statement of the objective(s) or question(s) the review addresses?
3. Specify the inclusion and exclusion criteria for the review and how studies were grouped for the syntheses?
4. Specify all databases, registers, websites, organizations, reference lists, and other sources searched or consulted to identify studies? Specify the date when each source was last searched or consulted? (Additional information: What databases/registers were used?)
5. Present the full search strategies for all databases, registers, and websites, including any filters and limits used?
6. Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and, if applicable, details of automation tools used in the process. (Additional information: How many reviewers extracted data? What automation tools were used?)
7. Specify the methods used to collect data from reports, including how many reviewers collected data from each report, whether they worked independently, any processes for obtaining or confirming data from study investigators, and, if applicable, details of automation tools used in the process?
8. a) List and define all outcomes for which data were sought? Do they specify whether all results that were compatible with each outcome domain in each study were sought (eg, for all measures, time points, analyses), and if not, the methods used to decide which results to collect?
b) List and define all other variables for which data were sought (eg, participant and intervention characteristics, funding sources), including describing any assumptions made about any missing or unclear information?
9. Specify the methods used to assess risk of bias in the included studies, including details of the tool(s) used, how many reviewers assessed each study and whether they worked independently, and, if applicable, details of automation tools used in the process? (Additional information: What risk of bias tools were used? What automation tools were used?)
10. Specify for each outcome the effect measure(s) (eg, risk ratio, mean difference) used in the synthesis or presentation of results?
11. a) Describe the processes used to decide which studies were eligible for each synthesis (eg, tabulating the study intervention characteristics and comparing against the planned groups for each synthesis)?
b) Describe any methods required to prepare the data for presentation or synthesis, such as handling of missing summary statistics or data conversions?
c) Describe any methods used to tabulate or visually display results of individual studies and syntheses?
d) Describe any methods used to synthesize results and provide a rationale for the choice(s)? If meta-analysis was performed, do they describe the model(s), method(s) to identify the presence and extent of statistical heterogeneity, and software package(s) used?
e) Describe any methods used to explore possible causes of heterogeneity among study results (eg, subgroup analysis, meta-regression)?
f) Describe any sensitivity analyses conducted to assess robustness of the synthesized results?
12. Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases)?
13. Describe any methods used to assess certainty (or confidence) in the body of evidence for an outcome?
14. a) Describe the results of the search and selection process, from the number of records identified in the search to the number of studies included in the review, ideally using a flow diagram?
b) Cite studies that might appear to meet the inclusion criteria, but which were excluded, and explain why they were excluded?
15. Cite each included study and present its characteristics?
16. Present assessments of risk of bias for each included study?
17. For all outcomes present for each study:
a) summary statistics for each group (where appropriate)
b) an effect estimate and its precision (eg, confidence/credible interval), ideally using structured tables or plots?
18. a) For each synthesis, briefly summarize the characteristics and risk of bias among contributing studies?
b) Present results of all statistical syntheses conducted? If meta-analysis was done, present for each the summary estimate and its precision (eg, confidence/credible interval) and measures of statistical heterogeneity? If comparing groups, do they describe the direction of the effect?
c) Present results of all investigations of possible causes of heterogeneity among study results?
d) Present results of all sensitivity analyses conducted to assess the robustness of the synthesized results?
19. Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed?
20. Present assessments of certainty (or confidence) in the body of evidence for each outcome assessed?
21. a) Provide a general interpretation of the results in the context of other evidence?
b) Discuss any limitations of the evidence included in the review?
c) Discuss any limitations of the review processes used?
d) Discuss implications of the results for practice, policy, and future research?
22. a) Provide registration information for the review, including register name and registration number, or state that the review was not registered? (Additional information: Was the review registered? If so, where?)
b) Indicate where the review protocol can be accessed, or state that a protocol was not prepared? (Additional information: Was a protocol prepared?)
c) Describe and explain any amendments to information provided at registration or in the protocol?
23. Describe sources of financial or non-financial support for the review, and the role of the funders or sponsors in the review?
24. Declare any competing interests of review authors?
25. Report which of the following are publicly available and where they can be found: template data collection forms; data extracted from included studies; data used for all analyses; analytic code; any other materials used in the review?
Risk of Bias in Systematic Reviews (ROBIS)
The following questions will be answered with yes, probably yes, probably no, no, or no information, except where otherwise specified.
Phase 1: Relevance assessment
1. Does the question addressed by the review match the question you are trying to answer (eg, in your overview or guideline)? (Answer: yes/no/partial)
Phase 2: Identifying concerns with the review process
Domain 1: Study eligibility
1.1 Did the review adhere to predefined objectives and eligibility criteria?
1.2 Were the eligibility criteria appropriate for the review question?
1.3 Were eligibility criteria unambiguous?
1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate?
1.5 Were any restrictions in eligibility criteria based on sources of information appropriate?
Concerns regarding specification of study eligibility criteria (domain 1): (answer: low/high/unclear)
Domain 2: Identification and selection of studies
2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
2.5 Were efforts made to minimize errors in selection of studies?
Concerns regarding methods used to identify and/or select studies (domain 2): (answer: low/high/unclear)
Domain 3: Data collection and study appraisal
3.1 Were efforts made to minimize error in data collection?
3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?
3.3 Were all relevant study results collected for use in the synthesis?
3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria?
3.5 Were efforts made to minimize error in risk of bias assessment?
Concerns regarding methods used to collect data and appraise studies (domain 3): (answer: low/high/ unclear)
Domain 4: Synthesis and findings
4.1 Did the synthesis include all studies that it should?
4.2 Were all predefined analyses followed or departures explained?
4.3 Was the synthesis appropriate given the nature and similarity in the research questions, study designs, and outcomes across included studies?
4.4 Was between-studies variation (heterogeneity) minimal or addressed in the synthesis?
4.5 Were the findings robust (eg, as demonstrated through funnel plot or sensitivity analyses)?
4.6 Were biases in primary studies minimal or addressed in the synthesis?
Concerns regarding methods used to synthesize results (domain 4): (answer: low/high/unclear)
Phase 3: Judging risk of bias
1a) Did the interpretation of findings address all of the concerns identified in the phase 2 assessment?
1b) Was the relevance of identified studies to the review’s research question appropriately considered?
1c) Did the reviewers avoid emphasizing results on the basis of their statistical significance?
Risk of bias introduced by methods used to identify and/or select studies: (answer: low/high/unclear)
Additional data
What was the clinical discipline?
What was the topic area?
Was a recommendation for practice made? • Was a grade assigned for any recommendation for practice made? • If so, what grade was used? • Was the recommendation actionable (ie, clear action/target listed)?
Was a recommendation for research made?
What study designs were included in the review?
Was the GRADE approach followed?
Was a GRADE Summary of Findings produced?
Is there a PRISMA checklist supporting the published manuscript?
What data synthesis approaches were used (eg, meta-analysis/SWIM/narrative)?
References
1. Pearson A, Wiechula R, Court A, Lockwood C. The JBI model of evidence-based healthcare. Int J Evid Based Healthc 2005;3(8):207–15.
2. Jordan Z, Lockwood C, Munn Z, Aromataris E. The updated Joanna Briggs Institute Model of Evidence-Based Healthcare. Int J Evid Based Healthc 2019;17(1):58–71.
3. Shah HM, Chung KC. Archie Cochrane and his vision for evidence-based medicine. Plast Reconstr Surg 2009;124(3):982–8.
4. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet 2017;390(10092):415–23.
5. Xiu-xia L, Ya Z, Yao-long C, Ke-hu Y, Zong-jiu Z. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy 2015;119(4):503–10.
6. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of
systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol 2018;18(1):5.
7. Tufanaru C, Munn Z, Aromataris E, Campbell J, Hopp L Chapter 3: Systematic reviews of effectiveness. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis [internet]. Adelaide, JBI; 2020 [cited 2022 May 26]. Available from:
https://synthesismanual.jbi.global.
8. Brignardello-Petersen R, Ioannidis JPA, Tomlinson G, Guyatt G Guyatt G, Rennie D, Meade MO, Cook DJ. Chapter 11.2: Surprising results of randomized trials. Users’ guide to the medical literature: a manual for evidence-based clinical practice. 3rd edn. New York: McGraw-Hill; 2015.
9. Jordan Z, Lockwood C, Aromataris E, Pilla B, Porritt K, Klugar M, et al. JBI series paper 1: Introducing JBI and the JBI Model of EHBC. J Clin Epidemiol 2022;150:191–5.
10. Aromataris E, Munn Z. JBI Manual for Evidence Synthesis [internet]. Adelaide, JBI; 2020 [cited 2022 May 26]. Available from:
https://synthesismanual.jbi.global.
11. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol 2021;134:103–12.
12. Brennan SE, Munn Z. PRISMA 2020: a reporting guideline for the next generation of systematic reviews. JBI Evid Synth 2021;19(5):906–8.
13. Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225–34; 14.
14. Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al.
Systematic review adherence to methodological or reporting quality. Syst Rev 2017;6(1):131.
15. Munn Z, Dias M, Tufanaru C, Porritt K, Stern C, Jordan Z, et al. The “quality” of JBI qualitative research synthesis: a methodological investigation into the adherence of meta-aggregative systematic reviews to reporting standards and methodological guidance. JBI Evid Synth 2021;19(5):1119–39.
16. Campbell JM, Kavanagh S, Kurmis R, Munn Z. Systematic reviews in burns care: poor quality and getting worse. J Burn Care Res 2017;38(2):e552–e67.
17. Perry R, Whitmarsh A, Leach V, Davies P. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev 2021;10(1):273.
18. Barker TH, Dias M, Stern C, Porritt K, Wiechula R, Aromataris E, et al. Guidelines rarely used GRADE and applied methods inconsistently: a methodological study of Australian guidelines. J Clin Epidemiol 2021;130:125–34.
19. Deshpande S, Misso K, Westwood M, Stirk L, de Kock S, Kleijnen J, et al. Not all Cochrane reviews are good quality systematic reviews. Value Health 2016;19(7):A371.
