Revising the JBI quantitative critical appraisal tools to improve their applicability: an overview of methods and the development process

Abstract

JBI offers a suite of critical appraisal instruments that are freely available to systematic reviewers and researchers investigating the methodological limitations of primary research studies. The JBI instruments are designed to be study-specific and are presented as questions in a checklist. The JBI instruments have existed in a checklist-style format for approximately 20 years; however, as the field of research synthesis expands, many of the tools offered by JBI have become outdated. The JBI critical appraisal tools for quantitative studies (eg, randomized controlled trials, quasi-experimental studies) must be updated to reflect the current methodologies in this field. Cognizant of this and the recent developments in risk-of-bias science, the JBI Effectiveness Methodology Group was tasked with updating the current quantitative critical appraisal instruments. This paper details the methods and rationale that the JBI Effectiveness Methodology Group followed when updating the JBI critical appraisal instruments for quantitative study designs. We detail the key changes made to the tools and highlight how these changes reflect current methodological developments in this field.

Introduction

Critical appraisal (or risk of bias assessment) of included studies is an essential undertaking for any trustworthy systematic review.¹ JBI offers a suite of critical appraisal tools that are freely available to anyone aiming to conduct a systematic review or critically appraise the literature.^2,3 These checklists are predominantly based on study design, with different checklists for the most common study designs likely to be encountered by systematic reviewers. These include checklists specific to quantitative data, such as randomized controlled trials (RCTs),⁴ and extends to case series⁵ and studies reporting prevalence data.⁶ There are even checklists for data specific to qualitative research⁷ and text and opinion.⁸

Historically, the JBI critical appraisal tools have been used as either a checklist or a scale for appraising literature, with the former approach being more commonly reported by JBI systematic reviews. As a checklist, users are asked to consider whether a study meets a predetermined set of questions (or criteria) stipulated within the tool. These questions are answered with a response of “yes,” “no,” “unclear,” or “not applicable.” For example, one question in the JBI checklist for RCT reads: “Was true randomization used for assignment of participants to treatment groups?” A user can answer this question with “yes” (ie, criterion met), “no” (ie, criterion unmet), “unclear,” or “not applicable.” How the results of this appraisal are considered in the review process is left largely to the review team, with little oversight or instruction provided on which approach is preferable.⁴

An example of a scale-based approach is the Downs and Black Checklist,⁹ which is structured for users to provide quantitative scores (ranging from 0 to 2) for each item presented, based on which safeguards the study has implemented to minimize risk of bias. The item scores of that study are then tallied to provide an overall quality score. Due to its robust design, this methodology can also be applied to the JBI critical appraisal instruments if authors so choose. Scale-based and checklist approaches differ from the newly reworked risk-of-bias tool (RoB 2) available from the Cochrane Collaboration, which follows a domain-based approach.¹⁰ Judgments on risk of bias using this tool are made across specific domains of bias, and an overall judgment and assessment of the risk of bias of each outcome for a study is made using qualitative judgments of value.

The JBI checklists have existed in their current format for approximately 20 years.¹¹ Anecdotal evidence suggests that the tools are popular among both newer and experienced reviewers, especially those conducting JBI systematic reviews. However, over time, limitations with the tools have been identified. Specifically, when considering the recent advancements in risk of bias assessment over the past few years,^10,12,13 one can reasonably argue that they are outdated. For example, it is now considered best practice in the evidence synthesis field to consider bias at the outcome and result levels,¹⁴ a concept that is not intuitive to the current design of the checklists, which were created to appraise at the study level. Additionally, it is not currently practicable for authors to map questions to relevant domains of bias owing to the current design of the tools.

The aim of this work was to revise the existing JBI checklists to improve their longevity and usefulness to reflect current advancements made in this space,^10,12,13 while adhering to the reporting and methodological requirements as established by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020¹⁵ and the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach.¹⁶ To address these (and other) concerns, the JBI Scientific Committee tasked a small working group, consisting of members of the JBI Effectiveness Methodology Group (EMG) to update the suite of JBI critical appraisal checklists for quantitative analytical study designs.¹⁷ This involved 3 specific objectives, which were to:

1. Review all current tools and categorize current checklist questions into bias domains and other validity constructs so that a risk of bias assessment can occur for each domain of bias if desired, including nuanced guidance regarding whether safeguard implementation was feasible (eg, blinding feasibility for hard outcomes). Checklists will then be able to be used as critical appraisal tools that also consider bias domains and can thus be used as risk of bias assessment tools as well.
2. Review all current tools for analytical quantitative study designs and move to focus only on internal validity, or “risk of bias” assessment rather than other issues related to reporting, external validity, imprecision, etc. These items can be removed from the tools or at least clearly separated out from internal validity questions.
3. Enable risk of bias assessments to be carried out at the result or outcome level (as opposed to the study level).

Additionally, it was agreed that this update would involve no changes to the wording of any questions currently presented in the existing checklists. Although change in the guidance on how to answer these questions and their positioning was accepted by the JBI Scientific Committee to reflect updates in the risk-of-bias science, the questions themselves should remain verbatim given their widespread use and acceptance by the synthesis community. While alternative tools with different structures containing different questions may be considered for the future, this was beyond the focus of this project.

Consideration of validity constructs

The JBI EMG began the update process by first cataloging the questions asked in each checklist for study designs that employ quantitative data. These included the following checklists: RCTs, quasi-experimental studies, cohort studies, analytical cross-sectional studies, case-control studies, case series, and prevalence studies.² As part of the objectives of this project, the JBI EMG wanted to clearly indicate that the assessment of risk of bias was aligned to the internal validity of that study. Initially, the intention was to highlight the questions regarding internal validity in the existing checklists and separate, or at least clearly differentiate, these questions from others. However, on review of the cataloged questions, it became apparent that several touched on concepts such as reporting quality and statistical conclusion validity. As a result, the JBI EMG, through a series of discussions, agreed on common constructs of validity that can be relevant when reviewing literature.

While only the construct of internal validity remains relevant to the assessment of a study’s risk of bias, it was agreed that 3 additional constructs were representative of these additional concepts. These additional constructs included statistical conclusion validity, comprehensiveness of reporting, and external validity. This categorization of questions to constructs helps users to clearly identify which questions are relevant to establish the risk of bias of an appraised study and, therefore, facilitate domain-based or construct-based judgments. Additionally, users will be able to identify, for example, whether that study is transparently reported and separate these judgments from those involving risk of bias.

Internal validity

Internal validity is whether the study is free from systematic error, also called bias. It is assessed to determine whether the results presented are likely to be “true.”^18-20 “Truth” is defined as how well a causal relationship between intervention/exposure and outcome can be inferred from the findings of the study. For example, an internally valid RCT implies that the differences observed between groups receiving different interventions are (apart from random error) attributable to the intervention under investigation.

Statistical conclusion validity

Statistical conclusion validity is achieved when the conclusions drawn from the results of a study are founded on adequate analysis of the data.^21,22 However, statistical conclusion validity is only related to the appropriateness (or not) of all statistical analyses within a paper, and not necessarily the inputs into that analysis. For example, if a study has failed to appropriately control for a confounder in the analysis, this would impact the internal validity of that study and not the statistical conclusion validity.²³ Rather, statistical conclusion validity refers to how likely a link (or lack thereof) can be established between the variable of interest, and includes the following concepts^21-24:

1. Was the study sufficiently powered to detect an effect if it exists?
2. Is there a risk that the statistical analyses conducted by study will reveal an effect that does not actually exist (ie, have the assumptions of the statistical tests performed been violated; have multiple statistical tests been performed on the same variables under different assumptions to fish for statistical significance?)?
3. Has the magnitude of effect been confidently estimated (ie, is the effect measure reliable and is the size of the effect correct; is there any heterogeneity of units?)?²³

While these questions are sometimes less relevant within the context of a systematic review (as the systematic review authors may address issues with primary analyses in their own extraction and synthesis) and are unrelated to the risk of bias of that study, they can still be useful for a reviewer to consider.

Comprehensiveness of reporting

Comprehensiveness of reporting is related to the reporting quality of a study. Poor quality of reporting may make the assessment of internal validity difficult; however, the relationship between the internal validity (and external validity) of a study and the overall quality of its reporting is still uncertain and potentially variable.^12,25,26 Factors taken into consideration when assessing the reporting quality of a study include the following:

1. Have the confounding factors been identified and described by the authors? If confounding factors have been incorrectly identified, then there is potential that this may impact on the internal validity of that study. Thus, it is important that confounders are identified and described in such a manner that is made clear to the readers of that paper.
2. Was an ethics statement required for this research, and if so, has it been provided? Most research requires approval from an appropriate research ethics committee. These committees monitor the progress of research to ensure that ethical issues have been considered at each stage of the study. As a reviewer, it may be important to determine that the primary researchers have not placed any unethical demands on the research participants.²⁷
3. Did the authors transparently report all potential conflicts of interest in their study? If the conflicts of interest were likely to influence the design, conduct, analysis, or selection of results reported, this would impact the internal validity of the study. Regarding comprehensiveness of reporting, it is important that the reader can transparently determine the conflicts of interest for each author, the role of the funding body, and in what capacity these factors influenced the study.

External validity

External validity refers to the extent to which the results of the study can be generalized to groups, populations, and contexts that did not participate in the study.^19,20,28 An externally valid study allows for greater confidence in the findings when considering whether they are applicable to other populations. An internally valid study does not automatically imply external validity. However, while this construct of validity was agreed to be relevant, no existing question in any JBI checklist could be categorized as external validity.

Constructs such as external validity, statistical conclusion validity, and comprehensiveness of reporting are often of limited importance when considering a study as part of a broader systematic review project. This is despite items related to these constructs often appearing in previous tools marketed as tools to assess risk of bias or methodological quality, including the JBI critical appraisal checklists.¹² However, under this revision process, no question from the existing checklist was to be added, modified, or removed; therefore, these constructs have been included, but the revised checklists clearly identify and separate the questions for these 3 constructs from those related to internal validity and risk of bias.

The categories created and the broad reasoning for question categorization are summarized in Table 1 and described in more detail in the following section.

Table 1 - Validity constructs and example questions for assessing the risk of bias in quantitative studies

Construct	Issue/question
Internal validity	Was the question related to the extent the results of the study were free from systematic error or bias (eg, Was true randomization used for assignment of participants to treatment groups?)?
Statistical conclusion validity	Was the question related to whether the conclusions made regarding the relationships between analyzed variables reasonable (eg, Was appropriate statistical analysis used?)?
Comprehensiveness of reporting	Was the question related to the readability, transparency, or consistency of the writing of the study results (eg, Were the study subjects and the setting described in detail?)?
External validity	Was the question related to the extent the results of that study can be generalized to groups, populations, or contexts that did not participate in the study? (Although this domain has been included in this categorization exercise, no current question in the JBI checklists fit this category.)

Facilitating a domain-based approach to risk of bias assessment

Following categorization of the questions to 1 of the 4 constructs of validity, all members of the JBI EMG examined the internal validity construct to investigate the specific domains of bias that each question may relate to. This exercise was performed using the Cochrane RoB 2 tool¹⁰ as an exemplar. This tool was chosen initially due to how recently it had been redeveloped (2019). Each question of every checklist was compared with a similar question that existed on this tool to assign it to a specific domain. For example, question 1 for the checklist for RCTs (“Was true randomization used for assignment of participants to treatment groups?”) mapped to the Cochrane RoB 2 domain of random sequence generation. Each question from the checklist for RCTs mapped to the domains of bias as specified by the Cochrane RoB 2 tool. However, this was not the case for the JBI analytical tools for quasi-experimental and observational studies. For example, question 2 from the JBI checklist for cohort studies (“Were the exposures measured similarly to assign people to both the exposed and unexposed group?”) was not adequately represented in the Cochrane RoB 2 tool. As such, a new procedure was required.

In addition to mapping each question from the specified checklists to the Cochrane RoB 2 tool, the group also mapped the questions to further peer-reviewed, validated risk-of-bias instruments. These included the Cochrane risk-of-bias (RoB 1) tool,²⁹ the risk of bias in non-randomized studies of interventions (ROBINS-I) tool,³⁰ the risk of bias in non-randomized studies of exposures (ROBINS-E) tool,^31,32 and the MASTER Standard scale.¹² Following this procedure, and using the example highlighted previously, the question “Was true randomization used for assignment of participants to treatment groups?” was mapped to the following domains: random sequence generation (RoB 1), bias arising from the randomization process (RoB 2), bias in selection of participants into the study (ROBINS-I and ROBINS-E), and equal recruitment (MASTER Standard scale). A complete list of this mapping exercise for each question of every checklist is provided in Appendices I to VII.

Following this, the members of the JBI EMG created a unifying nomenclature for the domains of bias to which each question was assigned. The group could not use the naming scheme of any tool, because no tool could be applied consistently across the different checklists offered by JBI. Therefore, the group created 8 new terms to define and characterize the domains of bias that were addressed in the JBI tools. The newly created terminology for these domains include bias related to selection and allocation; administration of the intervention/exposure; assessment, detection, and measurement of the outcome; participant retention; temporal precedence; classification of the exposure; confounding factors; and selective reporting and/or publication bias.

Bias related to selection and allocation

Bias related to selection may also be termed “selection bias” or “bias from the randomization procedure.” These terms refer to the phenomena of systematic differences between the baseline characteristics for each group compared. Biases related to selection can arise during the allocation of participants to groups and can be minimized through appropriate randomization and concealment of participant allocation. Randomization ensures that every participant has an equal chance of being selected for any group. Allocation concealment refers to securing a strict implementation of the schedule of random assignments by preventing foreknowledge of the forthcoming allocations.³³ Biases related to selection and allocation are also endemic in non-randomized studies where participants are sourced from a sample population, and the concern is largely whether the exposure–outcome relationship is the same as would be found under experimental conditions, where randomization can be implemented.³⁴ In these studies, the effects of these biases can be minimized by ensuring that participants in both groups are sampled from the same reference population and not selected on a common effect (eg, hospitalization).

Bias related to administration of intervention/exposure

Bias related to administration of intervention/exposure may also be termed “performance bias” or “deviations from the intended intervention.” Bias related to administration of intervention/exposure refers to systematic differences between groups in the care provided or in exposure to factors that exist beyond the scope of the study. In experimental studies, biases related to administration of intervention/exposure can arise if the researchers or participants (or both) have foreknowledge of treatment assignment. These biases can also be minimized in controlled studies by ensuring that both treatment groups are treated identically aside from the intervention of interest. This is often achieved through blinding the researchers or participants (or both) to their assigned treatment groups. Blinding can involve the use of placebo or sham treatments, where neither the researcher nor the participant has foreknowledge as to what group they are in or what treatment they are receiving.¹ In observational studies, blinding of the observers or reporters of records can often be achieved to ensure that evaluation of the exposure of interest remains impartial and consistent for all records.³¹

Bias related to assessment, detection, and measurement of the outcome

Bias related to assessment, detection, and measurement of the outcome may also be termed “detection bias” or “measurement bias” and refers to systematic differences between groups in how outcomes have been determined in both experimental and observational studies. These biases can arise if the outcome assessors have foreknowledge of treatment assignment. These biases can also occur if the assessor evaluates an outcome differently for patients depending on their group assignment. The effect of these biases to internal validity can be controlled by blinding the outcome assessors to which treatment group the patients have been assigned, which can be especially important for subjective outcomes.¹⁰ Additional statistical phenomena can also be considered here and are often more likely to be encountered when assessing observational studies. Some of these include regression to the mean, maturation, ability to recall, and the Hawthorne effect, among many others.

Bias related to participant retention

Bias related to participant retention may also be termed “attrition bias” or “bias due to missing outcome data.” It refers to systematic differences between groups in the number and nature of withdrawals from a study. Biases related to retention can arise if the researchers have not transparently reported in the manuscript the details and reasons for participant withdrawal. The effects of biases related to retention can be informed by accurate and complete reporting of losses and reasons for withdrawal, and any strategies to address missing data.³⁵

Bias related to temporal precedence

Bias related to temporal precedence may arise in research when there is not a clearly prospective sequence of exposure to outcome in which a causal relationship can be inferred.¹² Because of the design of some studies (eg, cross-sectional studies), variables may be examined at a single point in time and, as a result, causality is not able to be determined.

Bias related to classification of the exposure

Bias related to classification of the exposure may arise if the status of an exposure of interest has not been appropriately defined.³⁶ This can be of particular importance when exposure to a certain condition serves as inclusion criteria for participants’ acceptance into a study. When investigating biases related to the classification of exposures, reviewers need to consider the relationship between the exposure status and the outcome, and whether classification of the exposure may have been influenced by knowledge of the outcome, or vice versa. For example, when performing a case-control study, participants are selected based on the presence of an outcome. Participants are classified as exposed or unexposed based on their histories. If we were interested in correlating the outcome of oral cancer and drinking coffee, reviewers can classify some people as exposed (ie, drank coffee) and some as unexposed. Some participants may overstate or understate their pervious exposure status (ie, how much coffee they drank). The researchers similarly may not have defined a cut-off to classify someone as exposed or unexposed (eg, how much and for how long did people need to drink coffee in order to be classified as “exposed”). Both of these examples can introduce bias into how the exposure is classified.

Bias related to confounding factors

Bias related to confounding factors may arise when known or unknown study factors exist in the causal pathway between the dependent and independent variables of a study.³⁷ These study factors (confounders) are typically participant characteristics or demographic factors. Positive confounding may create spurious associations between the variables of interest. Negative confounding can mask potentially real associations between the variables of interest or underestimate the magnitude of these associations. When investigating biases related to confounding, reviewers can consider whether the study design and conduct appropriately identified and controlled for potential confounders between comparable groups (eg, appropriate randomization procedure).³⁸

Bias related to selective reporting

Biases related to selective reporting refer to systematic differences between the reported findings of a study and the unreported findings of a study. These biases may arise if the researchers do not follow an a priori protocol or trial registration or if they only report a few of the outcomes in which data were collected. Researchers can minimize reporting bias by registering their study in a trial or study registry, predefining the outcomes of interest for their research, and fully reporting the results for each of these predefined outcomes.³⁹

While this domain has been included in this framework, no current question in any JBI checklists have been categorized to this domain. It has remained to facilitate the development of new questions (if required) that may belong to this domain in future renditions of these checklists.

Judgments made at the study, outcomes, and results level

With the questions categorized to both a construct of validity and (if categorized to the construct of internal validity) a domain of bias, the final process involved sorting questions based on the level in which risk of bias can be assessed. Most recent tools, including the RoB 2, ROBINS-I, and ROBINS-E, facilitate assessments of risk of bias at multiple levels. These include the study level, the outcome level, and the result level, and, importantly, different questions can be answered at different levels.¹⁴

For example, returning to question 1 for the checklist for RCTs (“Was true randomization used for the assignment of participants to treatment groups?”), this question can only be answered at the study level, because the act of randomization is going to hold true for each outcome measured for that population. Question 7 in the same checklist (“Were outcome assessors blind to treatment assignment?”) is an example of a question that can be answered at the outcome level. Because multiple outcomes are often assessed on 1 participant of a study, it may be that some outcomes were blinded from the participant (eg, objective outcomes, such as blood pressure), whereas some outcomes may not be blinded (eg, subjective outcomes, such as self-reported quality of life). Finally, question 12 in the checklist (“Was appropriate statistical analysis used?”) can be answered at a result level.

Conclusion

The work documented here has aimed to update, simplify, and provide greater flexibility to users of the JBI critical appraisal checklists. It is expected that this work will increase the usability and applicability of these checklists, particularly in line with modern advances in evidence synthesis, such as reporting according to PRISMA 2020,¹⁵ the focus on internal validity, and establishing certainty in the evidence using GRADE.¹⁶ This document introduces readers to the methods and process of updating these tools. The updated versions of these tools will be available separately in subsequent papers in this series.

Funding

MK is supported by the INTER-EXCELLENCE grant number LTC20031—Towards an International Network for Evidence-based Research in Clinical Health Research in the Czech Republic. ZM is supported by an NHMRC Investigator Grant, APP1195676.

Acknowledgments

Coauthor Catalin Tufanaru passed away July 29, 2021.