From critical appraisal to risk of bias assessment: clarifying the terminology for study evaluation in JBI systematic reviews

Abstract

The foundations for critical appraisal of literature have largely progressed through the development of epidemiologic research methods and the use of research to inform medical teaching and practice. This practical application of research is referred to as evidence-based medicine and has delivered a standard for the health care profession where clinicians are equally as engaged in conducting scientific research as they are in the practice of delivering treatments. Evidence-based medicine, now referred to as evidence-based health care, has generally been operationalized through empirically supported treatments, whereby the choice of treatments is substantiated by scientific support, usually by means of an evidence synthesis. As evidence synthesis methodology has advanced, guidance for the critical appraisal of primary research has emphasized a distinction from the assessment of internal validity required for synthesized research. This assessment is conceptualized and branded in various ways in the literature, such as risk of bias, critical appraisal, study validity, methodological quality, and methodological limitations. This paper provides a discussion of the definitions and characteristics of these terms, concluding with a recommendation for JBI to adopt the term “risk of bias” assessment.

Introduction

Not all research studies are created equal. Decisions made during the conceptualization, design, conduct, and analysis of a research study can impact the quality of the study and the trustworthiness of its results. Identifying study limitations is of primary importance in ensuring that practice decisions incorporate the highest quality evidence so that the best outcomes are achieved for patients. In systematic reviews, guidelines, and health technology assessments, it is common to perform a critical assessment of potential limitations of a study. This assessment is often given different terms in the literature, which are sometimes used interchangeably but may be conceptualized differently, such as “risk of bias,” “critical appraisal,” “study validity,” “methodological quality,” and “methodological limitations.”^1–7 The terminology used to describe the assessment of study methodology has a long history grounded in medical education and practice, and reflects the different approaches and concepts surrounding the appropriate assessment of primary studies for use in synthesized research.

In this paper, we discuss the evolution of this process and the nomenclature used, aiming to bring some clarity to the discussion. Recent advances in the methodological literature and development of new tools have led to a change in direction at JBI and prompted the JBI Effectiveness Methodology Group to consider the way forward for critical appraisal of JBI systematic reviews of effectiveness.

Critical appraisal

Guidelines for the critical appraisal of primary research emerged in the 1980s with the rise of evidence-based medicine (EBM).^8–11 EBM is an approach to medical education and practice that is achieved through consultation of scientific research alongside clinical judgment and the consideration of patient values to assist clinical decision-making in a particular context.^8,9 One of the steps embedded within the EBM approach is the critical appraisal of research studies following a comprehensive search of the literature and analysis of the relevance of retrieved studies.¹² Critical appraisal was proposed by Sackett et al.¹¹ as a way to systematically evaluate the medical literature for its trustworthiness, value, and relevance within a particular context.^13,14 Accordingly, the defining characteristics of a critical appraisal are whether i) the study addresses a clearly focused question; ii) the study used valid methods to address the research question; iii) the valid results of the study are important; and iv) these valid, important results are applicable to the target patient or population.¹⁵ These criteria are the basis for the development of several structured approaches to critical appraisal supported by use of critical appraisal tools.

The Critical Appraisal Skills Program (CASP) and the Centre for Evidence Based Medicine (CEBM) were established in the 1990s in response to the need to integrate science into clinical education and practice.^16,17 These collaborations paved the way for the development of educational programs addressing the need for EBM, in which critical appraisal methods form a central part. The CASP and CEBM critical appraisal tools systematically assess the quality and utility of different research designs based on the critical appraisal criteria developed by Sackett et al.^11,12 Aligning studies with these criteria has the potential to influence study results and the way conclusions of a study are interpreted by research consumers.¹⁸ Therefore, the use of critical appraisal tools is crucial to ascertain whether the results of the study can be believed, interpreted, and transferred appropriately into other contexts, such as policy, further research, education, and clinical practice.¹⁸ There are now more than 100 critical appraisal tools available for use that contain a variety of items to evaluate study methodology; however, there is no consensus regarding the best tool for any particular research context.¹⁸ JBI has also developed a suite of critical appraisal tools for each study design, with targeted questions addressing the methodological characteristics of a study.³

While critical appraisal includes a process to assess the internal validity of a study (criterion ii),¹⁵ the original conception of critical appraisal and tools by groups such as CASP, CEBM, and JBI sometimes goes beyond internal validity into assessment of external validity, statistical conclusion validity, generalizability, and even reporting quality (see Appendix I for a glossary of terms). As such, we can view the term “critical appraisal” as a broad term that may cross into fields where other tools or processes exist, such as assessment of the certainty of review findings (eg, using the Grading of Recommendations, Assessment, Development and Evaluation [GRADE] assessment)¹⁹ or compliance with reporting standards.²⁰ Also important is that critical appraisal questions focusing on the applicability, transferability, or statistical conclusion validity of a particular result from an individual study may not be as relevant in a systematic review process where these judgments are made on a synthesized result. As such, many current critical appraisal tools may be better used to create critically appraised topics, in journal clubs, or as an educational tool.

Methodological quality

Perhaps an even broader definition of the assessment of study methodology is encapsulated by the term “quality” when used in clinical education and practice. Quality can be characterized as the assessment of study validity as well as other indicators that improve the overall standard of the published research, and tends to answer the question of whether the authors of a published research article have done their best in terms of the methodologies they employed. However, current reflections of the characteristics of critical appraisal and methodological quality have identified a broader scope than is required for the assessment of study validity for use in evidence synthesis.⁵ This has led to different ideas among research consumers and methodologists regarding how quality should be conceptualized when developing tools and evaluating primary studies for the specific purpose of synthesizing research.

Some methodological quality assessment tools have historically gone beyond the remit of methods and items related to the protection of a study’s internal validity to include external validity; other methodological components, such as sample size and power; as well as reporting quality and ethical considerations.^5,21 The inclusion of these various constructs in a tool depends largely on the source of the guidance in creating such an assessment (eg, the research group and the tools’ intended use), researcher knowledge and expertise in developing this assessment, and the theoretical framework underlying the concept of methodological quality.²² For example, unrelated items regarding sample size (random error) may “slip in” to an assessment for internal validity due to a lack of researcher expertise. This lack of clarity has been problematic for the assessment of primary research, particularly for evidence synthesis, because what is required for evidence synthesis is to summarize a body of research and understand how reliable the results of the included studies are with respect to the pooled estimate of effect.⁵ Therefore, in this context, measurement of the potential for systematic error in the results of the included studies is needed.^5,23 When used in evidence synthesis, methodological quality assessment tools should solely measure the implementation of internal validity safeguards that protect against systematic error (bias) during the design, conduct, and analysis of the study.²³ However, despite this understanding, the inclusion of unrelated constructs in methodological quality assessment tools has persisted. Importantly, we should distinguish between the concepts of “study quality” (which can include additional considerations to internal validity) and “methodological quality” (which is focused only on the internal validity of a study).

Risk of bias

To overcome this problem, in 2008, the Cochrane Collaboration advocated for broad use of the term “risk of bias” in place of “methodological quality” in order to emphasize the link to systematic error and overcome the ambiguity between quality of the research and other constructs, particularly reporting quality, random error, and external validity.^22,24 Similar recommendations have been made in environmental evidence synthesis research²⁵ by the Collaboration of Environmental Evidence, and in preclinical research synthesis by the Systematic Review Centre for Laboratory Animal Research (SYRCLE),²⁶ which has resulted in the development of risk of bias frameworks and principles for use in these fields. However, risk of bias assessment tools may still contain the same muddling of methodological quality constructs unrelated to bias, with some risk of bias tools still directing the user towards reporting items.^27,28 Moreover, as a result of the introduction of the term “risk of bias,” the term “methodological quality” has been used ambiguously to include constructs unrelated to internal validity and now often entails a broader definition, even when used in evidence synthesis. While these terms may help to distinguish between these assessments, they also imply that methodological quality is a separate and distinct construct from risk of bias, which may be inaccurate. Methodological quality, when defined only in terms of internal validity of a study, can be—and sometimes is—used interchangeably with risk of bias, particularly when the purpose is to assess primary studies included in synthesized research. Perhaps one notable distinction is that risk of bias tools usually contain rules for making absolute value judgments regarding the risk of bias.²

A recent article by Furuya-Kanamori et al.² attempts to redefine methodological quality and risk of bias based on the intent of the assessment. In this scheme, a methodological quality assessment tool contains a list of safeguards that can be counted, and it becomes a risk of bias assessment tool when the methodological quality assessment tool is updated with rules that enable safeguard (item) counts to be converted into a qualitative judgment (eg, high or low risk of bias judgments).² Tools are then labeled as a methodological quality assessment tool or a risk of bias assessment tool, depending on the intent. This paper distinguishes the 2 terms for researchers; however, this terminology is yet to be adopted by the research community.

Methodological quality based on a quality count (ie, safeguards enumerated to produce a quality score) can be interpreted to be the probability that a study is credible.²⁹ Significant attempts have been made to overcome the errors previously made in understanding methodological quality assessments, either through the intent of the assessment tool or by shifting the emphasis to the need for a strong link between the items included in a tool and bias through use of the term “risk of bias.”⁵ However, when methodological quality and risk of bias involve assessments through the same underlying safeguards, the intent is to describe the assessment of internal validity of a study as it relates to systematic error through the implementation or non-implementation of methodological safeguards in a study.

Different research groups have different interpretations of the term “methodological quality” and researchers may also associate this term with “quality of the evidence,” which was the historic term used by the GRADE Working Group that considered methodological limitations and other quality concepts, such as imprecision, publication bias, and indirectness.⁵ Quality is a diffuse concept and means different things to different groups. For example, when using an approach to assessing quality that does not focus on internal validity items alone, a study may be considered of “high quality” (for its particular objective and design) but still be at more than a low risk of bias. Due to these issues, GRADE now uses the term “certainty” of the evidence as opposed to “quality,” with risk of bias being just one of the contributing determinants of certainty.

Recommendation for JBI to adopt the term “risk of bias”

The current JBI critical appraisal tools for effectiveness reviews are checklists for each study design (herein focusing on analytical study designs), with the majority of questions addressing the methodological safeguards implemented in the primary study. The JBI Effectiveness Methodology Group has made the decision to move to the term “risk of bias” to describe the assessment of methodological safeguards in a study to mitigate biased results (for quantitative study designs). In formalizing this recommendation regarding the way the term “risk of bias” should and should not be used in relation to JBI reviews of effectiveness, bias is considered to be systematic error in the results of quantitative studies. The term “risk of bias” refers to the potential for bias, and not measurement of the actual bias in a study. Further, risk of bias does not include the assessment of other quality or critical appraisal constructs, such as reporting quality, random error, external validity, or ethical considerations.

This decision was made to align the JBI Collaboration with other research groups, such as Cochrane, to avoid some of the misunderstandings surrounding the terms “quality” and “critical appraisal.” Therefore, the way forward for JBI is to adopt the term “risk of bias” to reduce confusion for reviewers by using similar terms to collaborators and to unify the nomenclature in this field. As mentioned previously, risk of bias is one of the grading domains in a GRADE assessment where authors can lower their certainty in the body of evidence by 1 or 2 grades. Adopting the term “risk of bias” instead of “critical appraisal” and shifting to internal validity assessment will align JBI reviewers more closely with the GRADE approach. Importantly, this terminology only applies to JBI’s analytical quantitative tools, as different terminology is more appropriate when considering qualitative or other forms of evidence. A common language is necessary to ensure a universally understood assessment on which recommendations for clinical practice are made.

Funding

MK is supported by the INTER-EXCELLENCE grant number LTC20031—Towards an International Network for Evidence-based Research in Clinical Health Research in the Czech Republic. ZM is supported by an NHMRC Investigator Grant, APP1195676.

Author contributions

JCS for conception and design of the work. All authors critically revised the manuscript for intellectual content and approved the final version.