Language bias in systematic reviews: you only get out what you put in

Limiting study inclusion on the basis of language of publication is a common practice in systematic reviews. Neimann Rasmussen and Montgomery cite lack of time, insufficient funding, and unavailability of language resources (e.g. professional translators) as the most common reasons for not including languages other than English (LOTE) in a systematic review.¹ Thirty-eight percent (95% confidence interval, 34-42%) from a random sample of 516 reviews (out of a total of 18,140 systematic reviews published in 2016) reported language restrictions (source:www.ksrevidence.com). While often the most feasible option, it introduces the risk of ignoring key data, introducing bias (referred to as language bias), as well as missing important cultural contexts, which may limit the review's findings and usefulness.^2-4 Cultural context may simply be tied to geography, or in some instances, fundamentally entwined with the review question: for example, conducting a review on Chinese herbal remedies that does not include Chinese-language studies, nor searches Chinese databases or resources; or a review that focuses on health promotion strategies for indigenous populations in Canada that does not consider French-language studies. Such examples would seemingly demand the inclusion of LOTE.

Currently, JBI methodology does not require authors to include papers in LOTE but recommends that, where a review team has capacity, the search should ideally attempt to identify studies and papers published in any language, and may expand the search to include databases and resources that index LOTE.² Further, authors are advised to outline any language restrictions with appropriate justifications, and consider the potential consequences of language restriction in their discussion,¹ which aligns with the PRISMA Statement (Item 6: Eligibility criteria, and Item 25: Limitations of the review process).⁵ The Campbell Collaboration takes a similar stance and warns against the risk of language bias, recommending that “ideally no language restrictions should be included in the search strategy,”^6(p.28) while Cochrane advocates that searches should not be restricted by language.⁷

Despite this overarching recommendation, across the diverse range of synthesis methodology and methods espoused by JBI, there are other important considerations for LOTE. If we consider the type of review question and thus the methodological design required, there may be different implications for qualitative reviews and mixed methods reviews due to the nature of their data and the potential issues in their translation.⁸ Scoping reviews may also not fall under this remit due to their very nature; therefore, it is clear that we cannot assume a one-size-fits-all approach for the inclusion of LOTE.

Many protocols and reviews submitted to JBI Evidence Synthesis limit the search parameters to English only, with authors overwhelmingly stating this is due to the limited resources available. The infrequent exception to this arises from author teams in Europe, South America, and Asia who include at least one additional LOTE (largely based on the languages spoken by the author team) and search databases or resources in LOTE. Of the 17 reviews published in JBI Evidence Synthesis in the first half of 2020, seven (41%) did not limit the language to English. Pleasingly, in this issue, half of the protocols published also do not limit the language to English, with the languages chosen to represent those of the author team and/or those relevant to the cultural context (see examples^9,10).

A key message that JBI highlights in its global systematic review training program¹¹ is that an attempt should be made to locate all evidence (published and unpublished) that is relevant to a review question; however, by allowing reviews that limit by language, JBI systematic reviews are essentially overlooking this very feature that they should be promoting. JBI has reconsidered its stance on the inclusion of LOTE in JBI systematic reviews and is currently deliberating on how best to implement this; for example, standards regarding databases and other resources in LOTE (e.g. which to include as well as training and access), the use of Google Translate and other translation tools to screen/assess suitability, recruitment of collaborators to assist with LOTE, and acknowledgment versus authorship of collaborators.

There are also multiple ways to deal with difficulties in reading and managing LOTE studies in a systematic review. Rather than expensive full translations of published articles, which are often not necessary, a more economical solution may be for a reviewer to work closely with a person who can read the language and facilitate identification and extraction of the required information. In addition, studies for which nobody can be found to help with translation could be listed in the review with a remark that the reviewers could not process the study. This would at least enable the readers to make a judgment about the possible bias involved.

While it is clear this will impact authors, we must move forward to ensure we capture a truly global picture of the evidence. Should we expect authors to include every piece of research ever written that fits their review's inclusion criteria? It simply may not be feasible; however, by limiting a review to one language from the outset, we are violating the very essence of what a systematic review is and its purpose in assisting in making informed decisions from the best available evidence.