Secondary Logo

Journal Logo

METHODOLOGY

Should I include studies from “predatory” journals in a systematic review? Interim guidance for systematic reviewers

Munn, Zachary1; Barker, Timothy1; Stern, Cindy1; Pollock, Danielle1; Ross-White, Amanda2; Klugar, Miloslav3; Wiechula, Rick4; Aromataris, Edoardo1; Shamseer, Larissa5

Author Information
doi: 10.11124/JBIES-21-00138
  • Free

Abstract

Introduction

Systematic reviews and other types of evidence synthesis play a crucial role in the translation of evidence into practice and should underpin trustworthy recommendations for policy and practice.1 Systematic reviews aim to evaluate and synthesize the best available evidence and can be used to address a range of uncertainties across many different fields for various questions. Because systematic reviews include research studies or reports as their unit of observation, any potential threat to the validity of that unit must be evaluated and critiqued. Journals that have been labeled “predatory” (hereon termed “predatory journals”) pose unique challenges for evidence synthesis, academic publishing, and science itself. Although there is an absence of evidence demonstrating a relationship between journal integrity and research integrity (with studies starting to be conducted on this topic),2 predatory journals are concerning because they may incentivize researcher misconduct (eg, falsifying research, selectively reporting research) through the aggressive solicitation of submissions3; the promise of rapid publication4-6; and compromised or absent quality-control measures, such as editorial and peer-review processes.7 As such, these journals do not appear to adhere to widely accepted scholarly publishing practices, reporting standards, or publishing ethics, such as those from the Committee on Publication Ethics (COPE) or International Committee of Medical Journal Editors (ICMJE).5

As a result of their poor practices, research published in predatory journals may not align to reporting standards, could contain errors, or, in worst cases, be fraudulent.6 Of course, it is important to acknowledge that even journals considered to be legitimate have published fraudulent or erroneous research (although this research may be retracted when identified in legitimate journals).8 Conversely, there is the possibility that due to their low barrier to publishing, well-intended researchers are sending predatory journals potentially trustworthy research that would otherwise be rejected from or not submitted to traditional journals.9,10 Adding complexity to this issue is that authors may unknowingly submit good-quality research to journals they believe are legitimate when they are not. This unique situation poses both ethical and professional challenges for a systematic reviewer when deciding what to do with research in predatory journals.

JBI is an international organization with a mission to promote and support evidence-based health care.11 JBI has developed a large range of evidence synthesis approaches, which are taught and followed across the globe.1,12 Increasingly at JBI, we have been faced with questions regarding the impact of predatory journals on evidence synthesis and requests for guidance. In 2020, a subgroup of the JBI Scientific Committee was formed to investigate this issue and to develop further guidance in this field. In this paper we discuss some of the key issues regarding predatory journals, the potential impact on evidence synthesis, interim strategies for dealing with predatory journals in systematic reviews, and an initial work program to advance understanding in this field.

What is a predatory journal?

Until recently, the issue of characterizing predatory journals was difficult due to the lack of an agreed definition of the topic.4,13 In late 2019, Grudniewicz and colleagues provided the following definition of predatory journals:

“Predatory journals and publishers are entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices.”4(p.211)

Interestingly, the group decided to leave out peer-review quality as an item. Although inadequate or negligent peer review is a potential characteristic of predatory journals,4,5 given the lack of transparency of the peer-review process across legitimate and suspected predatory journals, it was not included in this definition. Other items left out were the intent to deceive and the quality of the journal, which were deemed too subjective to assess.4

Why are they a problem?

Research that goes nowhere

For those authors who do naively publish in predatory journals, they are less likely to see their research making any impact in society and contributing to advancing care or science.14 This is largely because studies published in such journals are harder to find (due to inconsistent indexing in trusted databases)15; the studies gain little attention due to the low readership of these journals; and the research contained in them is inadequately peer reviewed, reported, or conducted. Most problematically, however, there is little evidence that they engage in digital archiving practices, which ensure that research can be discovered in perpetuity.5 If you consider the countless hours that academics put into their work, from ethics approvals, the collection of data, and the writing of the manuscripts, to the submission and eventual publication in the journals, publishing in predatory journals is wasteful because such journals prevent optimal dissemination of research.

Waste of humans and animals

For preclinical animal studies that have been published in these journals, it is unlikely they will contribute to a translational pathway and progress to clinical studies or impact, suggesting the unnecessary waste of millions of research animals for seemingly little reason.14 Therefore, when preclinical research is published in predatory journals, it is almost in direct contradiction to the 3Rs (Replacement, Reduction, Refinement)16 for the ethical, humane, and responsible care of animals for scientific purposes. Of course, there is a human cost as well, both on behalf of the participants, subjects, patients, or clinicians taking part in these studies. The time these different groups have contributed and the impact on their care or day-to-day lives is largely for naught. In addition, if these journals do perform peer review, it likely a waste of the reviewers’ time as well. Due in part to this waste, publishing in these journals has been described as unethical.14

Potential hotbed for pseudoscience, fake science

Due to their often inadequate peer review and editorial practices, predatory journals have been found to accept erroneous or fraudulent research and pseudoscience17-19 at potentially higher rates than legitimate journals.7 Not only does this undermine the scientific community, but it could be contributing to the disastrous and growing lack of faith in science globally through the seemingly credible publication of such articles and their dissemination.20 For a lay audience, these articles can appear scientific and reputable and therefore a trustworthy source to be cited.

Dealing with a constant barrage of spam

The aggressive, indiscriminate, and unwarranted solicitation that predatory publishers engage in is also problematic, particularly for researchers with little previous publishing experience.21 Although it seems simple enough to delete or block these emails, the sheer volume of the emails from seemingly endless variations of journals presents serious implications for academic and clinician workloads and time management. Such email fatigue could be a contributing factor to academic and clinician burnout.22 It has been estimated that even trainees and junior faculty are receiving 1500 unsolicited emails per year,23 many of which come from predatory journals. As academic rank and authored publications increase, so does the amount of unsolicited email.23 Even with appropriate spam and junk filters, many journals still find a way to infiltrate inboxes.23 Many scientists are forced to waste countless time and energy deleting and managing emails in their inboxes, contributing to a loss of time equaling almost USD$1.1 billion per year.21 As universities and research institutions attempt to systematically block these spam emails, they may unintentionally make filters too strict, which has the potential to block relevant and important emails.

Who publishes in these journals and why?

Given the concerns and issues with predatory publishers, it is surprising they still receive so many submissions. This begs the questions: who is publishing in these journals, and why are they doing so? Publishing in predatory journals is a global issue, with authors from all countries, regardless of language or economic status14; although as a proportion of total research output, it seems that authors from low-income or low-middle-income economies are more likely to publish in these journals.24,25 The most frequent funder in a sample of nearly 2000 biomedical research studies published in predatory journals was the U.S. National Institutes of Health (NIH).14 Researchers from prestigious universities such as Harvard were among the top contributors in that sample.14

Many of those who do get caught in the trap of predatory journals are trainee and early career researchers or those who have little or no prior publication experience.17 Many authors who have published in predatory journals had not heard of predatory journals previously.14,24 As such, perhaps the most common explanation for researchers publishing in these journals is their lack of knowledge around scholarly publishing and publishing standards.10,26

In some jurisdictions, authors may receive payment for each publication they author, or there might be pressure from their institution or country research policy regarding the quantity rather than quality of published research, regardless of where it is published. This can incentivize people to publish in poor-quality journals or those claiming to be indexed in awarded databases.27 There is also, of course, the “publish or perish” mentality, whereby the quantity of publications is prioritized over their quality.28

In some cases it may not necessarily be the journals that are acting in a predatory fashion – it may be the authors themselves who exploit predatory journals for a quick win, to boost their CV, or potentially publish low-quality research. The push for academics to publish their work as open access, aligned to the movement for open science, could also be driving authors towards predatory journals.26 In biomedicine, predatory journals often have lower fees compared with other open access options,5 which can be an attractive option to researchers from low- to low-middle-income countries, who represent a disproportionate number publishing their research in these journals.24 Whilst these authors also face the challenges addressed earlier, there are also structural problems to consider: for example, tertiary institutions may be unable to provide the necessary facilities in terms of database access, education, and research sponsorship, thus resulting in researchers in low-income countries publishing in these journals.

Predatory journals and systematic reviews

Until now, much of the inquiry into predatory journals has focused on identifying such journals and their characteristics, as well as efforts to diminish their impact. However, little attention or research has been conducted in the field of evidence synthesis and predatory journals,6 and there appears to be a lack of consensus29 or guidance on how to address predatory journals in systematic reviews. Based on the experience of this group conducting webinars,30,31 focus groups, and Twitter polls, there appears to be a wide variation – and sometimes divisive opinions – on whether and how to consider studies from suspected predatory journals in systematic reviews.

Regardless of whether the evidence synthesis community agrees that studies from predatory journals should be included in systematic reviews, we now know that these studies are being cited in systematic reviews.32 Whether and how they are being included in reviews and meta-analyses is still being determined. There appear to be multiple potential entry points for a study from a predatory journal to find its way into the screening and selection process within a systematic review. Firstly, we know that some journals that have been listed on the (now defunct) Beall's List as predatory can be found in PubMed.15,33 This is thought to primarily have occurred through the author-upload feature of PubMed Central (PMC) allowing NIH-funded researchers (as well as those from select other funding organizations) to comply with their funders’ mandates to make all funded research freely available.15,34 Secondly, Google Scholar uses an automated approach to listing research, whereby anything that appears “scholarly” is included, which explains how content from predatory journals can be found. Although Google Scholar is a free tool that is simple for the public, journalists, and researchers to use, there is a potential downside in that it may include predatory journals. Additionally, if performing reference list searching or onwards citation searching for consideration in a systematic review, this presents another avenue for how studies from predatory journals might be encountered. There are also cases of legitimate journals that have been “hijacked” or purchased by suspected predatory publishers, yet have retained their previous indexing status in quality bibliographic databases, making it difficult to distinguish between the journals’ reputable years and those where it is owned by a predatory publisher.32

Although there is no consensus or guidance as yet for systematic reviewers, we suggest there are a number of avenues that authors could take regarding the inclusion of studies from predatory journals in systematic reviews. In the following section we outline some potential strategies for dealing with content from predatory journals encountered in systematic reviews. At this stage, we are not advocating for any particular approach to be prioritized; rather, we are urging reviewers to describe any planned methods and processes for considering the source of published research in their protocols and in their systematic review reports. Doing so will enhance transparency of the review search process and facilitate future methods research in this area. Regardless of the approach decided by the review team, we advocate for education for all researchers regarding predatory journals.

Deciding to exclude studies from predatory journals in systematic reviews

Avoiding studies from predatory journals

For reviewers aiming to avoid studies from potential predatory journals, they may choose to only search bibliographic databases with tight quality-control measures in place and stringent indexing criteria,32 such as MEDLINE and Embase (whilst recognizing this may exclude some journals/articles included in PubMed Central). When reviewing references for included studies and onwards citation searching, the journal of each study could be assessed to determine if it is potentially predatory.

Detecting potential predatory journals in reviews

Reviewers wishing to universally exclude studies from predatory journals will need to assess the journal of each otherwise eligible study. However, identifying predatory journals is not a straightforward process. Reviewers have several options and ought to be explicit about which approaches they use:

  • Measure journals against an agreed definition and the identified characteristics of predatory journals.4,5,26 Other criteria that could be considered are whether the journal is a member of the COPE,26 adheres to COPE Core Practices, or is listed on the database of open access journals (DOAJ).26 Reviewers could also refer to the Think, Check, Submit campaign checklist to determine a journal's likely credibility.26
  • Use a checklist. At least 93 checklists currently exist for this purpose35; however, none have been identified as the gold standard. It is also important to note that not all predatory journals will meet checklist criteria and there is no clear threshold or cut-offs for many of these checklists.35 If choosing to use a checklist to identify predatory journals in order to exclude studies from a systematic review, authors should clearly justify their rationale and the decision-making process, including cut-offs. For similar reasons to why we perform risk of bias assessment in duplicate, it may be worthwhile to have two independent reviewers complete the checklist. Alternatively, reviewers may choose a checklist to identify legitimate journals, rather than predatory journals.
  • Use a list. Perhaps the simplest strategy to identify a predatory journal is to use a list of predatory journals. Various options exist,4,36 including the original Beall's list or Cabell's predatory list. Importantly, however, the transparency and replicability of these lists is suboptimal, as their methodology for designating predatory journals is not always apparent or available. Alternatively, reviewers may wish to use a listing (or combination thereof) of trusted journals such as the DOAJ, Journal Citation Reports, Cabell's verified list, as well as subject-specific lists or regional databases. Notably, some crossover of classifications of individual publishers among predatory and legitimate journal lists has been identified.4 This prompted the development of an algorithm from the World Association of Medical Editors centered on the DOAJ as an important identification point of predatory and legitimate journals,36 but also includes extra decision points due to the shortcomings of this list (such as it not representing the totality of existing, legitimate open access journals). Additionally, its focus on open access journals means a strategy ought to be employed to identify trustworthy non–open access journals. Despite the use of lists being a straightforward and simple approach to classifying the journal type, we advise caution as no list has yet been identified as optimal.

Deciding to include studies from predatory journals in systematic reviews

No additional measures

It may be that systematic reviewers consider no additional measures are required to handle studies from predatory journals, and that the current processes of systematic reviews will be adequate to address any potential issues with these studies (eg, eligibility screening, risk of bias assessment, rigorous data extraction, contact of authors when there are questions, process to establish certainty in results [GRADE]37). In these cases, it may be worthwhile mentioning in the protocol and the report that all studies are treated identically and the authors do not plan to evaluate or report on journal integrity.

Searching for studies from predatory journals

Despite the common advice for systematic reviewers to disregard publication status in terms of eligibility criteria for a systematic review, it is extremely unlikely that even those who take the view to include studies from a predatory journal in systematic reviews would advocate for purposeful identification of predatory journals. Additionally, this is not practical given there is no bibliographic database indexing predatory journals in which to conduct such a search. Rather, if systematic review authors are not concerned about the publication source, they may continue to search Google Scholar, PMC, and other less-controlled search sources and not implement additional investigations into journal integrity for studies.

Assessing the quality and integrity of studies from predatory journals

A major concern regarding predatory journals is that they may publish lower-quality studies in comparison to legitimate journals. Currently, systematic reviewers have the skills and tools available to assess risk of bias and critically appraise studies; however, it may be necessary for risk of bias or critical appraisal tools to be modified for studies from predatory journals. For scoping reviews, which currently do not include a process of critical appraisal,38-40 there may be other strategies that need to be investigated.

Detecting error and fraud is more difficult and can often be missed. Traditional peer review may identify error and fraud, but in many cases it may not. If systematic reviewers are concerned with the credibility of the data in a study, there are some strategies that could be applied to detect error or perhaps fraudulent research, although further work is required in this field. For example, as a simple strategy, reviewers could critically examine the numbers and totals reported in any study to ensure they add up. Reviewers may recalculate effect sizes or run analyses again, which may be facilitated by requesting the original code or data from primary authors. Reviewers may also judiciously investigate time frames, sample sizes, and recruitment strategies to ensure that whatever is reported is feasible.41 There may also be specific fraud- and error-detection procedures/techniques that could be applied.42-46 Finally, forensic emails to primary study authors may provide crucial information regarding whether error or fraud is present.

If there are concerns regarding the data from predatory journals, the impact of the results from these studies could be investigated using subgroup or sensitivity analyses. The reviewer may be faced with a difficult choice where these analyses present divergent results and in the selection of a pooled estimate to present to the reader.

Preprint servers and predatory journals

There are some similarities between preprint servers and predatory journals. For example, studies published in predatory journals and those on preprint servers are unlikely to have received pre-publication peer review (adequate or otherwise). However, the fundamental difference is that preprint servers are transparent about not conducting peer review and some have quality control measures to detect potential ethical issues prior to the manuscript being posted.47 Additionally, because they are “preprint” publications, authors often have the intention of submitting them to a peer-reviewed journal.47 This “intention” for peer review may be a barrier to potential fraudulent or erroneous data within these preprints. Despite the differences, some of the decisions and strategies suggested above (eg, deciding whether to include/exclude, contacting of authors, sensitivity analyses) could be considered for preprint studies as well.

Discussion and conclusion

The JBI Predatory Publishing Practices Group aims to provide guidance for systematic reviewers regarding predatory journals. The ideas presented in this preliminary guidance paper require further evaluation and reflection. To assist us in developing guidance, we plan to undertake further meta-epidemiological work in this field, conduct surveys and focus groups of systematic reviewers, and undertake consensus processes to develop guidance. Outputs for the group alongside the guidance may be new software applications, tools, checklists, implications for reporting standards (such as PRISMA 2020 and PRISMA extensions),48-50 and educational materials.

We are cognizant of the danger of advocating to include studies from predatory journals in systematic reviews because this could be seen as a way to legitimize these journals or provide them with credibility, even through the simple act of citing studies (even where excluded or in the introduction/discussion).6 This is something that will need to be considered as the group develops formal guidance.

Encouragingly, there is now a meta-epidemiological study investigating the impact of studies from predatory journals in systematic reviews. In a Cochrane systematic review on low back pain (forthcoming),51 Hayden and colleagues distinguished studies in legitimate journals as those from selected trusted publishers, and those from potentially predatory journals by applying lists and the aforementioned definition.2 They investigated the characteristics of the studies from these two groups of journals and analyzed their impact on the results of the review and the final estimates of effect. They found that although studies in a predatory journal did not necessarily impact the review's outcomes, other characteristics of poorly conducted and reported studies are associated with an overestimation of treatment effect. The authors identified that trials published in predatory journals were more likely to have reporting and publication integrity issues.2 The authors advocated for such methodological studies to be replicated in other topic areas. In addition to these meta-epidemiological studies evaluating the impact of predatory journals in different types of evidence synthesis, we need further research into methods to detect predatory journals, methods to evaluate error and fraud in research, and studies exploring researcher attitudes and opinions towards predatory journals, especially in evidence synthesis.

Many are trying to combat the emergence and proliferation of predatory journals. There have been calls for changing models for payment (or rewards) for publication26 and even disincentivizing or penalizing publication in these journals.27,52 Increasing funding for open access publishing, keeping stringent lists of accredited journals, and raising awareness of predatory publishers are some ways to combat this issue. Research funders also have a major role to play, and should stipulate that investigators should not publish in these journals, provide measures to decrease the likelihood of papers being published in predatory journals, and indicate features of acceptable journals.27,53,54 Regulatory bodies and ethics approval bodies should mandate that studies are not published in these journals as part of the approval process.27 Public and patient groups could also be mobilized to combat this issue.27 There are also calls for additional education regarding publishing (including open access models),26,53 the need for specialist publications officer positions in institutions, and greater collaboration.26,53

It must be recognized that the onus of changing predatory publishing should not just be on researchers, especially for those with less exposure to publishing who may be more susceptible. Active change within the wider scientific community is needed. However, until we, as a scientific community, have identified measures and strategies to combat predatory journals, we need to develop strategies to manage these outputs, particularly in systematic reviews.

When conducting systematic reviews, we aim to include the best available evidence that is not fraudulent or subject to error. It is clear that predatory journals pose challenges for systematic reviewers, with these difficulties unlikely to disappear anytime soon. This article provides interim guidance that may facilitate systematic reviewers’ decision-making about studies published in predatory journals until further evidence is available.

References

1. Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol 2018;18 (1):5.
2. Hayden J, Ellis J, Ogilvie R, Boulos L, Stanojevic S. Meta-epidemiological study of publication integrity, and quality of conduct and reporting of randomized trials included in a systematic review of low back pain. J Clin Epidemiol 2021;134:65–78.
3. Clemons M, de Costa ESM, Joy AA, Cobey KD, Mazzarello S, Stober C, et al. Predatory invitations from journals: more than just a nuisance? Oncologist 2017;22 (2):236–240.
4. Grudniewicz A, Moher D, Cobey KD, Bryson GL, Cukier S, Allen K, et al. Predatory journals: no definition, no defence. Nature 2019;576 (7786):210–212.
5. Shamseer L, Moher D, Maduekwe O, Turner L, Barbour V, Burch R, et al. Potential predatory and legitimate biomedical journals: can you tell the difference? A cross-sectional comparison. BMC Med 2017;15 (1):28.
6. Hayden JA. Predatory publishing dilutes and distorts evidence in systematic reviews. J Clin Epidemiol 2020;121:117–119.
7. Bohannon J. Who's afraid of peer review? Science 2013;342 (6154):60–65.
8. Heathers J. The Lancet has made one of the biggest retractions in modern history. How could this happen? [internet]. The Guardian. June 5, 2020 [cited 2021 Apr 13]. Available from: https://www.theguardian.com/commentisfree/2020/jun/05/lancet-had-to-do-one-of-the-biggest-retractions-in-modern-history-how-could-this-happen
9. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull 1979;86 (3):638.
10. University of Ottawa, Shamseer L. “Predatory” journals: an evidence-based approach to characterizing them and considering where research ought to be published [thesis]. 2021.
11. Jordan Z, Lockwood C, Munn Z, Aromataris E. The updated Joanna Briggs Institute model of evidence-based healthcare. Int J Evid Based Health 2019;17 (1):58–71.
12. Stern C, Munn Z, Porritt K, Lockwood C, Peters MD, Bellman S, et al. An international educational training course for conducting systematic reviews in health care: the Joanna Briggs Institute's comprehensive systematic review training program. Worldviews Evid Based Nurs 2018;15 (5):401–408.
13. Aromataris E, Stern C. Supporting a definition of predatory publishing. BMC Med 2020;18 (1):125.
14. Moher D, Shamseer L, Cobey KD, Lalu MM, Galipeau J, Avey MT, et al. Stop this waste of people, animals and money. Nature News 2017;549 (7670):23.
15. Manca A, Moher D, Cugusi L, Dvir Z, Deriu F. How predatory journals leak into PubMed. CMAJ 2018;190 (35):E1042–E1045.
16. Guhad F. Introduction to the 3Rs (refinement, reduction and replacement). J Am Assoc Lab Ani Sci 2005;44 (2):58–59.
17. Bagues M, Sylos-Labini M, Zinovyeva N. A walk on the wild side: ‘predatory’ journals and information asymmetries in scientific evaluations. Res Policy 2019;48 (2):462–477.
18. Beall J. Chapter 12: Scientific Soundness and the Problem of Predatory Journals. In: Kauffman AB, Kauffman JC, editors. Pseudoscience: The Conspiracy Against Science [internet]. The MIT Press. 2018 [cited 2021 Jun 20]. Available from: https://mitpress.universitypressscholarship.com/view/10.7551/mitpress/9780262037426.001.0001/upso-9780262037426-chapter-012
19. Conner-Simons A. How three MIT students fooled the world of scientific journals [internet]. MIT News; 2015 Apr 14 [cited 2021 Apr 13]. Available from: https://news.mit.edu/2015/how-three-mit-students-fooled-scientific-journals-0414.
20. Ferris LE, Winker MA. Ethical issues in publishing in predatory journals. Biochem Med 2017;27 (2):279–284.
21. da Silva JAT, Al-Khatib A, Tsigaris P. Spam emails in academia: issues and costs. Scientometrics 2020;122 (2):1171–1188.
22. Wood KE, Krasowski MD. Academic e-mail overload and the burden of “academic spam.”. Acad Pathol 2020;7:2374289519898858.
23. Krasowski MD, Lawrence JC, Briggs AS, Ford BA. Burden and characteristics of unsolicited emails from medical/scientific journals, conferences, and webinars to faculty and trainees at an academic pathology department. J Pathol Inform 2019;10:16.
24. Xia J, Harmon JL, Connolly KG, Donnelly RM, Anderson MR, Howard HA. Who publishes in “predatory” journals? J Assoc Inform Sci Technol 2015;66 (7):1406–1417.
25. Demir SB. Predatory journals: who publishes in them and why? J Inform 2018;12 (4):1296–1311.
26. Lalu MM, Shamseer L, Cobey KD, Moher D. How stakeholders can respond to the rise of predatory journals. Nat Hum Behav 2017;1 (12):852–855.
27. Hedding DW. Payouts push professors towards predatory journals. Nature 2019;565 (7737):267–268.
28. Oxford University Press, De Leeuw E. ‘Collaborate now with worldwide physicians and scientists!’—the obscenity of predatory science. 2020.
29. Cukier S, Lalu M, Bryson GL, Cobey KD, Grudniewicz A, Moher D. Defining predatory journals and responding to the threat they pose: a modified Delphi consensus process. BMJ Open 2020;10 (2):e035561.
30. Munn Z. Predatory publishing in systematic reviews [internet]. 2021 [cited 2021 Apr 13]. Available from: https://www.youtube.com/watch?v=srMBpB_IZ54&ab_channel=JBI
31. Shamseer L. Should research in predatory journals be included in systematic reviews? [internet]. 2021 [cited 2021 Apr 13]. Available from: https://www.ti.ubc.ca/2021/02/01/feb-24-ti-methods-speaker-series-should-research-in-predatory-journals-be-included-in-systematic-reviews/.
32. Ross-White A, Godfrey CM, Sears KA, Wilson R. Predatory publications in evidence syntheses. J Med Libr Assoc 2019;107 (1):57–61.
33. Duc NM, Hiep DV, Thong PM, Zunic L, Zildzic M, Donev D, et al. Predatory open access journals are indexed in reputable databases: a revisiting issue or an unsolved problem. Med Arch 2020;74 (4):318–322.
34. Topper L, Marill J, Kelly C, Funk K. RE: How predatory journals leak into PubMed. CMAJ 2018.
35. Cukier S, Helal L, Rice DB, Pupkaite J, Ahmadzai N, Wilson M, et al. Checklists to detect potential predatory biomedical journals: a systematic review. BMC Med 2020;18 (1):104.
36. Laine C, Winker MA. Identifying predatory or pseudo-journals. Biochem Med 2017;27 (2):285–291.
37. Atkins A, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ 2004;328 (7454):1490.
38. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 2018;18 (1):143.
39. Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth 2020;18 (10):2119–2126.
40. Pollock D, Davies EL, Peters MDJ, Tricco AC, Alexander L, McInerney P, et al. Undertaking a scoping review: a practical guide for nursing and midwifery students, clinicians, researchers, and academics. J Adv Nurs 2021;77 (4):2102–2113.
41. Lock S. Lessons from the Pearce affair: handling scientific fraud. BMJ 1995;310 (6994):1547–1548.
42. van den Bor RM, Vaessen PWJ, Oosterman BJ, Zuithoff NPA, Grobbee DE, Roes KCB. A computationally simple central monitoring procedure, effectively applied to empirical trial data with known fraud. J Clin Epidemiol 2017;87:59–69.
43. Brown NJL, Heathers JAJ. The GRIM test: a simple technique detects numerous anomalies in the reporting of results in psychology. Soc Psychol Personal Sci 2016;8 (4):363–369.
44. Schumm WR, Crawford DW, Lockett L. Using statistics from binary variables to detect data anomalies, even possibly fraudulent research. Psychol Res Applic 2019;1 (4):112–118.
45. Heathers JA, Anaya J, van der Zee T, Brown NJ. Recovering data from summary statistics: sample parameter reconstruction via iterative techniques (SPRITE). PeerJ Preprints 2018;6:e26968v1.
46. Heathers JA, Brown NJ. DEBIT: a simple consistency test for binary data. Unpublished paper Bouve College of Health Sciences, Northeastern University 2019; 360.
47. Kirkham JJ, Penfold NC, Murphy F, Boutron I, Ioannidis JP, Polka J, et al. Systematic examination of preprint platforms for use in the medical and biomedical sciences setting. BMJ Open 2020;10 (12):e041849.
48. Brennan SE, Munn Z. PRISMA 2020: a reporting guideline for the next generation of systematic reviews. JBI Evid Synth 2021;19 (5):906–908.
49. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71.
50. Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018;169 (7):467–473.
51. Hayden JA, Cartwright J, van Tulder MW, Malmivaara A. Exercise therapy for chronic low back pain. Cochrane Database Syst Rev 2012; (4):CD009790.
52. Pond BB, Brown SD, Stewart DW, Roane DS, Harirforoosh S. Faculty applicants’ attempt to inflate CVs using predatory journals. Am J Pharmaceut Educ 2019;83 (1):7210.
53. Moher D, Moher E. Stop predatory publishers now: act collaboratively. Ann Intern Med 2016;164 (9):616–617.
54. Shamseer L, Cobey K, Page M, Brehaut J, Grimshaw J, Straus S, et al. Top health research funders’ guidance on selecting journals for funded research. F1000Research 2021;10 (100):
Keywords:

evidence synthesis; evidence-based practice; journals; predatory publishing; systematic reviews

© 2021 JBI