Secondary Logo

Journal Logo


Methodological research

open questions, the need for ‘research on research’ and its implications for evidence-based health care and reducing research waste

Puljak, Livia MD, PhD

Author Information
International Journal of Evidence-Based Healthcare: September 2019 - Volume 17 - Issue 3 - p 145-146
doi: 10.1097/XEB.0000000000000201
  • Free

With the ever increasing pace of research reports published, a new type of study has emerged, in which researchers analyze research practices and provide methodological guidance for the future. The overall aims of such studies are to explore planning, conduct, reporting, dissemination, and evaluation of research, and to provide recommendations for their improvement.

A myriad of studies have shown that studies are planned, conducted and reported in a suboptimal way, and that there is ample room for improvement. It has been suggested that studies with inadequate planning, conduct and reporting represent avoidable research waste,1 causing enormous waste of funds and researchers’ efforts, and yielding suboptimal research reports. Methodological studies represent research efforts that may help in reducing research waste, and therefore should be welcomed as a valuable contribution that may contribute to production of better evidence and justify investments in research.

However, there are multiple open questions about what exactly is analyzed in such studies, which collective name we should use for them, what type of study designs can be considered methodological studies, are they synonymous with systematic reviews and meta-analyses (SR/MA), and so on.2–5

Methodological studies may be observational or interventional. The unit of analysis in observational methodological studies can be broadly described as a research-related report. Such reports can be very diverse, including, but not limited to, study protocols, datasets, full reports with data analyses, and summary formats such as abstracts, plain language summaries, and so on. These reports may or may not be published in peer-reviewed literature. Some of those research-related reports that are not published in peer-reviewed literature are available online, including materials posted in registries and repositories, or organizational web sites. Interventional methodological studies explore various interventions for improving research.

Various global descriptors for such studies are used, such as methodological studies, research-on-research, meta-epidemiological studies, or meta-research, but currently there is no consensus about the definition of some of those terms, the distinction between them and whether all of those descriptors are synonymous. In this article, such studies are mainly referred to as ‘methodological studies’, but with the understanding that there is no uniformly accepted common descriptor for this type of research.

The term ‘methodological study’ or ‘methods study’ indicates that the study is about methods, but this term can also be misleading, as there are many types of studies exploring new and modifying research-related methods that are not analyzing research-related reports, such as studies of methods for laboratory protocols, methods for new experiments on animals, and so on.

The term ‘meta-epidemiology’ appeared in 1997,6 and in 2002, Sterne et al.7 wrote the following definition of a meta-epidemiological study: ‘Biases in systematic reviews and meta-analyses may be examined in “meta-epidemiological” studies, in which the influence of trial characteristics such as measures of study quality on treatment effect estimates is explored’. A PubMed search conducted on 13 August 2019 for studies with the descriptor ‘meta-epidemiological’ in the title resulted in 101 records; the first one was published in 2007, and in 2018 there were 22 PubMed records described in the title as ‘meta-epidemiological’. But a cursory look into those studies indicates that not all of them analyze the influence of study characteristics on treatment effect estimates. Some of the studies self-described as meta-epidemiological in the title are simply analyzing characteristics of research-related reports, without any attempts to analyze influence of those characteristics on treatment effects. Furthermore, Sterne et al.'s7 definition talks about ‘trial characteristics’, but methodological studies are not limited to clinical trials.

Similar confusion may exist with the term meta-research. An informal PubMed search indicates that the phrase ‘meta-research’ was first used in 2005 by Kranke,8 for expressing doubt about the worth of such research: ‘I question whether such “meta-research” really helps to improve patient care or facilitates the applicability of research results’. To many individuals even vaguely familiar with research methods, the term meta-research may be first associated with the term ‘meta-analysis’. So, if we talk about meta-research, it is unclear whether we all mean the same thing – for some, meta-research possibly covers also SR/MA. In 2018, Ioannidis9 wrote: ‘Meta-research is the study of research itself: its methods, reporting, reproducibility, evaluation, and incentives’, but it is unclear whether this is accepted and universally understood as a definition of the term.

Research-on-research is also a very broad descriptor that could also be understood as encompassing SR/MA, as in such studies research is also done on other research.

There is a practical obstacle to the varying terminology used to describe such research, as, for example, if one would like to search for such studies to analyze their characteristics, it would be very difficult to create a search strategy to find methodological studies, due to lack of consistent descriptors.

Some authors describe their methodological studies as systematic reviews or descriptive reviews because they may use elements of methodology used in systematic reviews, such as systematic literature search, screening, data extraction, and so on, which may be conducted by two authors independently to reduce inadvertent errors. My personal opinion is that methodological studies should be viewed as different studies from SR/MA because the aim of methodological studies is not to summarize results of completed studies.4 While systematic reviews are often considered secondary research and they may not be considered original research at all,10 as they are using results generated by others to create new results, methodological studies are creating new results from research-related reports, and thus should be considered original, and primary, studies.3

As methodological studies may be both observational, and interventional, and some may consider them to be systematic reviews, there is an open question of how to report such studies. Existing reporting checklists may not be fully applicable to methodological studies.2

Currently, there are no uniform requirements or expectations that a protocol of methodological study should be published prospectively. This is another open question to the research community – about whether it should be expected that protocols of all types of studies should be made publicly available.5

Evidence-based health care is implemented by using existing evidence, and to make this evidence better, we need to know what is the good, the bad and the ugly, and we need to know how we can make it all good. Methodological studies can help in this. Resolving open questions regarding methodological studies themselves can help us even further in achieving optimistic goals about producing flawless evidence.


Conflicts of interest

The author reports no conflicts of interest.


1. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009; 374:86–89.
2. Puljak L. Reporting checklist for methodological, that is, research on research studies is urgently needed. J Clin Epidemiol 2019; 112:93.
3. Puljak L. Research-on-research studies or methodological studies are primary research. J Clin Epidemiol 2019; 112:95.
4. Puljak L. Methodological studies evaluating evidence are not systematic reviews. J Clin Epidemiol 2019; 110:98–99.
5. Puljak L, Pieper D. Registration of methodological studies, that is, “research-on-research” studies-should it be mandatory? J Clin Epidemiol 2019; 115:35–36.
6. Naylor CD. Meta-analysis and the meta-epidemiology of clinical research. BMJ 1997; 315:617–619.
7. Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002; 21:1513–1524.
8. Kranke P. Review of publication bias in studies on publication bias: meta-research on publication bias does not help transfer research results to patient care. BMJ 2005; 331:638.
9. Ioannidis JPA. Meta-research: why research on research matters. PLoS Biol 2018; 16:e2005468.
10. Krnic Martinic M, Meerpohl JJ, Von Elm E, Herrle F, Marusic A, Puljak L. Attitudes of editors of core clinical journals about whether systematic reviews are original research: a mixed-methods study. BMJ Open 2019; e029704doi:10.1136/bmjopen-2019-029704.
International Journal of Evidence-Based Healthcare © 2019 The Joanna Briggs Institute

A video commentary on implementation project titled: How do health professionals prioritise clinical areas for implementation of evidence into practice? The commentary is provided by Andrea Rochon RN, MNSc, Research Assistant, Queen's University, Ontario, Canada