Secondary Logo

Journal Logo

Editorial

The Potential and Peril of Pilot Research: Editorial Guidelines to Maintain Transparency and Reduce Overinterpretation of Effects

Bartholomew, John B.1; Campbell, Kristin2; Moore, Justin B.3

Author Information
Translational Journal of the ACSM: Summer 2020 - Volume 5 - Issue 11 - p 1-2
doi: 10.1249/TJX.0000000000000139
  • Free

“Research is the process of going up alleys to see if they are blind.”

―Marston Bates

We have the great privilege to work in a field that is dedicated to improving the lives of others through sports medicine, public health, exercise, and nutrition science. We accomplish this through a discovery phase of basic research conducted in the laboratory and the clinic that is then translated to action in the community (1). This is no simple task. The work in the laboratory is highly challenging even when completed under ideal conditions. To then translate these findings to the wonderful variation that we find in everyday life—across different settings and cultures—requires a separate, multistep process. This begins with qualitative and quasi-experimental feasibility and pilot studies that are conducted with the express aim of supporting a future randomized control trial (RCT). Although there exists a range of definitions (2), there is general agreement that feasibility studies come early in the research process and assess a narrow component of a planned intervention. Pilot studies come later in the research process and test an intervention protocol. Neither are designed to determine the efficacy of the intervention and are, instead, focused on the integrity of the experimental design and if the intervention protocol can be implemented or requires revision. Thus, they provide a low-cost effort to determine if a protocol or trial is viable and worth further testing. As a such, feasibility and pilot studies are critical components in translational research and to the design of future RCTs. These RCTs are often revised and repeated before being disseminated and driving change in public policy. The Editorial Board of the Translational Journal of American College of Sports Medicine (TJACSM) supports all aspects of this process—and will publish protocol papers, feasibility and pilot studies, RCTs, implementation and dissemination research, scoping reviews, and policy papers.

Of these, the most challenging is the consideration of feasibility and pilot studies. RCTs are the gold standard for translational work because they come closest to the true experimental designs of the laboratory. By contrast, the quasi-experimental designs that are used for pilot work are at greater risk for overestimating effects because of the lack of appropriate control, the use of a single setting, and the small samples (3). In fact, a recent meta-analysis of studies in our field demonstrated considerable reductions in the observed effect from published pilot studies to the subsequent RCT (4). This is no small matter given the concerns regarding the ability to replicate findings (5) (6). Despite this, pilot studies remain a critical part of the translational process and are worthy of publication if done so in a transparent manner that draws attention to the limitations of the resulting data. To this end, TJACSM has adopted clear guidelines for the reporting and publishing of pilot studies. TJACSM will refer to the updated CONSORT guidelines for pilot studies (7) in the instructions to authors. In addition, although pilot studies help to reveal what might be possible, they are—by definition—limited in their ability to indicate cause and are, instead, focused on estimating effects and the feasibility of a trial. This is most problematic for the naïve reader who may interpret these studies as efficacy trials. In response, TJACSM will seek to maximize transparency and set aside a section of the journal dedicated to the publication of pilot studies. Each article will be required to include “pilot” or “feasibility” in the title and for the text to specifically refer to the goals of the subsequent RCT to test efficacy. These will allow readers to recognize the provisional nature of the study. Finally, authors will be dissuaded from using inferential statistics and hypothesis testing unless they can demonstrate sufficient statistical power. Instead, articles submitted as pilot studies will be asked to emphasize estimates of effects (e.g., 95% confidence intervals, Cohen’s d, etc.) and feasibility of implementation (e.g., dose, acceptability, rates of uptake, etc.) to judge the outcome. Together, these are intended to allow scholars to share results of pilot studies to guide their own and other translational research while ensuring that the limitations of the study are fully transparent and explicitly described for the naïve reader.

In the best case, translational research is a complicated and multistep process that seeks to apply scientific rigor to settings in which there is very little control. By embracing the entirety of the translational process, TJACSM seeks to improve the quality of this research, the training of translational scholars, and the quality of life for those affected by this research.

REFERENCES

1. Rubio DM, Schoenbaum EE, Lee LS, et al. Defining translational research: Implications for training. Acad Med. 2010;85(3):470–5.
2. Eldridge SM, Lancaster GA, Campbell MJ, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: Development of a conceptual framework. PLoS One. 2016;11(3):e0150205.
3. Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry. 2006;63(5):484–9.
4. Beets MW, Weaver RG, Ioannidis JPA, et al. Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: A systematic review and meta-analysis. Int J Behav Nutr Phys Act. 2020;17(1):19.
5. Ioannidis JPA. Why Most published research findings are false. PLoS Med. 2005;2(8):e124.
6. Halperin I, Vigotsky AD, Foster C, Pyne DB. Strengthening the practice of exercise and sport-science research. Int J Sports Physiol Perform. 2018;13(2):127–34.
7. Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: Extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239.
Copyright © 2020 by the American College of Sports Medicine