“Honesty and transparency make you vulnerable. Be honest and transparent anyway.”
It has been advanced that science has a reproducibility crisis (1). There are a host of causes. One is common in pilot studies, where the “generalizability biases” are common (2). These occur when the features of the intervention, setting, or sample or participants are not represented in the next stage of testing (2)—be it a full randomized controlled trial or the resulting effectiveness trial. In addition, even the best designed trials often fail to adhere to the protocol proposed for the trial. Although there are examples in which this is due to scientific misconduct and conflict of interest (3), it is far more likely that this is due to complex challenges related to implementation fidelity in the “real world” (4). Although these challenges are understandable and often unavoidable, the advancement of research requires authors to report departures from the protocol in the publication of results (4).
The transparency of methods would be significantly enhanced through the publication of protocol papers. It is, in fact, the position of the editors of the TJACSM that publication of study protocols and honest reporting of deviations from study protocol are critical to advance the field of translational science and improve the reproducibility of the outcomes.
Although the concept of publishing protocol manuscripts is not new, there are few journals publishing protocol papers. This leads to the inevitability that most studies will not have a publicly available protocol that the scientific community can reference when the main results are published. The lack of a published protocol can enable lapses in recall, judgment, and ethics upon the writing of the results paper. Very few intervention studies go as planned, but the evolution of protocols over time is rarely captured in the subsequent manuscript because of space limitations, failure to capture or recall the deviations, and authors’ desire to present a clean depiction of the intervention (despite the inherent messiness of intervention work). As such, the reader is left with an incorrect impression about the implementation of the intervention, which can affect future attempts to replicate the findings in research and practice settings. This challenge is compounded by the fact that it is exceedingly rare for a journal to publish both the foundational, protocol paper and the results of these trials. Doing so would significantly ease the efforts to match the a priori design and hypothesis to the resulting trial.
In hopes of reducing these issues, the TJACSM is adopting the policy of publishing protocol papers that are submitted before intervention delivery and data collection. It is the belief of the editors that the reporting of planned activities related to the delivery of interventions, data collection, and data analyses will enhance transparency in the reporting of the main findings of the study. Preference and expedited review will be given to studies that have undergone external peer review by an awarding funding agency (e.g., the National Institutes of Health) or foundation (e.g., Robert Woods Johnson), but protocol manuscript submissions are welcome regardless of funding source. In addition, we encourage authors to publish both the pilot studies that lead to this protocol (5) and to return to the TJACSM with their subsequent results, regardless of the findings (6). In so doing, the authors will be able to share the evolution of their trial from conception to implementation, which will do much to aid in the advancement of translational science and the training of translational scholars.
The subsequent outcome papers should report deviations to the delivery of interventions, data collection procedures, and data analyses in a special section within the article. All too often, we view these deviations as an experimental failure. It is, instead, the inevitable process of translational research. It is therefore our sincere belief that the field of translational science will benefit from knowing how implementation and evaluation was necessarily tailored to the realities of the intervention setting and participants. Transparency, despite the vulnerability that comes with it, will enhance the utility of the resulting science.
Ultimately, publication of protocols is only one of the many tools that journals can use to attack the reproducibility crisis. Improved study design, utilization of reporting standards, data sharing, and other open science practices are also necessary. As we position the TJACSM for the future of scientific publication, new innovations will undoubtedly be necessary. However, the transparency of methods will be a cornerstone of efforts to accelerate the translation of evidence into action and the realization of meaningful changes to the healthy behaviors of the populations we serve.
1. Ioannidis JPA. Why most published research findings are false. In: Getting to Good: Research Integrity in the Biomedical Sciences
. Springer International Publishing; 2018, pp. 2–8.
2. Beets MW, Weaver RG, Ioannidis JPA, et al. Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: a systematic review and meta-analysis. Int J Behav Nutr Phys Act
3. Schulz JB, Cookson MR, Hausmann L. The impact of fraudulent and irreproducible data to the translational research crisis—solutions and implementation. J Neurochem
4. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature
5. Bartholomew JB, Campbell KL, Moore JB. The Potential and Peril of Pilot Research: Editorial Guidelines to Maintain Transparency and Reduce Overinterpretation of Effects. Transl J Am Coll Sports Med
. 2020;5(11). DOI 10.1249/TJX.0000000000000139.
6. Campbell KL, Moore JB, Bartholomew JB. The importance of publishing null results: Editorial Guidelines to contribute to the reduction of publication bias in translational exercise research. Transl J Am Coll Sports Med
. 2020;5(11). DOI 10.1249/TJX.0000000000000141.