Secondary Logo

Journal Logo


Seeking Persuasively Null Results

Lash, Timothy L.a; Kaufman, Jay S.b

Author Information
doi: 10.1097/EDE.0000000000000318
  • Free

Those who recently submitted an article to EPIDEMIOLOGY may have noticed revised Instructions for Authors. Among the revisions are a reduced number of article categories, clearer statements about our expectations regarding the precision and validity of estimates of effect, and encouragement that authors should include quantitative analyses to evaluate the uncertainty arising from both random error and systematic error. Combining considerations of each of these revisions is our explicit welcome to “papers that present precisely measured persuasively null results.” Here, we expand upon this invitation.

Publication bias has been a long-standing concern in qualitative and quantitative syntheses of epidemiologic evidence.1 As explained by Begg in 1985,2 the concern is that studies in which the observed estimate of effect is distant from the null may be more likely to be published than studies in which the observed estimate of effect is near the null. Summaries of the effect estimate will therefore lie further from the null than the true effect; the distance between the summary and the truth equals the bias due to selective publication. Proposed solutions have included statistical tests to detect the bias,3,4 methods to adjust summary estimates to account for the bias,1,5,6 exhortations to authors, reviewers, and journals to publish null results,7 prior registration of studies so that relevant evidence can be identified even if unpublished,8–12 repositories for submission of null results,13 and special submission categories at some journals14–16—or even entire journals17,18—reserved for null findings.

EPIDEMIOLOGY does not intend to pursue any of these solutions, but rather to continue to evaluate null results just as any other research submitted for consideration by the journal. Nonetheless, we feel compelled to offer this sincere invitation to directly address the main barrier to the publication of null results: authors are shy about submitting them for publication, more so than editors are shy about publishing them.19–21 Part of this reticence may stem from confusion about which null results are most likely to be published in EPIDEMIOLOGY.

Our invitation welcomes most warmly null results that challenge prior beliefs—specified qualitatively or quantitatively—for the location or width of the study’s effect estimate. Our preference for null results of this type explains why our full invitation emphasizes that we seek “precisely measured persuasively null results for which either prior data or a compelling rationale exists for a nonnull effect.” Results of this type have an interval narrow enough, or distant enough, from the prior to call it into question. We expect that biases toward the null have been rendered unlikely by the design or conventional data analysis, or have been evaluated quantitatively by bias analysis methods. At EPIDEMIOLOGY, a persuasively null result is therefore evaluated by consideration of a qualitatively or quantitatively specified prior for the study’s effect, the result, and the influence of random and systematic errors. We expect authors to explain why they think that their result is more compelling than the sum of the previous literature, or the biologic rationale, which might entail detailing the biases or limitations that plagued the previous studies but which have been overcome in the submitted manuscript.

All credible scientific research endeavors and their results merit public access, but no single journal can promise to enable that universal access. We encourage authors to use Supplemental Digital Content as an opportunity to present results that do not receive their focus in the published manuscript. The use of online only appendices is one strategy to answer calls for publication of all results22 or all analyses.23 We also encourage authors to calibrate the length of their submissions to the importance of the results they report. EPIDEMIOLOGY offers research letters, brief reports, and original research articles as submission categories to present peer-reviewed research articles, each with the option of appending Supplemental Digital Content. Submissions that use the available space wisely, whether null or not, are most likely to be invited to stay.


1. Sterling TD.. Publication decisions and their possible effects on inferences drawn from tests of significances. Am Stat Assoc J. 1959;54:30–34
2. Begg CB.. A measure to aid in the interpretation of published clinical trials. Stat Med. 1985;4:1–9
3. Egger M, Davey Smith G, Schneider M, Minder C.. Bias in meta-analysis detected by a simple, graphical test. Br Med J. 1997;315:629
4. Begg CB, Mazumdar M.. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50:1088–1101
5. Terrin N, Schmid CH, Lau J, Olkin I.. Adjusting for publication bias in the presence of heterogeneity. Stat Med. 2003;22:2113–2126 Erratum in: Stat Med. 2005;24:825–826
6. Henmi M, Copas JB, Eguchi S.. Confidence intervals and P-values for meta-analysis with publication bias. Biometrics. 2007;63:475–482
7. Thornton A, Lee P.. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol. 2000;53:207–216
8. De Angelis C, Drazen JM, Frizelle FA, et al.International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351:1250–1251
9. Krleza-Jerić K, Chan AW, Dickersin K, Sim I, Grimshaw J, Gluud C.. Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ. 2005;330:956–958
10. Swaen GM, Carmichael N, Doe J.. Strengthening the reliability and credibility of observational epidemiology studies by creating an Observational Studies Register. J Clin Epidemiol. 2011;64:481–486
11. Williams RJ, Tse T, Harlan WR, Zarin DA.. Registration of observational studies: is it time? CMAJ. 2010;182:1638–1642
12. The Editors. . The registration of observational studies—when metaphors go bad. Epidemiology. 2010;21:607–609
13. Prechelt L.. Why we need an explicit forum for negative results. J Univers Comput Sci. 1997;3:1074–1083
14. Dirnagl U, Lauritzen M.. Fighting publication bias: introducing the Negative Results section. J Cereb Blood Flow Metab. 2010;30:1263–1264
15. Shields PG.. Publication bias is a scientific problem with adverse ethical outcomes: the case for a section for null results. Cancer Epidemiol Biomarkers Prev. 2000;9:771–772
16. . Missing pieces. Plos One. Available at: Accessed February 27, 2015
17. Pfeffer C, Olsen BR.. Editorial: Journal of negative results in biomedicine. J Negat Results Biomed. 2002;1:2
18. The Journal of Articles in Support of the Null Hypothesis. Available at: Accessed February 24, 2015
19. Dickersin K, Min YI.. Publication bias: the problem that won’t go away. Ann N Y Acad Sci. 1993;703:135–146; discussion 146
20. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008;3:e3081
21. Chalmers I, Dickersin K.. Biased under-reporting of research reflects biased under-submission more than biased editorial rejection. F1000Res. 2013;2:1
22. Demaria AN.. Publication bias and journals as policemen. J Am Coll Cardiol. 2004;44:1707–1708
23. Poole C.. A vision of accessible epidemiology. Epidemiology. 2010;21:616–618
Copyright © 2015 Wolters Kluwer Health, Inc. All rights reserved.