Preregistration of Study Protocols Is Unlikely to Improve the Yield From Our Science, But Other Strategies Might : Epidemiology

Secondary Logo

Journal Logo

Proposal to Register Observational Studies: Commentary

Preregistration of Study Protocols Is Unlikely to Improve the Yield From Our Science, But Other Strategies Might

Lash, Timothy L.

Author Information
doi: 10.1097/EDE.0b013e3181e9bba6
  • Free

A recent workshop focused on the idea that protocols for nonrandomized studies should be registered in advance of their conduct.1 In reviewing the workshop report and publications citing it,2,3 I note a number of unfounded assumptions and little attention to competing alternatives.

First among these assumptions is that epidemiologic research lacks credibility. Whenever I hear this accusation, I remind the accusers that they brush their teeth, eat fortified foods, exercise, worry about their weight and their diet, refrain from smoking, buckle their seat belt, use safety equipment at work, put their babies to bed on their backs, and practice safe sex largely because of epidemiologic research. If our science is not credible, why does it affect almost every waking moment of our daily routine?

There is, however, room for improvement, so consider some of the assumptions that underlie the suggestion that preregistration of study protocols will improve the yield from our science (ie, ratio of advance in knowledge to resources expended). The impetus to preregister rests on the foundation that studies reporting results based on a priori hypotheses have a higher yield than studies reporting results based on hypotheses that arise during data collection and analysis.1,3,4 This assumption has been long debated,5,6 and compelling counter-arguments remain unanswered.7,8 For example, the credibility of the hypothesis confounds the association between whether it was stated in advance and the accuracy of research on the topic.9 If we limit research to topics with highly credible hypotheses stated a priori, we will stifle the generation of new ideas, and thus decrease our yield.

The proposed preregistration process suggests that peers and editors should compare submitted manuscripts with registered protocols as a component of review. If reviewers measure quality as a function of fidelity to the preregistered protocol, then they would implicitly assume that results of studies that deviate from a pre-existing protocol have a lower yield than studies that rigidly adhere to it. Protocol adaptations can improve recruitment, allow more accurate measurement of study variables, implement alternative analyses to control confounding, and incorporate new knowledge published by others. Such amendments would be discouraged by this review criterion, reducing the yield. This process also embeds assumptions that reviewers would actually compare the methods sections in submitted articles with pre-existing protocols, and that this comparison will improve the quality of review. I doubt it. A summary of the evidence regarding the quality of peer review written by a former editor of BMJ concluded that peer review is inconsistent, biased, and subject to abuse.10 Adding an opportunity for reviewers to complete reviews quickly and poorly will not improve the yield from our science.

Finally, to improve the yield from our science, the marginal gain in evidence from studies complying with the registration process would have to lead to a different consensus or to the same consensus reached earlier. This topic has not been well studied, but the only evidence I have found suggests that such improvement is unlikely.11 The workshop report recognized this absence of an evidence base, and recommended development of evidence to convince stakeholders about the need to preregister protocols.1(p. 18, section 6.3) This stakeholder finds it ironic that the workshop would recommend a post hoc search for evidence to support what is apparently a preordained conclusion.

More appealing and likely more effective alternatives to preregistration exist. If we view all study results as samples from the underlying truth about an association, then we must recognize that no evidence base will ever be complete, and that inferences and policies proceed despite this uncertainty. The important concern is whether the evidence base is incomplete and differentially selected. The only well-documented and clearly differential selection is the deficit of null small studies in most literature syntheses. This deficit arises from strong self-selection and review-selection forces that prevent publication of results that are not statistically significant.12–15 Access to these results can be obtained simply; stop the tyranny of significance testing16 for selection of articles to be published, and stop the charade of significance questing17,18 when analyzing data.

An analogous solution to improve the credibility of our science would be to stop promoting the results of individual studies in the media.19 Investigators, institutions, and journals that release news about single studies promote the view that one study should influence policy or behavior. Such a study comes along very infrequently.

I oppose preregistration of study protocols for 2 reasons: first, the idea that preregistration will improve the credibility and yield of our science rests on unfounded assumptions, and second, there are alternatives for achieving the same goals that do not rest on unfounded assumptions. I have not had the space here to describe the potential disadvantages of the registration process, but I suspect that the list is much longer than has been recognized.1–3

REFERENCES

1. European Center for Ecotoxicology and Toxicology of Chemicals. Workshop. Enhancement of the Scientific Process and Transparency of Observational Epidemiology Studies. Workshop Report No. 18. Available at: http://www.ecetoc.org/workshops.
2. Should protocols for observational research be registered? Lancet. 2010;375:348.
3. Loder E, Groves T, MacAuley D. Registration of observational studies. The next step towards research transparency. BMJ. 2010;340:375–376.
4. Swaen GG, Teggeler O, van Amelsvoort LG. False positive outcomes and design characteristics in occupational epidemiology studies. Int J Epidemiol. 2001;30:948–954.
5. Cole P. The hypothesis generating machine. Epidemiology. 1993;4:271–273.
6. Rothman KJ, Greenland S, Lash TL. Types of epidemiologic studies. In: Rothman KJ, Greenland S, Lash TL, eds. Modern Epidemiology. 3rd ed. Philadelphia: Lippincott, Williams, and Wilkins; 2008:99.
7. Hertz-Picciotto I. What you should have learned about epidemiologic data analysis. Epidemiology. 1999;10:778–783.
8. Michels KB, Rosner BA. Data trawling: To fish or not to fish. Lancet. 1996;348:1152–1153.
9. Savitz DA. Commentary: Prior specification of hypotheses: cause or just correlate of informative studies. Int J Epidemiol. 2001;30:957–958.
10. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99:178–182.
11. MacLean CH, Morton SC, Ofman JJ, Roth EA, Shekelle PG; Southern California Evidence-Based Practice Center. How useful are unpublished data from the Food and Drug Administration in meta-analysis? J Clin Epidemiol. 2003;56:44–51.
12. Moreno SG, Sutton AJ, Turner EH, et al. Novel methods to deal with publication biases: secondary analysis of antidepressant trials in the FDA trial registry database and related journal publications. BMJ. 2009;339:488–502.
13. Bowden J, Jackson D, Thompson SG. Modelling multiple sources of dissemination bias in meta-analysis. Stat Med. 2010;29:945–955.
14. Emerson GB, Brand RA, Heckman JD, Warme WJ, Wolf FM, Leopold SS. Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Presented at the: International Congress on Peer Review and Biomedical Publication; September 2009;Vancouver, Canada. Available at: http://www.ama-assn.org/public/peer/abstracts_2009.html#27.
15. Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer. 2007;43:2559–2579.
16. Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol. 2010;25:225–230.
17. Rothman KJ. Significance questing. Ann Intern Med. 1986;105:445–447.
18. Lash TL. Re: Insulin-like growth factor 1 and prostate cancer risk: a population-based, case-control study. J Natl Cancer Inst. 1998;90:1841.
19. Woloshin S, Schwartz LM, Casella SL, Kennedy AT, Larson RJ. Press releases by academic medical centers: not so academic? Ann Intern Med. 2009;150:613-618.
© 2010 Lippincott Williams & Wilkins, Inc.