Proposal to Register Observational Studies: Commentary
The idea for the registration of observational epidemiologic studies (with their protocols, hypotheses, and analysis plans) is modeled on the registration of randomized clinical trials (RCTs). The movement that led to the registration of RCTs was fed by distrust that important findings would be withheld—against the best interests of future patients. Randomized controlled trials test the end-product of a scientific process: whether a therapy should be applied to patients. Indeed, the application of a new therapy to hundreds of thousands of future patients should not depend on the whim of whomever decides what to publish and not to publish.
Science that aims to explain the workings of nature proceeds differently. A myriad of totally aimless mutations is needed for evolution to happen. For the same reason, scientific progress requires a myriad of scientists, all with subjective views of their own data and other people's data. Whenever a scientist has a new idea—upon seeing a patient, seeing the odd results of a laboratory experiment, or some strange data—she will try to find some quick confirmation. Scientists have a duty to pursue hypotheses. Sometimes they publish because they themselves do not have the means to pursue the idea in depth. Others respond with new experiments or reanalysis of data. Sometimes, it will be immediately obvious that progress was made; at other times a protracted debate may follow.1
One thread in the arguments favoring registration of observational studies is that conclusions from data that bear on a “prespecified hypothesis” are stronger than if the hypothesis was not specified before seeing the data. Although seemingly intuitive, logicians have a hard time proving this; the explanation a scientist arrives at might be exactly the same, whether thought of before or after seeing the data, so what's the difference?1–3
Explanations evolve while doing experiments, or doing data analysis, until things fit—the “eureka” moment. After the results of a new hereditary mutation came in, a fresh look at data from a study on venous thrombosis sparked a new question: why do we find the homozygotes for the mutation only in young women with venous thrombosis when the mutation's prevalence is neither age- nor sex-dependent?1 This led to new analyses about the interaction of the mutation and use of oral contraceptives—and a new line of research.4
A contrast between the regulatory and the explanatory uses of science may give the impression of pitting extremes against each other. Results from observational epidemiology are also used for regulatory purposes. There is the nub of the problem. The original document that sparked the debate on the registration of observational studies, resulted from a workshop organized by the European chemical industry.5 The industry feels that observational epidemiology generates too many false signals. Hence, the streak of “hyperscientific”6 argumentation in the document: “good science” should have a prespecified hypothesis and a prespecified analysis plan, and raw data should be made publicly available for the sake of openness.
Observational studies that incriminate environmental exposures lead to discussions about bias and confounding as alternatives to causation: was the potential bias in one study remedied by a reanalysis of other data? This process of arguing has nothing to do with the registration of hypotheses. The historical model is how the smoking-and-lung-cancer association was debated in 1959 by Cornfield et al—an historical paper that was recently reprinted with commentaries.7 Rival explanations are clearly specified, arguments and counterarguments are weighed, epidemiologic and basic-science data are judged in conjunction, and a conclusion is reached: there are still loose ends, but the totality of the evidence is a basis for action.
The unfounded rhetoric of epidemiology and “false positives” has been debunked before.8 The hyperscientific tendency of the original document is dangerous. If followed to the letter, for example, important drug safety measures would never have been taken in response to studies on adverse effects of drugs. The call for sharing “raw data” is a 2-edged sword. Reanalyzing data is a superb tactic to delay regulation. All data contain errors, and all data analysts make choices. When an opposing party gets hold of data, it will start a campaign denouncing the “errors” and the “choices.” Only the few people who intimately know the data can follow these arguments. Outsiders will conclude: “Well, even the experts do not agree, so we do not know what to do.” Bringing the public to that conclusion was the stated aim of the tobacco industry in their “doubt is our product” memo.9
The calls for registration of observational epidemiologic studies may have as their most important consequence the initiation of a public discussion to explain how science makes progress—not just in epidemiology, but also in other fields.
1. Vandenbroucke JP. Observational research, randomised trials, and two views of medical science. PLoS Med
2. Lipton P. Testing hypotheses: prediction and prejudice. Science
3. Brush SG. Accommodation or prediction? Science
4. Seligsohn U, Lubetsky A. Genetic susceptibility to venous thrombosis. N Engl J Med
5. Enhancement of the scientific process and transparency of observational epidemiology studies. European Centre for Ecotoxicology and Toxicology of Chemicals (ECETOC) Workshop report. Brussels, Belgium: ECETOC; 2009.
6. Vineis P. Viewpoint: the skeptical epidemiologist. Int J Epidemiol
7. Cornfield J, Haenszel W, Hammond EC, Lilienfeld AM, Shimkin MB, Wynder EL. Smoking and lung cancer: recent evidence and a discussion of some questions. 1959 [reprinted with commentaries]. Int J Epidemiol
8. Blair A, Saracci R, Vineis P, et al. Epidemiology, public health, and the rhetoric of false positives. Environ Health Perspect