From the Department of Preventive Medicine, Keck School of Medicine and the Institute for Global Health, University of Southern California, Los Angeles, CA.
Editor's note: Related articles appear on pages 607, 612, 614, 616, and 619.
Correspondence: Jonathan M. Samet, Department of Preventive Medicine, Keck School of Medicine and the Institute for Global Health, University of Southern California, 1441 Eastlake Ave. Room 4436, MC 9175, Los Angeles, CA 90089. E-mail: firstname.lastname@example.org.
The report from the workshop held in September 2009, “Enhancement of the Scientific Process and Transparency of Observational Epidemiology Studies,”1 has already motivated substantial discussion, including this series of commentaries.2–6 Under discussion is the recommendation that observational studies should be registered using an approach comparable to that in place for clinical trials, requiring prospective registration. One motivation for registration is to address concerns as to the transparency of the flow of evidence that begins with study initiation and ends with publication. As a workshop organizer and participant, I welcome the surfacing of an issue that needs consideration, while I remain undecided as to the need for registration.
The workshop's second recommendation calls for the collection of “. . .empirical evidence for the need of registering observational epidemiological studies.” Arguably, this should have been the leading recommendation: the gathering and analysis of evidence to establish the dimensions and characteristics of the problem to be addressed through registration. We have long been aware of the problem of publication bias, which has been given in-depth consideration for some controversial exposure-disease associations, such as second-hand smoke and lung cancer (see the 2006 US Surgeon General's report7). We also have empirical evidence that stronger associations found in analyses of a given data set have greater likelihood of being published, as in the example of short-term studies of ozone.8 Periodically, commentaries have pointed to a purported tendency of epidemiologists to “cherry pick” the strongest associations and to then “hype” them (see Boffetta et al 20089 and Kabat 2008,10 for example). We know little, however, about how many observational studies have been initiated that could contribute evidence on a particular question, but are overlooked or set aside because of the implications of their findings. This problem—the “file-drawer” problem—needs characterization for observational studies, having already received considerable attention in relation to clinical trials. I concur with the workshop's second recommendation—empirical research is needed.
Is registration of observational studies needed as well? Lacking the pending empirical research, my judgment draws on whether registration can be anticipated to be largely of benefit, with little potential for harm, and further, on whether registration can be anticipated to address stakeholder concerns about using findings of epidemiologic research in decision-making.
Turning first to potential harm, how could the existence of registration have adverse consequences? A burdensome, demanding, or costly process could become an obstacle to initiating a study or to compliance with registration. Calling for a priori registration of hypotheses could stifle useful data exploration. Registration of unique cohorts might be suppressed to avoid acknowledgment of particular exposures, processes, or risks. One outcome might be a reluctance to carry out research on a potentially-hazardous exposure because even the existence of concern would be revealed through registration.
What are the countervailing benefits? Above all, if done correctly, registration would leave no doubt as to the extent of data potentially available on associations of a particular exposure with its consequences for human health. For many exposures, the findings of observational studies may have implications for a range of stakeholders—including those exposed and possibly harmed, those responsible for the exposure, regulators, lawyers, and politicians. Inevitably, if the interests of stakeholders are touched by findings of epidemiologic evidence, the validity of the research finding may be questioned. Points of concern often involve matters that would be covered by registration: Was the body of evidence considered complete? Was there suppression of findings or publication bias towards “positive” or “negative” results? Notably, the funding for the workshop came from the European Centre for Ecotoxicology and Toxicology of Chemicals, an industry-supported organization. While this organization had no direct input into the recommendations of the workshop participants, this funding by one key stakeholder raised concerns as to how registration might serve the interests of its sponsors. Presumably, studies undertaken by the commercial sector would also be subject to registration, enhancing transparency.
Beyond transparency, other benefits of registration can be anticipated. Registration would facilitate systematic reviews, pooled analyses and other collaborations. It should benefit researchers as they give consideration to initiating new studies, and it should provide a starting-point for data-sharing by listing the universe of studies on hand.
During the last decade, practices related to epidemiologic research have changed. Data-sharing is encouraged and even required in some circumstances11; some lines of research necessitate pooling of data and materials across multiple studies, often leading to large collaborative groups; fewer new populations are being studied; and clinical trials are registered. Should this evolution continue with the registration of observational studies?
1. ECETOC. Workshop: Enhancement of the scientific process and transparency of observational epidemiology studies. Workshop report no. 18. Brussels: European Centre for Ecotoxicology and Toxicology of Chemicals; 2009.
2. Sorensen HT, Rothman KJ. The prognosis for research. BMJ
3. Loder E, Groves T, Macauley D. Registration of observational studies. BMJ
4. Should protocols for observational research be registered? Lancet
5. Hemingway H, Riley RD, Altman DG. Ten steps towards improving prognosis research. BMJ
6. Vandenbroucke JP. Registering observational research: second thoughts. Lancet
7. US Department of Health and Human Services. The health consequences of involuntary exposure to tobacco smoke. A report of the Surgeon General. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention, Coordinating Center for Health Promotion, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health; 2006.
8. Bell ML, Dominici F, Samet JM. A meta-analysis of time-series studies of ozone and mortality with comparison to the national morbidity, mortality, and air pollution study. Epidemiology
9. Boffetta P, McLaughlin JK, La Vecchia C, Tarone RE, Lipworth L, Blot WJ. False-positive results in cancer epidemiology: a plea for epistemological modesty. J Natl Cancer Inst
10. Kabat GC. Hyping health risks: environmental hazards in daily life and the science of epidemiology. New York: Columbia University Press; 2008.
11. Samet JM. Data: to share or not to share? Epidemiology