Neutra, Raymond Richard; Cohen, Aaron*; Fletcher, Tony†; Michaels, David‡; Richter, Elihu D.§; Soskolne, Colin L.¶
Controversy in the public health sciences has more often been resolved through replication rather than reanalysis and reinterpretation of existing data. Reanalysis and reinterpretation occur when a person other than the original investigator obtains an epidemiologic data set and conducts analyses to evaluate the quality, reliability or validity of the dataset, methods, results or conclusions reported by the original investigator.
Epidemiologic studies play a central role in proceedings in which government and international agencies formulate policies and regulations designed to protect public health. They are also important in the legal arena, assisting in the determination of the cause of an individual's illness or injury. Since the results and interpretation of epidemiologic studies can have significant financial implications, it is not surprising that parties who may be negatively impacted by these studies could be highly motivated to demand the opportunity to examine them closely.
Under U.S. law, investigators must provide the raw data from federally funded studies in response to requests under the Freedom of Information Act.1 The rationale for this provision, originally passed by the U.S. Congress in 1998, was to enable parties affected by government regulation to examine (and presumably challenge) the data used in developing these regulations. There are no equivalent legal requirements for access to raw data of studies that have been paid for with private funding, whether or not they are used in regulatory proceedings.2 For this reason, most reanalyses to date have been performed on publicly supported studies.
Reanalyses are often commissioned by commercial interests not in support of the results of a particular study. Many of these are not hypothesis-driven investigations; rather, they are undertaken with the explicit intention of influencing regulatory or legal proceedings. Examples of this are reanalyses of studies conducted by scientists at the National Institute for Occupational Safety and Health3,4 and the Environmental Protection Agency5 commissioned by producers of beryllium,6 petroleum,7,8 and chromium.9 In each of these examples, the reanalysis found reduced or no increased risk associated with the product manufactured by the sponsor of the reanalysis.
We can expect reanalysis to become an increasingly common practice, particularly for epidemiologic data that are relevant to policy. This is a potentially positive development if care is taken to ensure the fairness, impartiality, inclusiveness, and transparency of the reanalyses (see subsequently). In many instances, there is enough variation among accepted analytic strategies that it may be preferable to explore alternative analyses of an existing dataset rather than to gather new data at great cost for the sole purpose of applying other analytic strategies.
Government agencies that oversee policy formation should work to provide a “level playing field” so that stakeholders with differing interests and ideologic positions can assert their right to be part of the scientific and policymaking process. Indeed, stakeholders often do provide valuable insights. This is likely to make requests for reanalysis and reinterpretation more frequent both from the stakeholders themselves and from the government agencies that are responsible for an orderly application of science to public policy. An example of a process in which the reanalysis was performed in a transparent, responsible manner is the reanalysis and reinterpretation of the American Cancer Society and Harvard 6-cities air pollution studies10,11 organized by the Health Effects Institute.12 This new environment creates new technical and ethical responsibilities for original investigators, those who are tasked with funding and overseeing the reanalysis and reinterpretation, and the reanalyzers and reinterpreters themselves.
The Ethical Imperatives of Scientific Argumentation are Different From Those of Legal Argumentation
There are ethical implications in the way the reanalysis is carried out, in the way its results are interpreted for the scientific community, and in the ways it is summarized for peers, the general public, and for decision-makers. This becomes an ethical matter because the interests and rights of the challenging stakeholders, the original authors, and society at large may be different and need to be properly balanced. Furthermore, these ethics are different in crucial ways from those that guide the legal process.
If the reanalysis and reinterpretation of another researcher's work was merely part of a legal argument in an adversarial procedure before a jury or judge, then the goal would simply be to win according to fixed rules of evidence and procedure. The tobacco industry, in particular, as part of legal or regulatory proceedings, has used reanalysis and reinterpretation as part of an effort to “manufacture doubt” and to avoid increased regulation and victim compensation.13,14 Ethical concerns for the original author or parties outside the controversy are usually not paramount in such adversarial procedures.
In contrast, most original research is done as part of a process of rational scientific “argumentation.” The goal of such arguments is not to “win” one round of the argument, but to converge toward the truth in hopes that this will be of use to society at large. There is also a hope that the argument will be conducted in a manner that preserves the ability of the scientific community to have other fruitful scientific arguments in the future. The guidelines we propose reflect those scientific commitments.
Because of the differences between the goals of legal/regulatory arguments and the goals of scientific arguments, lawyers and judges may not see the need for the procedures proposed here. Their professional focus is on legal argumentation and the resulting decision, not its aftermath. From the perspective of the scientific community and public policy, however, it is helpful to avoid extremely stated, “dueling analyses” that are cleverly “spun” so as to influence a jury or an administrative law judge. Instead, these draft guidelines aim to preserve the scientific process and its ongoing commitment to converging toward scientific truth. We should consider, then, the pros and cons of reanalysis and reinterpretation, and possible guidelines for conducting them in an ethical manner.
The Advantages of Reanalysis and Reinterpretation
There are many positive reasons for carrying out reanalysis and reinterpretation. It is widely assumed that impartial truth-telling and open-mindedness are virtues central to science in general and to results-oriented social policy in particular. It follows that one should be open to competent impartial reanalysis. Reasonable people working in the public interest may differ as to the proper strategy for analyzing an epidemiologic study, or what elements of the pattern of evidence are emphasized, or what summary conclusions can be drawn.
There is the potential to improve health and social policy by changing or refining the results and interpretations. For example, reanalysis can correct the exaggeration or false minimization of the prevalence of problems, or it can adjust the weight given to an inference of causation. There is always a possibility of discovering falsification or fabrication of data.
Procedural and Ethical Recommendations Supporting Reanalysis and Reinterpretation
We propose the following ethical guidelines for investigators and research funding agencies regarding the reanalysis and reinterpretation of their data.
1. Data from epidemiologic studies should be available for impartial reanalysis and reinterpretation regardless of whether the study was funded by public monies or by groups with particular interests or ideologies.
2. The original authors have a responsibility to cooperate with, and facilitate, impartial and competent reanalysis and reinterpretation of their data.
3. The sponsors of research must provide budgets to pay for maintaining adequate log books that document the entire project, including coding manuals and strategies used in analysis. They should also pay for creating archives for such records and for the records' retrieval when required.
4. All appropriate and necessary measures should be taken to ensure that the confidentiality and privacy of participants in the original study be preserved throughout the reanalysis, including scrupulous adherence to agreements that may have been undertaken between the original investigators and participants or between the original investigators and providers of data.
5. There should be an adequately funded agency to store the documented data beyond the lifetime of the original investigators, and to oversee and fund the procedural aspects of the process of reanalysis and reinterpretation to ensure its transparency, competence, inclusiveness, and impartiality.
Pitfalls of Reanalysis and Reinterpretation
There are potential pitfalls in the reanalysis and reinterpretation process. It is necessary to balance the interests and rights of the challenging stakeholders, the original researchers, and of society at large. First of all, the resources available for reanalysis, reinterpretation, and dissemination are unequally available to stakeholders in society. There are situations in which political pressure is exerted by powerful stakeholders to create a process for reanalysis that serves the interests of one set of stakeholders over those of another. Groups with special interests or ideologies can select scientists who will share their biases or are willing to analyze and interpret data in ways that serve those interests and ideologies. There are situations in which incompetent persons want to undertake a reanalysis and reinterpretation. Using the reanalysis process to “manufacture doubt” can serve the interests of groups with special interests and ideologies.
Engaging indiscriminately or maliciously in reanalysis and reinterpretation, or with the intent to “manufacture doubt,” can create a climate in which scientists decide to avoid policy-relevant research in which powerful stakeholders have an interest. The ways in which reanalysis and reinterpretation are done can unjustifiably jeopardize the reputation of the original author. Even under the best of conditions, they represent an unfunded opportunity cost for the original authors by requiring their help in facilitating reanalysis and by necessitating a response to the reinterpretation of their data. Unfair practices in pursuing scientific controversy often stir up emotions in ways that interfere with the clarification of the true state of affairs.
Finally, one can create confusion and “manufacture doubt” in a variety of ways. For example, one can be unclear about the specific alternative hypotheses that should be explored in the reanalysis or the extent to which different patterns of evidence would warrant an increased or decreased support for those hypotheses. Sometimes, a reanalysis is declared “inconclusive” when it never had a chance of being conclusive in the first place.
Ethics Guidelines to Protect the Fairness, Impartiality, Inclusiveness, and Transparency of Reanalysis and Reinterpretation
The previously mentioned pros and cons for reanalysis or reinterpretation suggest that more than mere availability of data and data documentation is required. The sponsor of the reanalysis, the agency that oversees it, and the epidemiologist who will conduct the reanalyisis all have ethical responsibilities. These include:
1. A process that leaves the analyst free of actual or apparent conflicts of interests.
2. Open communication and fair and respectful dealings with the original investigators. Those conducting the reanalysis should create an environment of collegial yet critical truth-finding and provide the original researchers the opportunity to comment on the methods chosen for reanalysis before it takes place as well as on the interpretation of the results of any new analysis. If there are differences in interpretation, all parties should respectfully explain the reasons for these differences in written form so that third parties can fairly draw their own conclusions.
3. Creation of an independent advisory structure for the reanalysis and reinterpretation that can correct unintended bias in the analysis and in the scientific and public reinterpretations, and assure that the persons chosen to do the reanalysis and reinterpretation are professionally competent to do so.
4. Procedures that afford equal opportunity for stakeholders and their scientific advisors on various sides of the issue to comment for the public record on the reanalysis and reinterpretation.
5. Agreement ahead of time as to which hypotheses will be explored and on the extent to which different patterns of evidence would support or not support each of the different hypotheses.
6. Avoidance of activities that obfuscate the facts, harass epidemiologists who have presented unwelcome results, or intimidate future investigators from working in this area.
7. Assurance ahead of time that the results of the reanalysis and reinterpretation will be widely available regardless of the result.
8. Acknowledgment (when appropriate) of the role of the original investigators in the development of the original methods and instruments and in the collection of the data. The process must protect their ability to publish their work within a reasonable time before any reanalysis and reinterpretation is published.
9. A reasonable budget to facilitate the involvement of the original investigators if the reanalysis and reinterpretation has outside funding.
10. Publication of the reanalysis and reinterpretation in a way that respectfully clarifies the factual grounds, the scientific claims, the inferential assumptions that warrant those claims, and the reasons behind any differences in interpretation.
11. Resistance to political or other pressures to deviate from the previously mentioned guidelines.
The International Society for Environmental Epidemiology (ISEE) has previously proposed “Ethics Guidelines for Environmental Epidemiologists” that deals with obligations to research subjects, society at large, sponsors and employers, and colleagues.15 The present draft guidelines deal with the ethical duties of those who sponsor and carry out a reanalysis and reinterpretation of data collected, analyzed, and reported by another researcher. Our purpose is to generate discussion and stimulate suggestions for their improvement and ultimate adoption by ISEE. We encourage letters to the editor of this journal and correspondence to us through email@example.com.
1. US Office of Management and Budget, Uniform Administrative Requirements for Grants and Agreements With Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations. Circular A-110, Sept 30, 1999. Available at: http://www.whitehouse.gov/omb/circulars/a110/a110.html
. Accessed January 20, 2006.
2. Wagner W, Michaels D. Equal treatment for regulatory science: extending the controls governing the quality of public research to private research. Am J Law Med
3. Ward E, Okun A, Ruder A, et al. A mortality study of workers at seven beryllium processing plants. Am J Ind Med
4. Infante PF, Rinsky RA, Wagoner JK, et al. Leukemia in benzene workers. Lancet
5. Gibb HJ, Lees PSJ, Pinsky PF, et al. Lung cancer among workers in chromium chemical production. Am J Ind Med
6. Levy PS, Roth HD, et al. Beryllium and lung cancer: a reanalysis of a NIOSH cohort mortality study. Inhal Toxicol
7. Paustenbach DJ, Price PS, et al. Reevaluation of benzene exposure for the Pliofilm (rubberworker) cohort (1936–1976). J Toxicol Environ Health
8. Paustenbach DJ, Bass RD, Price P. Benzene toxicity and risk assessment, 1972–1992: implications for future regulation. Environ Health Perspect
. 1993;101(suppl 6):177–200.
9. Reanalysis of Lung Cancer Mortality Study for Workers in the Baltimore Chromium Production Facility
. Irvine, CA: Exponent; November 2002.
10. Dockery D, Pope CA, et al. An association between air pollution and mortality in six US cities. N Engl J Med
11. Pope CA III, Thun MJ, Namboodiri MM, et al. Particulate air pollution as a predictor of mortality in a prospective study of US adults. Am J Respir Crit Care Med
12. Health Effects Institute. Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of Particulate Air Pollution and Mortality. A Special Report of the Institute's Particle Epidemiology Reanalysis Project. Final version, July 2000. Available at: http://www.healtheffects.org/Pubs/Rean-Statement.pdf
13. Baba A, Cook DM, McGarity TO, et al. Legislating “sound science”: the role of the tobacco industry. Am J Public Health
. 2005;95(suppl 1):S20–27.
14. Michaels D, Monforton C. Manufacturing uncertainty: contested science and the protection of the public's health and environment. Am J Public Health.
© 2006 Lippincott Williams & Wilkins, Inc.