Judging from the recent spate of articles about false-positive results in epidemiology, initiated by Ioannidis’s thought-provoking piece in our July issue , something seems wrong with our field. Still, the mainstay of epidemiologic reasoning is making comparisons. Hence the question: has our field more problems than other fields of medical science?
Let’s start with randomized trials. And, just as critics of epidemiology focus on specific fields (say, nutritional and life style), let’s also focus – for example, on antidepressants. John Ioannidis produced another thought-provoking paper, entitled: “Effectiveness of antidepressants; an evidence based myth constructed from a thousand controlled trials."  Right, that seems to settle the score with epidemiology.
Would this problem exist for anti-depressant randomized trials only? The people who first signalled the selectiveness of the evidence construction on antidepressants were employees of the Swedish drug licensing authorities. They stated that in their experience the problem is not confined to this class of drugs.  Just read the extremely funny paper about a new class of antipsychotics which showed that in head-to-head randomized trials Drug A was better than B, B better than C, and in turn C better than A – all for the same indications!  The outcome just depended on the sponsor. All in all, should we be mistaken in concluding that the field of randomized trials has as many problems with its credibility as epidemiology?
Let’s continue with Genome Wide Association Studies (GWAS). A paper in The Economist called the whole genomic scene into question with the title: “The looming crisis in human genetics” with subheading: “They have simply not been delivering the goods”.  The crucial sentence: “Even when they do replicate, they do never explain more than a tiny fraction of any interesting trait”.
Of course, we, epidemiologists knew this all along. Let’s focus on cancer. In a 1988 (!) study of adopted children and their biological and adoptee parents in Scandinavia,  the relative risk of developing any malignancy by children whose biological parent had developed a malignancy before age 50, but who had been educated in adoptive families, was 1.2 – while the relative risk if the adopting parent had developed cancer became 5-fold. A later editorial made clear that only rather weak concordance of cancer was found in twin studies.  Mere logic should have alerted us that it is impossible to find strong explanations of complex diseases by single somatic genes: these diseases come into being because of multiple pathways that go wrong, and each pathway can go wrong in multiple ways. Of course, there are rare families where cancer is hereditable – these have been detected, and that was it.
We pass the fact that almost none of those great GWAS discoveries has yet resulted into anything meaningful clinically or to public health – in great contrast to the record of epidemiology. And, as the Economist wrote, the yield of classic genetics has been much higher than GWAS, and clinically much more important. My only (but rather successful) brush with genetics concerned a mutation that was quite prevalent with a high relative risk, and that was discovered by first elucidating the biochemical abnormality and thereafter reasoning backwards to the gene.  GWAS would never have found it. If Ioannidis calls antidepressant randomized trials a well-constructed myth, should we not call GWAS the same?
For almost any field of medical science, we can point out easily that its published record must contain a massive amount of irrelevance and error. This is nothing to worry about. It is normal science – it has always been like that and it will continue to be so. Again, John Ioannidis comes to the rescue and proves the point. A personal parenthesis first: for years, during lectures, I had been telling audiences that if you want to understand how science evolves you should go to the library and look at The Lancet or BMJ or NEJM or JAMA of 50 years ago, or better 100 years ago – most of the papers you cannot understand anymore, and the rest are either irrelevant or plainly wrong. I only did the thought experiment, and never even published it, but John Ioannidis and his collaborators gathered real data, and confirmed my prejudices. In a paper entitled: “Fifty-year fate and impact of general medical journals”,  they write “Only 226 of the 5,223 papers published in 1959 were cited at least once in 2009 and only 13 of them received at least 5 citations in 2009.” They were mostly clinical papers, describing syndromes.
Perhaps, with the latter publication Ioannidis is biting his own tail. I am thinking about his 2005 “Why most published research findings are false”.  In that paper he seemed to single out observational epidemiology as the main culprit. Judging from his later judgement about myth creating by ‘a thousand RCTs’ and his recent judgement about the transiency of all science, the ultimate question becomes whether all scientific processes can be improved – not just epidemiology - so as to be less wasteful and yield more often truth.
Having just returned from the 3rd North American Congress of Epidemiology in Montreal, I started wondering whether it is epidemiologists who are most acutely aware of the tentativeness of any scientific finding. At that congress you could go from one session to the other hearing about problems of data, analysis and inference and listen to plenary lectures about wrong turns in our science. Would the same happen at, say, a congress of cardiologists? Or geneticists?
In 1906 Sir William Osler delivered an Harveian oration about“The growth of truth” in which he wrote about the vagaries of truth, and the many detours and false alleys of scientific research: “Truth may suffer all the hazards incident to generation and gestation…”.  His views were echoed almost 100 years later in a Millenium essay by Stephen Jay Gould who described how science progresses “in a fitful and meandering way”.  Perhaps the wastefulness of science is inevitable and might be compared to the zillions of meaningless mutations happening in biological systems – very few of which might carry any survival advantage.
One day, when discussing a paper with our PhD students, one asked in exasperation: “How can you ever be certain that a paper is true”. My spontaneous answer was: “Grow 25 years older – and even then…”.
If you like to comment, Email me directly at email@example.com or submit your comment via the journal which requires a password protected login. Unfortunately, comments are limited to 1000 characters.
 Ioannidis JP, Tarone R, McLaughlin JK. The False-positive to False-negative Ratio in Epidemiologic Studies. Epidemiology. 2011;22:450-6.
 Ioannidis JP. Effectiveness of antidepressants: an evidence myth constructed from a thousand randomized trials? Philos Ethics Humanit Med. 2008;3:14.
 Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003;326:1171-3
 Heres S, Davis J, Maino K, et al.. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. Am J Psychiatry. 2006;163:185-94.
 Miller G. The looming crisis in human genetics. In: The World in 2010. The Economist, November 13, 2009: page 150-51.
 Sørensen TI, Nielsen GG, Andersen PK, Teasdale TW. Genetic and environmental influences on premature death in adult adoptees. N Engl J Med. 1988;318(12):727-32.
 Hoover RN. Cancer--nature, nurture, or both. N Engl J Med. 2000;343:135-6.
 Vandenbroucke JP, Rosendaal FR, Bertina RM. Factor V Leiden, Oral Contraceptives and Deep Vein Thrombosis. IN: Khoury JM, Little J, Burke W. Human Genome Epidemiology. New York, Oxford Univ Press 2004: 322-332.
 Ioannidis JP, Belbasis L, Evangelou E. Fifty-year fate and impact of general medical journals. PLoS One. 2010;5(9). pii: e12531.
 Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124.
 Osler W. Harveian oration. The growth of truth, as illustrated in the discovery of the circulation of the blood. BMJ 1906;ii:1077–1084.
 Gould SJ. Pathways of discovery: deconstructing the “science wars” by reconstructing an old mold. Science 2000:287:253–261.
© Jan P Vandenbroucke, 2011