Why Most Discovered True Associations Are Inflated

Ioannidis, John P. A.

doi: 10.1097/EDE.0b013e31818131e7
Interpretations: Original Article

Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated—for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results.

From the Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece; and Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts.

Submitted 17 March 2008; accepted 27 May 2008; posted 14 July.

Editors’ note: Related articles appear on pages 649, 652, 655, and 657.

Correspondence: John P. A. Ioannidis, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, 45 110 Ioannina, Greece. E-mail: jioannid@cc.uoi.gr.

Why most discovered true associations are inflated: Erratum

On page 645, right column, third line from the bottom, the number 256 should be 461. The same correction should be made in the legend for Figure 2.

© 2008 Lippincott Williams & Wilkins, Inc.