Impact Factor: Good Reasons for Concern

This commentary emphasizes the importance of Hernán's contention that the impact factor's strong dependence on nonquality factors makes it utterly flawed as a way to evaluate quality of journals or papers.

In the current issue of Epidemiology, Hernán¹ uses a fictional report of results from an epidemiologic study of epilepsy to highlight the serious limitations of Thomson Scientific's bibliographic impact factor (BIF). The impact factor has been widely used to evaluate the quality of individual papers and groups of investigators,² and even as a criterion for academic promotion in many countries (in my observation). That this is wholly inappropriate is underscored by Hernan's pointed analogy between the epilepsy study and the BIF. The BIF's many limitations include its failure to adjust for self-citations and its dependence on the number of literature reviews a journal publishes³-both highlighted in Hernán's discussion of flaws in the numerator and denominator of the brain irritability factor. In addition, as underscored in Hernán's table, the calculation of the impact factor should-but does not-consider its strong relation with the size of the field of specialty. Thus, for example, the impact factor of any epidemiology journal is necessarily much lower than that of traditional general medicine journals, such as JAMA (impact factor, 23), The Lancet (17), or The New England Journal of Medicine (51). Factors such as size of the field and self-citations are clearly analogous to the comorbidities not adjusted for in the hypothetical epilepsy study-again bringing to mind the utter failure of the BIF as a marker of quality when comparing different journals.

The popularity of the impact factor as an index of quality probably results from the assumption that it is more difficult to have a paper accepted in a high than in a low-impact-factor journal, with the corollary that better papers are mostly accepted in the former. Two recent studies have underscored the tenuousness of this assumption: Kurmis and Kurmis⁴ showed a poor correlation between impact factors and initial rejection rate in radiology journals, and in a study by Callaham et al,⁵ citations of emergency medicine articles appeared to be more dependent on the impact factors of the journals in which they were published than on their subjective quality scores. (Callaham et al's findings should not come as a surprise to epidemiologists, as all papers, good or bad, benefit from being published along with a few frequently cited papers that contribute disproportionately to a journal's impact factor.)

Hernán's apt analogy strongly highlights Williams' contention⁶ that, in its present format, the impact factor should be killed off, and the sooner the better.

ABOUT THE AUTHOR

MOYSES SZKLO is a Professor of Epidemiology and Editor-in-Chief of the American Journal of Epidemiology. He is currently one of the Principal Investigators of the Multi-Ethnic Study of Atherosclerosis.