Commentary: IMPACT FACTOR
This commentary emphasizes the importance of Hernán's contention that the impact factor's strong dependence on nonquality factors makes it utterly flawed as a way to evaluate quality of journals or papers.
From the Johns Hopkins Bloomberg School of Public Health, Department of Epidemiology, The Johns Hopkins University, Baltimore, MD.
Submitted 8 January 2008; accepted 30 January 2008.
Editors' note: Related articles appear on pages 370, 372, and 373.
Correspondence: Moyses Szklo, Johns Hopkins Bloomberg School of Public Health, Department of Epidemiology, The Johns Hopkins University, 615 North Wolfe Street, Baltimore, MD 21205. E-mail: firstname.lastname@example.org.
In the current issue of Epidemiology, Hernán1 uses a fictional report of results from an epidemiologic study of epilepsy to highlight the serious limitations of Thomson Scientific's bibliographic impact factor (BIF). The impact factor has been widely used to evaluate the quality of individual papers and groups of investigators,2 and even as a criterion for academic promotion in many countries (in my observation). That this is wholly inappropriate is underscored by Hernan's pointed analogy between the epilepsy study and the BIF. The BIF's many limitations include its failure to adjust for self-citations and its dependence on the number of literature reviews a journal publishes3—both highlighted in Hernán's discussion of flaws in the numerator and denominator of the “brain irritability factor.” In addition, as underscored in Hernán's table, the calculation of the impact factor should—but does not—consider its strong relation with the size of the field of specialty. Thus, for example, the impact factor of any epidemiology journal is necessarily much lower than that of traditional general medicine journals, such as JAMA (impact factor, 23), The Lancet (17), or The New England Journal of Medicine (51). Factors such as size of the field and self-citations are clearly analogous to the comorbidities not adjusted for in the hypothetical epilepsy study—again bringing to mind the utter failure of the BIF as a marker of quality when comparing different journals.
The popularity of the impact factor as an index of quality probably results from the assumption that it is more difficult to have a paper accepted in a high than in a low-impact-factor journal, with the corollary that better papers are mostly accepted in the former. Two recent studies have underscored the tenuousness of this assumption: Kurmis and Kurmis4 showed a poor correlation between impact factors and initial rejection rate in radiology journals, and in a study by Callaham et al,5 citations of emergency medicine articles appeared to be more dependent on the impact factors of the journals in which they were published than on their subjective quality scores. (Callaham et al's findings should not come as a surprise to epidemiologists, as all papers, good or bad, “benefit” from being published along with a few frequently cited papers that contribute disproportionately to a journal's impact factor.)
Hernán's apt analogy strongly highlights Williams' contention6 that, in its present format, the impact factor should be killed off, and the sooner the better.
ABOUT THE AUTHOR
MOYSES SZKLO is a Professor of Epidemiology and Editor-in-Chief of the American Journal of Epidemiology. He is currently one of the Principal Investigators of the Multi-Ethnic Study of Atherosclerosis.
1.Hernán M. Epidemiologists (of all people) should question journal impact factors. Epidemiology
2.Williams G. Misleading, unscientific, and unjust: the United Kingdom's research assessment exercise. BMJ
3.Walter G, Bloch S, Hunt G, et al. Counting on citations: a flawed way to measure quality. Med J Aust
4.Kurmis AP, Kurmis TP. Exploring the relationship between impact factor and manuscript rejection rates in radiologic journals. Acad Radiol
5.Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA
6.Williams G. Should we ditch impact factors? BMJ