Skip Navigation LinksHome > May 2008 - Volume 19 - Issue 3 > Impact Factor: Good Reasons for Concern
doi: 10.1097/EDE.0b013e31816b6a7a

Impact Factor: Good Reasons for Concern

Szklo, Moyses

Free Access
Article Outline
Collapse Box

Author Information

From the Johns Hopkins Bloomberg School of Public Health, Department of Epidemiology, The Johns Hopkins University, Baltimore, MD.

Submitted 8 January 2008; accepted 30 January 2008.

Editors' note: Related articles appear on pages 370, 372, and 373.

Correspondence: Moyses Szklo, Johns Hopkins Bloomberg School of Public Health, Department of Epidemiology, The Johns Hopkins University, 615 North Wolfe Street, Baltimore, MD 21205. E-mail: mszklo@jhsph.edu.

Collapse Box


This commentary emphasizes the importance of Hernán's contention that the impact factor's strong dependence on nonquality factors makes it utterly flawed as a way to evaluate quality of journals or papers.

In the current issue of Epidemiology, Hernán1 uses a fictional report of results from an epidemiologic study of epilepsy to highlight the serious limitations of Thomson Scientific's bibliographic impact factor (BIF). The impact factor has been widely used to evaluate the quality of individual papers and groups of investigators,2 and even as a criterion for academic promotion in many countries (in my observation). That this is wholly inappropriate is underscored by Hernan's pointed analogy between the epilepsy study and the BIF. The BIF's many limitations include its failure to adjust for self-citations and its dependence on the number of literature reviews a journal publishes3—both highlighted in Hernán's discussion of flaws in the numerator and denominator of the “brain irritability factor.” In addition, as underscored in Hernán's table, the calculation of the impact factor should—but does not—consider its strong relation with the size of the field of specialty. Thus, for example, the impact factor of any epidemiology journal is necessarily much lower than that of traditional general medicine journals, such as JAMA (impact factor, 23), The Lancet (17), or The New England Journal of Medicine (51). Factors such as size of the field and self-citations are clearly analogous to the comorbidities not adjusted for in the hypothetical epilepsy study—again bringing to mind the utter failure of the BIF as a marker of quality when comparing different journals.

The popularity of the impact factor as an index of quality probably results from the assumption that it is more difficult to have a paper accepted in a high than in a low-impact-factor journal, with the corollary that better papers are mostly accepted in the former. Two recent studies have underscored the tenuousness of this assumption: Kurmis and Kurmis4 showed a poor correlation between impact factors and initial rejection rate in radiology journals, and in a study by Callaham et al,5 citations of emergency medicine articles appeared to be more dependent on the impact factors of the journals in which they were published than on their subjective quality scores. (Callaham et al's findings should not come as a surprise to epidemiologists, as all papers, good or bad, “benefit” from being published along with a few frequently cited papers that contribute disproportionately to a journal's impact factor.)

Hernán's apt analogy strongly highlights Williams' contention6 that, in its present format, the impact factor should be killed off, and the sooner the better.

Back to Top | Article Outline


MOYSES SZKLO is a Professor of Epidemiology and Editor-in-Chief of the American Journal of Epidemiology. He is currently one of the Principal Investigators of the Multi-Ethnic Study of Atherosclerosis.

Back to Top | Article Outline


1.Hernán M. Epidemiologists (of all people) should question journal impact factors. Epidemiology. 2008;19:366–368.

2.Williams G. Misleading, unscientific, and unjust: the United Kingdom's research assessment exercise. BMJ. 1998;316:1079–1082.

3.Walter G, Bloch S, Hunt G, et al. Counting on citations: a flawed way to measure quality. Med J Aust. 2003;178:280–281.

4.Kurmis AP, Kurmis TP. Exploring the relationship between impact factor and manuscript rejection rates in radiologic journals. Acad Radiol. 2006;13:77–83.

5.Callaham M, Wears RL, Weber E. Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. JAMA. 2002;287:2847–2850.

6.Williams G. Should we ditch impact factors? BMJ. 2007;334:568.

Cited By:

This article has been cited 15 time(s).

Journal of the American College of Radiology
Distribution of Scholarly Publications Among Academic Radiology Departments
Morelli, JN; Bokhari, D
Journal of the American College of Radiology, 10(3): 190-194.
Journal of Child Neurology
"Worshiping false idols: The impact factor dilemma": Correcting the record - Response
Brumback, RA
Journal of Child Neurology, 23(9): 1092-1094.
American Journal of Respiratory Cell and Molecular Biology
Impact Factor and its Role in Academic Promotion
Leff, AR; Abraham, E; Adler, KB; Brusasco, V; Calverley, P; Dinh-Xuan, AT; Glenny, RW; Levy, ML; Panettieri, R; Reddel, HK; Singh, D; Virchow, JC
American Journal of Respiratory Cell and Molecular Biology, 41(2): 127.
Preventive Medicine
Don't Step on Our 2007 Citation Footprint (CF)
Morabia, A; Costanza, MC
Preventive Medicine, 47(4): 351-353.
Impact Factor and Its Role in Academic Promotion
Lee, YCG; Bardin, P
Respirology, 14(7): 914.
European Respiratory Journal
Impact factor and its role in academic promotion
Brusasco, V; Dinh-Xuan, AT; Leff, AR; Adler, KB; Glenny, RW; Dempsey, JA; Levy, ML; Panettieri, R; Reddel, HK; Singh, D; Virchow, JC
European Respiratory Journal, 34(6): 1499-1500.
Journal of Applied Physiology
Impact factor and its role in academic promotion: A statement adopted by the International Respiratory Journal Editors Roundtable
Dempsey, JA
Journal of Applied Physiology, 107(4): 1005.
Respiratory Medicine
Impact factor and its role in academic promotion
Virchow, JC
Respiratory Medicine, 103(): 1399.
Archives of Environmental & Occupational Health
The Historical Development of Academic Journals in Occupational Medicine, 1901-2009
Smith, DR
Archives of Environmental & Occupational Health, 64(): 8-17.

Archives of Environmental & Occupational Health
Citation Analysis and Impact Factor Trends of 5 Core Journals in Occupational Medicine, 1985-2006
Smith, DR
Archives of Environmental & Occupational Health, 63(3): 114-122.

Journal of Child Neurology
Impact Factor Wars: Episode V-The Empire Strikes Back
Brumback, RA
Journal of Child Neurology, 24(3): 260-262.
More on Impact Factors
Giuliani, F; De Petris, MP
Epidemiology, 19(5): 763.
PDF (240) | CrossRef
A Correction Regarding Bibliographic Impact Factors
Hernán, MA
Epidemiology, 20(5): 785.
PDF (50) | CrossRef
Impact Factor: A Call to Reason
Hernán, MA
Epidemiology, 20(3): 317-318.
PDF (84) | CrossRef
What Are We BIF-fing About?: Science Needs Impact Metrics
Colditz, IG; Colditz, GA
Epidemiology, 20(3): 462.
PDF (502) | CrossRef
Back to Top | Article Outline

© 2008 Lippincott Williams & Wilkins, Inc.

Twitter  Facebook 


Article Tools


Article Level Metrics