Secondary Logo

Journal Logo

Commentary

Beyond the Impact Factor: An Alternative Approach to Measure the Influence of the Journal Sexually Transmitted Diseases on Sexually Transmitted Infection Prevention and Practice

Rietmeijer, Cornelis A. MD, PhD, MSPH; Miller, William C. MD, PhD, MPH

Author Information
Sexually Transmitted Diseases: July 2022 - Volume 49 - Issue 7 - p 459-461
doi: 10.1097/OLQ.0000000000001628

A journal's impact factor is the yardstick by which the journal's scientific impact is commonly measured and compared with other journals. The impact factor is calculated by dividing the number of citations in a given year to items published in a given journal in the previous 2 years, divided by the total number of articles published by that journal in the previous 2 years.1 In 2020, the impact factor for the journal Sexually Transmitted Diseases was 2.8, earning the journal the position of 60 in the category of infectious disease journals (n = 106 journals). Higher-impact journals included the Lancet Infectious Diseases (rank, 1; impact factor, 25.1), Clinical Infectious Diseases (rank, 3; impact factor, 9.0), Journal of Infectious Diseases (rank, 18; impact factor, 5.2), and AIDS (rank, 32; impact factor, 4.1).

Measuring impact this way has important consequences. For example, individual investigators, particularly young investigators, may be dissuaded from submitting their work to journals with a low impact factor believing that it might influence judgment of their academic progress, only to submit to lower-impact journals if they consider their work to not be competitive or when it is rejected by journals with a higher impact factor. This behavior often creates a vicious circle for a journal as publishing second-tier articles results in a second-tier journal that continues to attract second-tier articles.

The impact factor, though, has important limitations. Although it may say something meaningful about the standing of a journal within the scientific community, its influence on practice is less clear, as practitioners may use innovations described in the literature to change practice without necessarily contributing to the literature themselves and thus not adding citations to the calculation of the impact factor. The impact factor may also be influenced by biases, including self-citation.

The Sexually Transmitted Infections (STI) Treatment Guidelines*, published periodically by the US Centers for Disease Control and Prevention (CDC), is arguably the single most authoritative publication affecting the prevention, diagnosis, and treatment of STI in the United States and may also have considerable influence on STI practice in other countries. The guidelines are the product of a multidisciplinary collaboration, involving leaders in the field of STI prevention, diagnosis, and treatment that, every 5 to 6 years, review the literature, write background papers, and finally come together for a 2-day meeting, following which CDC staff prepares and publishes the new guidelines, usually but not universally reflecting the opinion of the external consultants.

The latest guidelines were published in 20152 and 2021.3 Since the publication and online availability of the 2021 guidelines in July 2021, the CDC guidelines Web site has attracted between 300,000 and 350,000 visits per month (CDC, unpublished data).

We postulated that a review of the literature cited in these guidelines would identify the most important sources of evidence on which the guidelines are built and thus provide an indicator of scientific and practical significance beyond the impact factor.4 In this article, we examined both the 2015 and 2021 guidelines. As, respectively, an associate editor and the editor in chief of the journal Sexually Transmitted Diseases, we were particularly interested in how our journal would fare in such an analysis.

METHODS

We abstracted sources and year of publication from numbered references in both the 2015 and 2021 STI Treatment Guidelines. We tabulated the publication sources by frequency for all references in each of the 2 guidelines and then repeated the tabulation when limiting the year of publication to a 5-year time interval before publication, corresponding with the interval between guideline publications. All data analyses were conducted in SAS On-Demand for Academics (https://www.sas.com/en_us/software/on-demand-for-academics.html).

RESULTS

The 2015 treatment guidelines had 892 numbered references compared with 1455 in the 2021 guidelines, an increase of 63.1%. The 2015 guidelines listed 188 individual reference sources compared with 304 individual sources for the 2021 guidelines. As shown in Table 1, the top 20 reference sources accounted for 65.9% and 62.8% of all references in the 2015 and 2021 guidelines, respectively. For reference, we also included the 2020 impact factors in Table 1.

TABLE 1 - Citation Sources Listed in the 2015 and 2021 Centers for Disease Control and Prevention (CDC) Sexually Transmitted Infections Treatment Guidelines—Ranked by Frequency
Rank All Citations Citations Wihin Previous 5 y
2015 2021 2015 2021
Source n (%) Source n (%) 2020 Impact Factor Source n (%) Source n (%)
1 Sex Transm Dis 125 (14.0) Sex Transm Dis 186 (12.8) 2.8 Sex Transm Dis 47 (13.9) Sex Transm Dis 48 (9.9)
2 Clin Infect Dis 67 (7.5) Clin Infect Dis 119 (8.2) 9.0 Clin Infect Dis 23 (6.8) Sex Transm Infect 36 (7.4)
3 Sex Transm Infect 50 (5.6) Sex Transm Infect 93 (6.4) 3.5 Sex Transm Infect 18 (5.3) Clin Infect Dis 34 (7.0)
4 J Infect Dis 44 (4.9) J Infect Dis 50 (3.4) 5.2 Morb Mort Wkly Rep 17 (5.0) Morb Mort Wkly Rep 24 (4.9)
5 Morb Mort Wkly Rep 34 (3.8) Int J STD AIDS 46 (3.2) 1.3 J Infect Dis 11 (3.2) JAMA 19 (3.9)
6 Obstet Gynecol 31 (3.5) Morb Mort Wkly Rep 41 (2.8) 50.0 PLoS One 10 (2.9) Int J STD AIDS 18 (3.7)
7 Am J Obstet Gynecol 27 (3.0) Obstet Gynecol 39 (2.7) 7.6 Ann Intern Med 9 (2.7) Emerg Infect Dis 12 (2.5)
8 N Engl J Med 26 (2.9) N Engl J Med 38 (2.6) 91.2 J Clin Microbiol 9 (2.7) PLoS One 10 (2.1)
9 Ann Intern Med 22 (2.5) JAMA 36 (2.5) 56.3 CDC 9 (2.7) J Clin Microbiol 9 (1.9)
10 Int J STD AIDS 22 (2.5) J Clin Microbiol 33 (2.3) 4.1 In J STD AIDS 8 (2.4) AIDS 9 (1.9)
11 J Clin Microbiol 19 (2.1) Am J Obstet Gynecol 31 (2.1) 8.1 N Engl J Med 7 (2.1) Lancet 9 (1.9)
12 AIDS 17 (1.9) PLoS One 31 (2.1) 3.2 AIDS 6 (1.8) J Allerg Clin Immunol 9 (1.9)
13 Lancet 16 (1.8) AIDS 29 (2.0) 4.1 Pediatrics 6 (1.8) Obstet Gynecol 8 (1.8)
14 CDC 15 (1.7) Lancet 23 (1.6) 79.3 Euro Surv 6 (1.8) J Acquir Immune Defic Syndr 8 (1.8)
15 JAMA 15 (1.7) Am J Public Health 21 (1.4) 9.3 Sex Health 6 (1.8) Am J Obset Gynecol 7 (1.4)
16 PLoS One 14 (1.6) Emerg Infect Dis 21 (1.4) 6.8 Obstet Gyecol 5 (1.5) CDC 6 (1.2)
17 Am J Public Health 12 (1.4) Ann Intern Med 17 (1.2) 25.4 Cochrane 5 (1.5) N Engl J Med 5 (1.0)
18 J Aquire Immune Defic Syndr 12 (1.4) J Acquire Immune Defic Syndr 17 (1.2) 3.7 Am J Public Health 4 (1.2) Lancet Infec Dis 5 (1.0)
19 Infect Dis Obstet Gynecol 10 (1.1) CDC 14 (0.9) NA J Acquir Immune Defic Syndr 4 (1.2) AIDS Behav 5 (1.0)
20 Pediatrics 10 (1.1) J Allerg Clin Immunol 14 (0.9) 13.2 Lancet 4 (1.2) Am J Public Health 3 (0.6)
Vaccine 4 (1.2) Cochrane 3 (0.6)
Vaccine 3 (0.6)
Total citations from sources listed above 588 (65.9) 913 (62.8)
All other citations 304 (34.1) 541 (37.2)
Total citations 892 (100) 1454 (100)

Overall, Sexually Transmitted Diseases had the most publications cited, with, respectively, 14.0% of all citations in 2015 and 12.8% in 2021, followed in both instances by Clinical Infectious Diseases with 7.5% and 8.2% citations, respectively. Sexually Transmitted Infections had the third most articles, with 5.6% of citations in 2015 and 6.4% in 2021. The other primary STI specialty journal, International Journal of STD & AIDS, was ranked 10th (2.5%) in 2015 and 5th (3.2%) in 2021, respectively. Ranking was similar when limiting the analyses to articles published within 5 years of the publication date of the guidelines, although a downward shift was noted for Sexually Transmitted Diseases when comparing the 2015 (13.9%) and 2021 (9.9%) guidelines, and Sexually Transmitted Infections edged into second in 2021 at 7.4% (Table 1). A full listing of all reference sources is available as a Supplemental Table Online (https://links.lww.com/OLQ/A817).

DISCUSSION

The extensiveness of the reference lists in the CDC STI Treatment Guidelines, bolstered by a 63% increase of cited literature in 2021, is testament to the impressive work by the Guidelines Workgroup (including >120 external consultants and CDC staff in 2021), leaving few, if any, stones unturned in providing the evidence supporting the guidelines. The guidelines citations clearly provide evidence for the importance of the work published in Sexually Transmitted Diseases, as well as Clinical Infectious Diseases and Sexually Transmitted Infections.

For the journal Sexually Transmitted Diseases, the results of this analysis are gratifying but must be placed in context. Although the CDC STI Treatment Guidelines are internationally recognized and referenced, the primary objective is to optimize and standardize STI treatment in the United States. Consequently, the workgroup members are almost exclusively US based. Moreover, many of them are members of the Sexually Transmitted Diseases Editorial Board and have published articles in this journal, thus presenting a likely bias.

Sexually transmitted infection specialty journals, including Sexually Transmitted Diseases, Sexually Transmitted Infections, and the International Journal of STD & AIDS, exclusively publish articles in the STI field. This focus increases the number of articles available for potential use in the guidelines; general journals publish far fewer STI-focused articles. To a lesser extent, the same holds true for general infectious diseases journals that also disproportionally publish STI articles compared with other medical journals—Clinical Infectious Diseases is the prime example as the second most cited publication in our listing.

Our analysis does not provide a measure of quality and potential impact of individual articles. It could be argued that articles published in journals with high impact factors like JAMA (56.3 in 2020) or the New England Journal of Medicine (91.2 in 2020) have greater potential influence given their prestige and larger readership and are more likely to be included in the guidelines. Thus, our alternative analysis should be viewed in context with other metrics, including the impact factor.

Although the analysis was encouraging for Sexually Transmitted Diseases, we did observe a slight downward trend in the proportion of citations of this journal between 2016 and 2021; this trend was more pronounced when limiting to articles published in the previous 5 years. This finding needs further analysis.

Despite these cautions, we conclude that Sexually Transmitted Diseases and other STI specialty journals have a greater influence on the practice of STI diagnosis, treatment, and prevention than one would surmise from the impact factor alone. We hope that these findings will stimulate STI investigators, especially young investigators and the more senior academicians who mentor them, to consider Sexually Transmitted Diseases and other STI specialty journals when presenting their best work. These journals are also more likely to recognize the importance of a given article and promote it, with a commentary, for example, rather than just publish it. We also hope that promotion and tenure committees of academic institutions look beyond impact factors and consider the policy and practice influence of articles, even when published in a journal with a lower impact factor. As shown in this analysis, articles published in these journals have a proven track record to affect practice, which should ultimately be the primary driving force behind publication. Enhancing the prestige of STI specialty journals may even increase their impact factor. But as we have shown in this analysis, as a metric, the impact factor is not enough to measure the true influence of a publication.

REFERENCES

1. Garfield E. The history and meaning of the journal impact factor. JAMA 2006; 295:90–93. doi:10.1001/jama.295.1.90.
2. Workowski KA, Bolan GA; Centers for Disease Control and Prevention. Sexually transmitted diseases treatment guidelines, 2015. MMWR Rcomm Rep 2015; 64(RR-03):1–137.
3. Workowski KA, Bachmann LH, Chan PA, et al. Sexually transmitted infections treatment guidelines, 2021. MMWR Recomm Rep 2021; 70:1–187. doi:10.15585/mmwr.rr7004a1.
4. Rietmeijer C. Blog Post. Available at: http://www.stdpreventiononline.org/index.php/resources/download/2000. Accessed September 14, 2021.

*In 2021 the name was changed from “Sexually Transmitted Diseases Treatment Guidelines” to “Sexually Transmitted Infections Treatment Guidelines.” In this text, we use the latter name for both Guidelines to avoid confusion.

Supplemental Digital Content

Copyright © 2022 American Sexually Transmitted Diseases Association. All rights reserved.