Secondary Logo

Journal Logo

Study Comparing Traditional Versus Alternative Metrics to Measure the Impact of the Critical Care Medicine Literature

Kaul, Viren MD1; Bhan, Rohit MS2; Stewart, Nancy H. DO3; Behrens, Deanna M. MD4; Gutman, Amy MD5; Dangayach, Neha MD6; Geisler, Benjamin P. MD7; Carroll, Christopher L. MD8

Critical Care Explorations: August 2019 - Volume 1 - Issue 8 - p e0028
doi: 10.1097/CCE.0000000000000028
Original Report
Open
SDC

Our objective was to evaluate the association between traditional metrics such as Impact Factor and Eigenfactor with respect to alternative metrics. The Altmetric Attention Score for the top nine pulmonary and critical care journals was compared with Impact Factor, Eigenfactor, and citations over two time periods (2007–2011 and 2012–2016). There was a significant increase in the Altmetric Attention Score (52 from 2007 to 2011 vs 1,061 from 2012 to 2016; p < 0.001) but no significant differences in Total Citations, Impact Factor, or Eigenfactor. There was a strong positive correlation between citations and Altmetric Attention Score, negative correlations between Eigenfactor and Altmetric Attention Score for most journals, and no clear association between Impact Factor and Altmetric Attention Score. Over time, the digital reach of traditional publications has increased significantly, while no significant increase was noted for the traditional metrics. These findings likely reflect discussions of articles online that are not captured by traditional metrics and hence their impact on the community at large.

1Mount Sinai School of Medicine/Elmhurst Hospital Center, New York, NY.

2Icahn School of Medicine at Mount Sinai, New York, NY.

3University of Nebraska Medical Center, Omaha, NE.

4Advocate Children’s Hospital, Park Ridge, IL.

5Health Alliance Hospital, Kingston, NY.

6Mount Sinai Hospital, New York, NY.

7Massachusetts General Hospital/Harvard Medical School, Boston, MA.

8Connecticut Children’s Medical Center/University of Connecticut, Hartford, CT.

Supplemental digital content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal’s website (http://journals.lww.com/ccejournal).

The authors have disclosed that they do not have any potential conflicts of interest.

For information regarding this article, E-mail: jishuviren@gmail.com

Online date: August 01, 2019

This is an open-access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

The global output of scientific research, in terms of the number of articles published, doubles every 9 years (1). With this rapid growth in output, it becomes important to measure the impact of these publications. Journals and publishers use a variety of methods to assess the reach of these articles and attempt to quantify the impact and quality of the articles that they are publishing (2).

Foremost among these methods are the citations, Journal Impact Factor (IF), and the Eigenfactor score. The IF is the most popularly used metric; it was first described by Garfield (3) and has been calculated yearly since 1975. It is calculated as the summation of the current and previous year’s citations for a journal, divided by the summation of the current and previous year’s total number of publications (Supplemental Table 1, Supplemental Digital Content 1, http://links.lww.com/CCX/A67). The Eigenfactor, developed by Bergstrom et al (4), rates total importance of a scientific journal and takes into account self-citations (5, 6). Additionally, citations from a highly ranked journal are weighted more heavily than citations from a lesser ranked journal.

However, with the development of the internet, in addition to traditional access to articles in printed journals, readers are now able to engage with scientific literature online (7). Clinicians commonly discuss, share articles and participate in conversations regarding the published literature online and on social media (8). This increased access to scientific literature necessitates a need for metrics that can reflect an article or journal’s reach in today’s digitally connected world.

Altmetrics or alternative metrics are a commonly used method to assess the impact of an article or journal using publicly available application programming interfaces across various online platforms (9). Altmetric LLC, quantifies the online reach of research publications by using the Altmetric Attention Score (AAS), a weighted score that accounts for the dissemination of individual articles and journals through channels such as Wikipedia, Twitter, Facebook, public policy documents, blogs, newspapers, and others (10, 11) (Supplemental Table 2, Supplemental Digital Content 2, http://links.lww.com/CCX/A68). A higher Attention Score represents greater visibility and readership received via the described channels.

However, there is little comparison data about the trends, correlation, and association between traditional metrics such as IF and Eigenfactor with respect to alternative metrics in the field of medicine. In this study, we sought to determine the correlation of these traditional metrics to altmetrics in medical literature and focused on the area of pulmonary and critical care medicine.

Back to Top | Article Outline

MATERIALS AND METHODS

AASs were obtained from Altmetric (Altmetric LLP, London, United Kingdom) for the 25 highest ranked medical journals by Journal IF in the year 2016 per the SCImago Journal Ranking. Although Altmetric does not offer journal-level scores, we created journal-level AASs by adding the Attention Scores from all articles published in the journals during the time period studied. Journals that did not have traditional metrics available were excluded from analysis. The top nine pulmonary and critical care journals with more than 1,000 shares in the year 2016 on this list were included in the present analysis. An AAS for each journal was collected for each year between 2007 and 2016.

Traditional journal metrics were then examined for each of these nine journals from publicly available sources. IF, Eigenfactor, and citations were stratified by the journals for each year between 2007 and 2016. In order to compare the growth of traditional publishing metrics and online reach over time, we compared data between two time periods (2007–2011 and 2012–2016). A secondary analysis was then conducted to examine how different metrics compare to each other. Changes to each traditional metric (citations, IF, and Eigenfactor) were compared with changes in AAS over time.

Data were analyzed using JMP 10.0.1 (SAS Institute, Cary, NC). Data are reported as median with interquartile ranges due to the nonparametric nature of the data. For the primary analysis, relationships between time periods and metrics were assessed using the Wilcoxon rank-sum test. For the secondary analysis of comparisons between the changes in IF, Eigenfactor, and citations in relationship to the AAS over time, relationships between two continuous variables were assessed using correlation coefficients (R2) and slope (b). A p value of less than 0.05 was considered statistically significant. Institutional Review Board approval was not obtained due to the public nature of this research data.

Back to Top | Article Outline

RESULTS

The nine highest ranked journals included in the analysis were American Journal of Respiratory and Critical Care Medicine, American Journal of Respiratory Cell and Molecular Biology, Annals of Allergy, Asthma, and Immunology, Chest, Critical Care Medicine, Current Opinion in Critical Care, Pediatric Critical Care Medicine, Pediatric Pulmonology, and Respiratory Care.

There was a significant increase (52 from 2007 to 2011 vs 1,061 from 2012 to 2016; p < 0.001) in the AAS for the nine journals between the two time periods (Table 1 and Fig. 1). However, there were no significant differences in the number of total citations, IF, or Eigenfactor between the two time periods (Table 1; and Supplemental Fig. 1 [Supplemental Digital Content 3, http://links.lww.com/CCX/A69], Supplemental Fig. 2 [Supplemental Digital Content 4, http://links.lww.com/CCX/A70], and Supplemental Fig. 3 [Supplemental Digital Content 5, http://links.lww.com/CCX/A71]).

TABLE 1

TABLE 1

Figure 1

Figure 1

When analyzing these trends at the individual journal level (Supplemental Table 3, Supplemental Digital Content 6, http://links.lww.com/CCX/A72), there were some differences noted over time in these metrics. All of the journals had significantly increased AAS between the two time periods. Almost all of the journals (8/9) had significant increases in number of total citations between the two time periods. Only the Annals of Allergy, Asthma, and Immunology did not show a statistically significant increase in citations. Most of the journals (6/9) had no significant changes in IF between the two time periods. Three journals had a statistically significant change in IF; American Journal of Respiratory and Critical Care Medicine and Pediatric Pulmonology had an increased IF, and American Journal of Respiratory Cell and Molecular Biology had a decreased IF between the two time periods. Most of the journals (7/9) had significant decreases in Eigenfactor between the two time periods. Two journals (Current Opinion in Critical Care and Pediatric Critical Care Medicine) had no change in Eigenfactor.

Next, we examined the correlations between changes in the AAS and traditional journal publishing metrics (Eigenfactor, IF, and citations) over time. All of the journals had statistically significant positive correlations between Total Citations and AAS (Table 2). In other words, for all journals as the Total Citations increased, so did the AAS. Most of the journals (7/9) had statistically significant negative correlations between Eigenfactor and AAS. Two of the journals (Pediatric Critical Care Medicine and Respiratory Care) had statistically significant positive correlations between Eigenfactor and AAS. When comparing IF and AAS, four of nine journals (American Journal of Respiratory and Critical Care Medicine, Annals of Allergy, Asthma, and Immunology, Critical Care Medicine, and Pediatric Pulmonology) had statistically significant positive associations. However, the remaining five had no significant association between IF and AAS.

TABLE 2

TABLE 2

Back to Top | Article Outline

DISCUSSION

In today’s digital world, journals and publishers need to go beyond traditional publication metrics to fully assess the reach of their articles. A significant amount of discussion about medical literature occurs online, and these discussions are not captured by the more traditional metrics such as citations, Eigenfactor, and IF (12). In our review of pulmonary and critical care journals, we found no significant increase in the traditional journal publication metrics over the last decade but found a significant increase in the digital reach during the same time frame. Additionally, we found strong positive correlation between citations and AASs for all journals and negative correlations between Eigenfactor and AASs for most of the journals. However, there was no clear association between IF and AASs.

In the current literature, correlation between tweets and citations has been found to be low across all journals, specialties, and disciplines, prompting need for novel social media-based metrics to be developed, such as the AAS (13). Individual components of the AAS (such as tweets and Mendeley cites) have been previously shown to have varying degrees of correlation with traditional publication indicators (14–18). For example, in the field of medicine, traditional citations have been shown to have weak to moderate degrees of correlation to individual alternative metrics, however, there is a paucity of data overall (19). The correlation differs in terms of the type of articles being assessed, with editorials and news items not commonly cited in traditional literature as opposed to over social media (20). The relationship between the composite AAS with the traditional metrics, however, has been sparsely studied in medicine, including the field of pulmonary and critical care medicine.

IF and Eigenfactor are two of the most commonly used journal metrics to quantify the journal-level impact. Because they are derivations of citations, they are used as a reflection of the journal’s quality and influence within traditional academic systems. IF is derived from the number of citations and articles published over previous 2 years. Eigenfactor rates the total impact of a journal based on incoming citations that are weighted to factor for citations from more highly ranked journals as opposed to poorly ranked journals. Although both IF and Eigenfactor are derived from citations, they may correlate differently with the AAS due to the differences in how they are derived. These traditional publication metrics also fail to account for article-level analysis and dissemination via new media channels such as television, radio, podcasts, and other social media platforms. Today, much of the discussion of an article occurs outside traditional publication platforms, where an article can be read and shared by nonacademic physicians, providers, patients, and caregivers (8, 12). Although the AAS is not a replacement for traditional publishing metrics, it seems to better reflect the impact of a publication on the community at large (17).

Although all of the journals had a significant rise in the AASs over time, three of the journals (American Journal of Respiratory and Critical Care Medicine, Critical Care Medicine, and Chest) had higher rates of rise compared with the rest (Fig. 1). The reasons for this are unclear. It may be that unmeasured characteristics such as the presence of a social media team, dedicated editor, progressive social media strategy, or high new media engagement may have contributed to these increases.

Our study has several limitations. First, this descriptive study can only suggest associations, and cannot be used to determine causation due to the fact that a limited number of journals are included in the current analysis. Second, given complexity of the data and limitations in traditional publishing metrics, we were not able to compare citations and Altmetrics at the article level, and relied instead on journal-level metrics. Hence, if an individual article received an inordinate amount of attention on social media, it could single-handedly skew the annual AAS for the particular journal. Third, we limited this analysis to pulmonary and critical care journals. These fields have active online and social media presence which may have inflated the AASs, and the findings may not be generalizable to other fields. As such, the findings from our study should be considered hypothesis generating and need further study over the coming years across various specialties and journals across the spectrum of impact. Finally, as noted previously, the impact of the journals’ own social media presence, activity, and efforts on the AAS is an important aspect to analyze. However, currently, we are limited in not having these data available publicly.

To summarize, in this analysis of major pulmonary and critical care journals, we found that the correlation between traditional publishing metrics and AASs was fair, with a strong positive correlation between citations and AASs for all journals, negative correlations between Eigenfactor and AASs for most journals, and no clear association between IF and AAS. Additionally, there was no significant change in the traditional publishing metrics for these journals over the last decade, although there was a significant increase in AASs. This likely reflects expanded discussion of articles online and on social media, platforms that are not captured by traditional publication metrics. Our findings open the forum for further discussion regarding how the impact of an academic output should be assessed. We anticipate that alternative metrics will gain prominence in supplementing traditional metrics for the purpose of demonstrating impact of a publication in academic circles and the medical community as well as that of authors with a role in advancement and promotion of faculty.

Back to Top | Article Outline

ACKNOWLEDGMENTS

We would like to acknowledge Altmetric for providing access to the research data. Altmetric was not involved in study design, implementation, data collection, analysis, or article preparation. We would also like to acknowledge Amanda Alexander, DO for her contributions.

Back to Top | Article Outline

REFERENCES

1. Van Noorden R. “Global Scientific Output Doubles Every Nine Years.” Newsblog, Nature, May 7, 2014.Available at: http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html. Accessed September 1, 2018
2. Carpenter CR, Cone DC, Sarli CC. Using publication metrics to highlight academic productivity and research impact.Acad Emerg Med2014211160–1172
3. Garfield E. The history and meaning of the journal impact factor.JAMA200629590–93
4. Bergstrom CT, West JD, Wiseman MA. The eigenfactorTM metrics.J Neurosci20082811433–11434
5. Eigenfactor.orgAbout the Eigenfactor Project.Available at: http://www.eigenfactor.org/about.php. Accessed September 1, 2018
6. Davis PM. Eigenfactor: Does the principle of repeated improvement result in better estimates than raw citation counts?J Am Soc Inf Sci Technol2008592186–2188
7. Giustini D:. How web 2.0 is changing medicine.BMJ20073331283–1284
8. Choo EK, Ranney ML, Chan TM, et al. Twitter as a tool for communication and knowledge exchange in academic medicine: A guide for skeptics and novices.Med Teach201537411–416
9. Galligan F, Sharon D-C:. Altmetrics: Rethinking the way we measure.Serials Rev20133956–61
10. AltmetricWhat Are Altmetrics?Available at: https://www.altmetric.com/about-altmetrics/what-are-altmetrics/. Accessed September 1, 2018
11. Altmetric SupportHow Is the Altmetric Attention Score Calculated.2019Available at: https://help.altmetric.com/support/solutions/articles/6000060969-how-is-the-altmetric-attention-score-calculated. Accessed September 1, 2018
12. Trueger NS, Thoma B, Hsu CH, et al. The altmetric score: A new measure for article-level dissemination and impact.Ann Emerg Med201566549–553
13. Haustein S, Peters I, Sugimoto CR, et al. Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature.J Am Soc Inf Sci Technol201465656–669
14. Zahedi Z, Costas R, Wouters P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications.Scientometrics20141011491–1513
15. Bornmann L. Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics.Scientometrics20151031123–1144
16. Costas R, Zahedi Z, Wouters P. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective.J Am Soc Inf Sci Technol2015662003–2019
17. Shema H, Bar-Ilan J, Thelwall M. Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics.J Am Soc Inf Sci Technol2014651018–1027
18. Eysenbach G. Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact.J Med Internet Res201113e123
19. Rosenkrantz AB, Ayoola A, Singh K, et al. Alternative metrics (“altmetrics”) for assessing article impact in popular general radiology journals.Acad Radiol201724891–897
20. Haustein S, Costas R, Larivière V. Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns.PloS One201510e0120495
Keywords:

altmetrics; citations; Eigenfactor; Impact Factor; pulmonary and critical care medicine; traditional metrics

Supplemental Digital Content

Back to Top | Article Outline
Copyright © 2019 The Authors. Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine