Secondary Logo

Journal Logo

Original Articles

Assessing number and quality of urology open access journals: 2011 to 2018

AlRyalat, Saif Aldeena,∗; alessa, Zeyadb; Mansour, Mustafab; Hamidi, Mohammedb; Obeidat, Muhannadb; Odtallah, Omarb

Author Information
doi: 10.1097/CU9.0000000000000007
  • Open

Abstract

1 Introduction

The field of urology is among the rapidly developing fields, particularly in terms of technologic developments, where the last years were associated with several developments in robotic surgery, imaging, as well as its nanotechnology applications among others.[1] One of the remarkable developments in urology field is its research output. Previous studies have pointed to an increase in the number of publications,[2,3] as well as the quality and impact of publications in different urology fields.[4,5] A study that assessed the trend of impact factor of urology journals between 2000 and 2006 found an increase in the impact factor throughout the studied years.[4] Yet, none of the previous studies assessed the number or impact of open access (OA) journals in urology.

The overall proportion of OA literature has increased from 27% in 2006 to 50% in 2010, the main source of which are OA journals.[6] There is clear evidence that publishing research in an OA journal or as an OA model is associated with higher impact, through number of reads and citation rates.[7] Moreover, OA journals lead to better dissemination of knowledge with the benefit of more citations to the authors.[8] The development of OA journals and their quality are poorly studied in the field of urology. Using data from Scopus database, we aim to assess the number of OA urology journals, their quality in terms of CiteScore, percent cited, and quartiles, and their scholarly production during the years from 2011 to 2018.

2 Materials and methods

We obtained data about journals from www.scopus.com, where the list of journals and their corresponding indices are openly available for download. We obtained data for all Scopus-indexed journals during the period from 2011 to 2018. We filtered the list for urology journals. OA journals covered by Scopus are indicated as OA if the journal is listed in the directory of open access journals (DOAJ) and/or the directory of open access scholarly resources (ROAD).

2.1 Variables

For each journal, we extracted the following variables:

  • CiteScore: Measures average citations received per document published in the serial.
  • Citation count: Citations received in 1 year (e.g., 2017) for the documents published in the previous 3 years (e.g., 2014–2016).
  • Scholarly output: Summation of documents published in the serial title (e.g., 2017) during the 3 years before the year of the metric (e.g., 2014–2016).
  • SCImago quartiles: Quartile 1 (Q1) = 99th–75th CiteScore percentile. Q2 = 74th–50th CiteScore percentile. Q3 = 49th–25th CiteScore percentile. Q4 = 24th–0 CiteScore percentile.

2.2 Statistical analysis

We used SPSS version 22.0 (Chicago, IL) in our analysis. We used mean ± SD to describe continuous variables (i.e., journal indices). We used count (frequency) to describe other nominal variables (i.e., OA journals). We used independent sample t test to analyze the mean difference in CiteScore and scholarly output with OA status in each year. All underlying assumptions were met, unless otherwise indicated. We adopted a p value of 0.05 as the significance threshold.

3 Results

Number of urology journals has increased from 66 journals in 2011 to 99 journals in 2018 (Fig. 1). This increase was also associated with an increase in the quality metrics, that the mean CiteScore increased from 1.09 ± 1.03 in 2011 to 1.37 ± 1.12 in 2018 (Table 1).

Figure 1
Figure 1:
Number of OA and non-OA urology journals in the years 2011–2018. OA = open access.
Table 1 - Citation and output indices for urology journals from 2011 to 2018.
CiteScore Percentile Citation count Scholar output SJR SNIP Percent cited







Year Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD Mean SD
2011 1.09 1.03 49.48 29.13 820.20 1,729.49 474.05 690.62 0.60 0.64 0.61 0.52 38.65 25.00
2012 1.04 1.07 49.68 29.02 753.81 1,678.91 429.20 681.95 0.57 0.70 0.59 0.56 37.81 25.45
2013 1.07 1.08 49.54 29.02 736.85 1,663.16 422.05 686.90 0.63 0.86 0.64 0.60 38.91 25.52
2014 1.01 1.04 49.45 29.10 677.86 1,586.41 405.91 677.61 0.58 0.84 0.65 0.64 38.43 25.68
2015 1.05 1.03 49.44 29.06 683.80 1,543.75 414.01 643.03 0.60 0.88 0.61 0.61 38.89 24.40
2016 1.13 1.02 49.54 29.01 662.05 1,446.25 403.68 590.24 0.65 0.87 0.64 0.60 41.16 23.38
2017 1.33 1.10 49.54 29.05 702.81 1,498.22 393.29 531.16 0.69 0.94 0.71 0.62 45.68 23.68
2018 1.37 1.12 50.10 28.70 684.56 1,433.17 387.69 530.01 0.69 0.99 0.72 0.64 46.69 23.57
SJR = SCImago journal rank, SNIP = source normalized impact per paper.

The number of OA urology journals has increased from only 10 (15.2%) journals in 2011 to 33 (33.3%) journals in 2018 (Fig. 1). This was associated with an increase in quality of these OA journals, as the number of Q1 journals has increased from only 1 journal in 2011 to 5 journals in 2018, and the CiteScore has increased from 0.75 ± 0.75 in 2011 to 1.10 ± 0.73 in 2018. On the other hand, we observed a decrease in mean scholarly output of OA journals from 236 publications per journal in 2011 to around 199 publications per journal in 2018 (Table 2).

Table 2 - Citation metrics and scholarly output for urology OA journals from 2011 to 2018.
Year Number of OA journals Number of Q1 OA journals CiteScore for OA journals, mean ± SD Percent cited for OA journals, mean ± SD Citation count, mean ± SD Scholarly output, mean ± SD Rank, mean ± SD
2011 10 1 0.75 ± 0.75 33.5 ± 24.3 158.5 ± 171.3 236.3 ± 226.1 37.9 ± 16.6
2012 15 1 0.61 ± 0.60 31.53 ± 21.85 121.8 ± 182.6 188.6 ± 214.2 43.87 ± 16.75
2013 16 2 0.6 ± 0.75 28.88 ± 24.0 147.31 ± 231.83 200.88 ± 196.17 49.25 ± 21.26
2014 19 3 0.58 ± 0.62 29.68 ± 21.77 151.74 ± 253.80 184.47 ± 175.30 53.21 ± 21.19
2015 20 3 0.73 ± 0.65 34.75 ± 21.26 175.15 ± 262.58 184.15 ± 149.16 52.5 ± 21.81
2016 26 4 0.88 ± 0.64 40.08 ± 20.32 173.23 ± 232.86 190.54 ± 148.66 53.73 ± 23.75
2017 26 6 1.13 ± 0.71 46.19 ± 20.07 217.27 ± 270.80 199.11 ± 155.77 51.46 ± 24.33
2018 33 5 1.10 ± 0.73 45.12 ± 21.18 210.36 ± 224.55 198.94 ± 144.64 55.61 ± 24.01

Upon comparing CiteScore quality index between OA and non-OA urology journals, we found that non-OA journals had significantly higher CiteScore compared with OA journals till 2015, after which the mean difference in CiteScore became smaller with insignificant p value (Table 3). Scholarly output for non-OA journals is significantly higher than OA journals’ output in all over the period from 2011 to 2018 (Table 4).

Table 3 - CiteScore citation metrics for urology OA versus non-OA journals from 2011 to 2018.
Year CiteScore for non-OA journals CiteScore for OA journals Mean difference 95% CI p
2011 1.15 ± 1.07 0.75 ± 0.75 0.40 −0.19–0.98 0.169
2012 1.15 ± 1.13 0.61 ± 0.60 0.54 0.12–0.97 0.014
2013 1.17 ± 1.13 0.63 ± 0.75 0.54 0.07–1.02 0.027
2014 1.13 ± 1.11 0.58 ± 0.62 0.55 0.16–0.94 0.006
2015 1.15 ± 1.10 0.73 ± 0.65 0.42 0.03–0.81 0.036
2016 1.21 ± 1.12 0.88 ± 0.64 0.33 −0.34–0.69 0.075
2017 1.40 ± 1.21 1.13 ± 0.71 0.27 −0.13–0.67 0.177
2018 1.50 ± 1.25 1.10 ± 0.73 0.41 −0.06–0.88 0.086

Table 4 - Scholarly output in terms of mean publications per year for urology OA versus non-OA journals from 2011 to 2018.
Output for non-OA journals Output for OA journals Mean difference 95% CI p
2011 516.5 ± 737.0 236.3 ± 226.1 280.2 35.44–525.0 0.026
2012 489.35 ± 744.32 188.6 ± 214.2 300.75 79.70–521.80 0.008
2013 477.34 ± 753.0 200.88 ± 196.17 276.47 65.2–487.8 0.011
2014 466.88 ± 749.56 151.74 ± 253.80 282.41 85.99–478.83 0.005
2015 478.76 ± 711.51 184.15 ± 149.16 294.61 114.12–475.1 0.002
2016 480.65 ± 667.48 190.54 ± 148.66 290.11 123.37–456.85 0.001
2017 464.39 ± 599.27 199.11 ± 155.77 265.28 111.50–419.06 0.001
2018 482.06 ± 621.36 198.94 ± 144.64 283.12 65.04–501.20 0.011

4 Discussion

We observed an increase in both number and quality of urology journals during the period from 2011 to 2018. The proportion of OA journals has also increased from 15.2% in 2011 to 33.3% in 2018, this was also associated with an increase in the quality of OA journals, as indicated by CiteScore measure. Although the scholarly output and the citation metrics of non-OA journals are generally higher than OA journals, this difference has decreased over the years, becoming nonstatistically significant since 2015.

The impact of OA status on number of citations received and the citation metrics of a journal was thoroughly studied. A study that analyzed several citation indices between OA and non-OA medical journals found that OA journals had higher CiteScore compared with non-OA journals, although the latter had higher scholarly output.[9] This was inconsistent with our findings about urology journals, where non-OA journals had generally higher CiteScore compared with OA journals, although the difference between them has decreased throughout the studied years. Another study on medical journals that assessed the predicting factors associated with citations found that journal's current citation index was a predictor of future citations.[10] In a study that assessed predictors of citations in 4 non-OA urology journals, its main finding was that study design was the main predictor of future citations,[11] where randomized controlled trials were expected to have higher number of citations compared with other study designs.

In 2009, Hennessey et al[12] published an article analyzing the top 100 cited articles in urology, they only included 15 journals, none of which was an OA journal. In 2013, Nason et al[13] published an update for the list with 16 journals, and none of these was an OA journal as well. We believe the main reason behind these observations was that most of these articles were old articles, published in as early as 1956.[13] Another measure of the impact of an article is its altmetric score, which measures the impact of journal articles by tracking social media, Wikipedia, public policy documents, blogs, and mainstream news activity.[14] In a study that assessed the top 100 urology articles in terms of altmetric score, articles published in OA journals were among the top 10.[5] Recently, several changes are being taken in the landscape of regulations urging publishing in OA journals, especially for the US National Institute of Health and the UK Wellcome Trust funding bodies.[15,16] In a study that analyzed various citations and output indicator of the OA journal “Korean Journal of Urology,” the author found an increase in the number of citations, which was associated with changing the journal official language to English and changing the journal into an OA journal.[17]

Readers of this and other bibliometric articles should bear in mind several limitations. Although we used rigorous indices to reflect the strength of journals (i.e., CiteScore and citations), we didn’t use the more popular “impact factor” index, as it is a product of other databases. Moreover, there are different bibliometric databases that can be used for such an analysis, including Scopus (which was used in the present study), PubMed, and Web of Science. A study that compared bibliometric analysis using PubMed, Web of Science, and Scopus found that PubMed database was only devoted to biomedical sciences and did not provide advanced analytic tools. For Web of Science, although it provided a user-friendly interface, its restricted inclusion criteria for journals may hamper the generalizability of the results. Bibliometric analysis using Scopus database includes a wide range of high-quality journals with advanced analytic tools and comparative indicators, which is why we chose it for our current analysis.[9]

Number and quality of OA journals in the field of urology have increased throughout the last few years. Despite this increase, non-OA urology journals still have higher quality and output. Funders and initiatives that promote OA publishing should further stress on publishing in OA journals, as we believe this will enhance their quality and impact and further enhance the development of the field of urology.

Acknowledgments

None.

Statement of ethics

The study did not use patient's data, and was waived from ethical approval.

Conflict of interest statement

The authors declare that they have no financial conflict of interest with regard to the content of this report.

Funding source

None.

Author contributions

Research Conception: SAA.

Data collection: SAA.

Literature review: ZA, MM, MH, MO, OO.

Data analysis: SAA, ZA.

Manuscript drafting: ZA, MM, MH, MO, OO.

Manuscript revision: SAA.

Approval of final version: SAA, ZA, MM, MH, MO, OO.

References

[1]. Venkatramani V. Urovision 2020: the future of urology. Indian J Urol 2015;31(2):150–155.
[2]. Rahman M, Sakamoto J, Fukui T. Japan's share of research output in urology and nephrology. Int J Urol 2003;10(6):353–355.
[3]. Rezaee ME, Johnson HA, Munarriz RM, Gross MS. Bibliometric analysis of erectile dysfunction publications in urology and sexual medicine journals. J Sex Med 2018;15(12):1426–1433.
[4]. Montorsi F. Current and future trends of publications in urological journals: the increasingly leading role of Europeans. Eur Urol 2007;52(4):939–942.
[5]. O’Connor EM, Nason GJ, O’Kelly F, Manecksha RP, Loeb S. Newsworthiness vs scientific impact: are the most highly cited urology papers the most widely disseminated in the media? BJU Int 2017;120(3):441–454.
[6]. Kurata K, Morioka T, Yokoi K, Matsubayashi M. Remarkable growth of open access in the biomedical field: analysis of PubMed articles from 2006 to 2010. PloS One 2013;8(5):e60925.
[7]. Davis PM. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. FASEB J 2011;25(7):2129–2134.
[8]. Antelman K. Do open-access articles have a greater research impact? Coll Res Libr News 2004;65(5):372–382.
[9]. AlRyalat SA, Malkawi LW, Momani SM. Comparing bibliometric analysis using PubMed, Scopus, and Web of Science Databases. J Vis Exp 2019;24(152):e58494.
[10]. Falagas ME, Zarkali A, Karageorgopoulos DE, Bardakas V, Mavros MN. The impact of article length on the number of future citations: a bibliometric analysis of general medicine journals. PLoS One 2013;8(2):e49476.
[11]. Willis DL, Bahler CD, Neuberger MM, Dahm P. Predictors of citations in the urological literature. BJU Int 2011;107(12):1876–1880.
[12]. Hennessey K, Afshar K, MacNeily AE. The top 100 cited articles in urology. Can Urol Assoc J 2009;3(4):293–302.
[13]. Nason GJ, Tareen F, Mortell A. The top 100 cited articles in urology: an update. Can Urol Assoc J 2013;7(1–2):E16–E23.
[14]. Priem J, Groth P, Taraborelli D. The altmetrics collection. PLoS One 2012;7(11):e48753.
[15]. Available at: https://publicaccess.nih.gov. Accessed August 2, 2019.
[16]. Available at: https://wellcome.ac.uk. Accessed August 2, 2019.
[17]. Huh S. Citation analysis of the Korean journal of urology from Web of science, Scopus, Korean medical citation index, KoreaMed synapse, and Google scholar. Korean J Urol 2013;54(4):220–228.
Keywords:

Bibliometrics; Journals; Open access; Scopus; Urology

Copyright © 2021 The Authors. Published by Wolters Kluwer Health, Inc.