|Year : 2021 | Volume
| Issue : 1 | Page : 156
nvestigating of association between altmetrics activity indicators and citation quality indicators in iranian medical journals
Mansoureh Serati Shirazi1, Ali Valinejadi2
1 Islamic World Science Citation Center (ISC), Research Department of Scientometrics, Shiraz, Iran
2 Social Determinants of Health Research Center, Semnan University of Medical Sciences, Semnan, Iran
|Date of Submission||09-Feb-2020|
|Date of Acceptance||14-Jul-2020|
|Date of Web Publication||30-Nov-2021|
Social Determinants of Health Research Center, Semnan University of Medical Sciences, Semnan
Source of Support: None, Conflict of Interest: None
Background: Given the limitations of traditional citation indicators, more indicators are needed to examine the effectiveness and improvement of existing indicators. The present study aimed to investigate the association between Altmetrics activity and quality citation indicators in Iranian journals based on Clarivate Analytics, Scopus, and Medline. Methods: The research was carried out using Altmetrics method through scientometric approach. The population of the present study was Iranian medical journals, which were available at three databases of Clarivate Analytics, Scopus, and Medline. In order to obtain quality information, we've used the indicators of CiteScore, SJR, and SNIP at Scopus database as well as the impact factor at JCR database; besides, to find Altmetrics indicators and Altmetrics scores of articles, the Altmetrics explorer database was used. Results: About 16% of the articles in the reviewed journals were cited at least once in the social media and had Altmetrics scores. Among the reviewed journals, the highest rate of social media coverage was related to the Iranian Journal of Basic Medical Sciences, and the highest Altmetrics score obtained from the mean Altmetrics score of the papers was related to Cell Journal (Yakhteh). A review of quality indicators of journals with Altmetrics coverage and Altmetrics score of journals showed that there was a significant and positive correlation between the Altmetrics score and impact factor. However, any significant association was neither found between the journal's coverage and SiteScore, SNIP, SJR, nor between the journal's Altmetrics score and quality indicators. Conclusions: According to the results of this study, policy makers of scientific journals should adopt strategies that bring about social media presence; thus, we will find further Iranian articles in the social media.
Keywords: Altmetrics, bibliometrics, clarivate analytics, journal impact factor, medline, scopus, social media
|How to cite this article:|
Shirazi MS, Valinejadi A. nvestigating of association between altmetrics activity indicators and citation quality indicators in iranian medical journals. Int J Prev Med 2021;12:156
|How to cite this URL:|
Shirazi MS, Valinejadi A. nvestigating of association between altmetrics activity indicators and citation quality indicators in iranian medical journals. Int J Prev Med [serial online] 2021 [cited 2022 Jan 19];12:156. Available from: https://www.ijpvmjournal.net/text.asp?2021/12/1/156/331646
| Introduction|| |
Altmetrics indicators which are derived from the social web are increasingly becoming popular and are used as the primary impact factors. Altmetrics—stands for alternative metrics—means alternative indicators, which is a term to describe web-based indexes to impact the research articles and emphasizes the social media for data collection. The concept was first introduced in 2010 by Priem et al. The web age has provided a new environment for the discussion and dissemination of research information and according to this new environment there is a need for more indicators to examine the impact and to complete the existing indicators. Altmetrics indicators have eliminated several limitations of traditional citation-based indicators, in which other aspects of impact has developed, such as social, cultural, and education impacts, which were neglected by traditional methods of impact assessment. Taylor referred to Altmetrics indicators as a revolution in scientometrics; and Lin and Fenner described it as a new paradigm in research evaluation. Altmetrics focuses on research outputs in social networks such as Facebook, Twitter, blogs, news outlet and reference management tools, and goes beyond citation in analyzing academic activities. Instead of just analyzing the articles of scientific journals, Altmetrics considers the impact of scientific output in different formats. Altmetrics, unlike the citation-based indicators, such as the journal's impact factor or the H-index, evaluates the impact of publication of a single article. The indicators of article level measure the impact of a single article through quantitative indicators, such as counting the number of times that an article is downloaded or shared on Twitter, as well as quality indicators such as media coverage or blog posts by the experts in that field. However, some scientometrics experts consider Altmetrics as a threat to the alignment process and quality control of research activities; nevertheless, pursuing the use of scientific articles in social media means that researchers are able to examine the impact of these products more extensively; hence, along with the citation, these indicators should be investigated and their impact should be considered. Salajegheh and Diari examined the association between citation indices and Altmetrics in 111 medical sciences journals in Scopus database. Altmetrics data was obtained from Altmetrics official website According to their results, there was a correlation between the citation indicators (except the impact factor) and the mean scores of Altmetrics. Goltaji, and Jowkar studied the extent of availability of medical informatics articles in social media through Altmetrics approach. The results showed that the scientific outputs were often used through social networks. In this study, a significant positive association was found between most of Altmetrics indicators and the number of received citations on the Web of Science platform. The study of Erfanmanesh on the relationship between Altmetrics activity and quality of the library and information sciences journals showed that the published articles in higher quality scientific journals were more widely seen on social networks, and attracted more attention of users.
Serati Shirazi and Goltaji carried out a research on Scientific Articles of “Health Literacy” in Social Media. The results of their study revealed that Mendeley and Twitter are the most used social media by health literacy scholars for sharing scientific outputs and there is significant relationship between most alternative metrics and the number of citations in Web of Science.
Zahedi, Costas and Wouters studied the use of Web of Science journals at Mendeley, as well as the articles' citation indexes through the Impactstory website, and reported a moderate correlation between these two indicators. In a comparative study, Cho studied the impact of Korea's research papers in four academic disciplines using the Altmetrics approach. His research population included 383 Korean international research papers in medicine, engineering, social sciences, arts and humanities. The results of this study suggested that the highest number of research articles that were “discussed” in international media such as Twitter was in medicine; and the highest number of research papers that had been stored via reference management tools such as Mendeley was seen in social sciences, arts and humanities.
Articles of medical sciences journals, considering the novelty of the subject in this discipline, are being studied shortly after publication; and the spent time on the publication of the article may reduce the value of the content. Therefore, the use of articles in online social networks can be effective in updating the publications. Hence, it is essential to assess the Altmetrics approach in medical sciences articles, in order to be used in research policies and in evaluating the publications quality.
Since the Altmetric activity of Iranian medical journals has not been studied so far, thus, the objective of this study was to investigate the association between Altmetrics activity and quality citation indicators in Iranian journals on three databases of Clarivate Analytics, Scopus, and Medline.
The results of this study can persuade policymakers that they should pay more attention to the social media and the rules governing them such as limitations and filtering. They should inform everyone especially the scientific community about social networks and the way to be an active one in using them and also their benefits.
In addition, they can encourage researchers to publish their research in online and open access format, and can make research policy to promote faculty members based on their Altmetric score.
| Methods|| |
The research was carried out using Altmetrics method through scientometric approach. The research population consisted of Iranian medical journals, which were available on databases of Clarivate Analytics, Scopus, and Medline. In order to obtain quality information, the indicators of CiteScore, SJR, SNIP were used, which were extracted from the Scopus database, and the impact factors of journals were obtained from SJR database. During the research, since the last calculation for these indicators was in 2017, the same year was considered for data collection. The Altmetric Explorer was used to obtain Altmetrics indices and Altmetrics scores.
The Altmetric Explorer database provides the possibility of searching articles in a variety of ways, such as the use of scientific identifiers including the digital object identifier (DOI), and it also provides the articles from a journal using ISSN. Because the aim of this study is to evaluate all articles in medical journals, we used the journals' ISSN.
The Altmetrics score was calculated by giving weight to the scores obtained from the above-mentioned social media and was assigned to the article. For each journal, Altmetrics scores were obtained for all articles. Then, the mean Altmetrics scores for articles of each journal was considered as Altmetrics activity index of the journal and was used in correlation tests. Statistical tests were performed using SPSS 20 statistical software and Spearman's correlation test.
| Results|| |
The results of this study showed that in 2017, 15 journals published a total of 1,674 articles. Iranian Red Crescent Medical Journal with 269 articles had the highest number of articles, and DARU Journal of Pharmaceutical Sciences with 25 articles had the smallest number of articles in 2017. As shown in [Graph 1], the number of published articles in 2017 in the journals does not follow the equality distribution. In other words, in 1 year according to the journal's size and the consumed time for publishing, each journal published various numbers of articles.
The results of reviewing 15 journals using the ISSN on the Altmetrics Explorer showed that only 13 journals were retrieved, for instance, by entering the DOI of the published articles, the information of Arch. Iran. Med. Journal was retrieved, while no information was obtained from DARU Journal. The findings showed that among the 1,674 articles published in these journals (including articles in DARU Journal), there has been paid attention to 276 articles for at least once, which had Altmetrics score. Accordingly, only about 16% of these articles had Altmetrics score. In total, these documents were taken into consideration 685 times, and on average, the share of each of the 256 papers was about 2.7. Among the studied journals, the most frequent Altmetrics coverages were 50.75, 34.38, and 34.24, respectively, in the Iranian Journal of Allergy, Asthma & Immunology, Iranian Journal of Immunology, and Iranian Journal of Basic Medical Sciences.
According to the results of [Table 1], the highest presence of articles in social media were mentioned for 649 (94.7) times, among which Twitter (583 times) was ranked first, and Facebook, Google+, and Reddit Posts were in the next ranks long after Twitter. After the social media, the highest presence rate was referred to the category of “other sources” with 19 times of presence (8.2), the presence of resources in Wikipedia was 6 times and they were 11 times mentioned in the videos. News and weblogs ranked third with 17 times of presence (2.4). In this category, news with 11 and weblogs with 16 times mentioning were respectively in the first and second rank. As stated above, the examined articles had the highest levels of presence on Twitter. The geographic distribution of Twitter users showed that the highest number of tweets was from the United States, while England and Spain were in second and third places, respectively, and Iran ranked tenth. Facebook had the most number of posts after Twitter. A survey of Facebook users showed that the highest number of users was from the United States, India, and Spain, respectively; however, no definite user was from Iran.
|Table 1: The rate of mentioning the articles in the reviewed journals in various media|
Click here to view
Specifically, Hepatitis Monthly Journal, Iranian Journal of Allergy, Asthma and Immunology, Iranian Journal of Immunology, Iranian Journal of Kidney Diseases, Iranian Journal of Pharmaceutical Research, Iranian Journal of Radiology, Iranian Red Crescent Medical Journal, and Urology Journal were only available in social media; and Jundishapur Journal of Microbiology was only present at news and blogs. Other journals were present at more than one category. [Table 2] represents the rate of citation to the articles in the reviewed journals in different media.
|Table 2: The rate of reference to the articles in the reviewed journals in different media|
Click here to view
Positively, the most frequent presence in the social media was related to the Iranian Journal of Basic Medical Sciences, from which 63 articles were shared at least once and had Altmetrics score. The articles in this publication were 144 times referenced; the articles in this journal were totally cited 144 times. Following that, 44 articles from Journal of Research in Medical Sciences and 34 articles from Cell Journal (Yakhteh) have been shared at least once, each of which were cited 135 and 102 times, respectively. Both journals ranked second and third, respectively. [Table 3] illustrates the number of mentioned articles and the number of mentions in each journal (the presence of each journal).
|Table 3: The number of mentioned articles and the number of mentions, along with Altmetrics score of journals|
Click here to view
However, since the articles presence was considered in relation to the number of articles in the journal, the highest rate of articles presence (75.50) was assigned to Iranian Journal of Allergy, Asthma & Immunology. 34 articles out of 67 articles from this journal were mentioned at least once. Thus, more than half of this journal's articles were present on social networks. [Table 3] exhibits the percentage of each journal's articles which had presence in social media. In [Table 3], the mean Altmetrics scores of the examined journals was reported from the mean Altmetrics score of the journals' articles; so that Cell Journal had the highest Altmetrics score with 1.27 Altmetrics score. DARU Journal of Pharmaceutical Sciences, with an impact factor of 2.667, CiteScore of 2.13, and SNIP of 1.001 had the highest rate of these qualitative indicators; however, according to SJR indicator, Jundishapur Journal of Microbiology with 2.297 had the highest rate for SJR index. [Table 4] shows the qualitative indicators of the reviewed journals.
Spearman's correlation test was used to investigate the association between citation-based qualitative indicators (impact factor, SJR, SNIP, and CiteScore) and Altmetrics activity of journals including the journal's Altmetrics coverage and Altmetrics score. The results of this test for the journals showed that there was a significant and positive correlation between journals coverage and the impact factor (r = 0.546; P = 0.035), but there was no significant association between journals coverage and CiteScore, SNIP, and SJR. Moreover, there was no significant association between Altmetrics scores and the qualitative indicators of the journals.
| Discussion|| |
The present research was carried out aiming to investigate the association between Altmetrics activity and citation quality indicators in Iranian journals on three databases of Clarivate Analytics, Scopus, and Medline, which was conducted through scientometric approach and Altmetrics methodology. From the reviewed journals, Archives of Iranian Medicine and DARU were not retrieved by using ISSN. But by entering the DOI of the articles, the information of Archives of Iranian Medicine was retrieved, but no information was obtained from DARU Journal. This issue should be addressed by journalists and policy makers of the journals; because today, apart from the impact of articles on science, which is calculated by counting citations and calculating the impact factor, the impact of articles in the community is measured by Altmetrics in terms of dimensions other than scientific dimension. Failure to retrieve articles in databases that assess Altmetrics indicators means that the articles are not seen and it is not possible to examine the social impact of the article. To solve this problem, journal should increase their visibility on the web by creating weblogs, ResearchGate, and Linkedin pages and so on.
The findings suggested that just about 16% of these published articles (considering the articles of DARU Journal) had Altmetrics scores. This trend is little for medical journals that should be available to the scientific community and even other people as soon as possible. Social media and cyberspace in general allow scientific content to be readable, regardless of the time-consuming process of publishing the papers. The lack of using this possibility is clearly seen for Iranian medical journals that have been investigated in this study.
According to the results, the examined articles had the highest rate of presence on Twitter, and Facebook had the highest number of posts after Twitter. Social media, especially Twitter, are powerful tools in terms of having Altmetrics data and can be a comprehensive tool for researchers; saving, tagging, and tweeting articles on Twitter can reflect the impact of these articles on users. In other words, increasing social impact has led to an increase in citation. The geographic distribution of Twitter users showed that the highest number of tweets was from the United States, while England and Spain ranked second and third respectively; Iran ranked tenth. The filtering of the Twitter network in Iran can be one of the most important reasons for not using this network as much as it can be used and also in comparison the United States and some other countries such as England the English language as a second language in Iran can be the other reason.
A review of Facebook users demonstrated that the highest number of users was from the United States, then from India and Spain; and no specific user from Iran was found. The results of Goltaji and Jowkar's study showed the active presence of members from the United States, England, Canada, and Spain in social media. The geographic review of tweets about Iranian articles in Erfanmanesh's study showed that the highest number of tweets belonged to the United States, England, and Canada, which was in line with the present study. Erfanmanesh argued that articles published in higher quality scientific journals were more widely present on social networks and attracted more attention. By increasing electronic access, readers can have access documents with less effort, as users are more eager to retrieve documents with less access barriers. According to the principle of least effort, the user tries to achieve the desired result by spending minimum time and effort. One of the grounds provided to achieve this goal is social media. Therefore, it is necessary for researchers to use this environment in order to be seen and effective. Mazov and Gureev suggested that some journals shared all the published articles in their latest volumes manually or mechanically via Twitter. Accordance of the results of investigating the geographical dispersion of tweets confirmed this, so it is essential for the publishers to make serious efforts to inform about the articles.
Absolutely among the reviewed journals, the articles of “Iranian Journal of Basic Medical Sciences” had the highest levels of social media presence. However, in terms of the presence of journal's articles comparing to the number of articles, the highest number of articles was belonged to “Iranian Journal of Allergy, Asthma & Immunology.” The highest Altmetrics score was 1.27, which was belonged to “Cell Journal.” “DARU” journal of pharmaceutical sciences with an impact factor of 2.667, CiteScore of 2.13, and SNIP of 1.001 had the highest rates among these qualitative indicators, but considering the SJR index, “Jundishapur Journal of Microbiology” had the highest rate (2.297). The results of reviewing the relationship between citation-based qualitative indicators (impact factor, SJR, SNIP, and CiteScore) and Altmetrics activity of journals, including Altmetrics coverage and Altmetrics score, indicated that there was no significant relationship between journals coverage and CiteScore, SNIP, and SJR; besides, there was not found any significant relationship between the journals' Altmetrics score and qualitative indicators. These results were not in line with Erfanmanesh's research. But there was a significant and positive correlation between the journals coverage and the impact factor (r = 0.546, P = 0.035), which was consistent with Erfanmanesh's research. The results of Goltaji and Jowkar's study indicated that there was a significant association between the received citations on the Web of Science and Altmetrics indicators of most social media, so that the existence of articles increased the number of citations to those articles. In most studies conducted in the field of Altmetrics, researchers reported a positive and significant relationship between the quality indicator (citation) and the Altmetrics score of the articles, so that the findings of this study were consistent with the results of Zahedi et al., Mohammadi et al., Ebrahimy and Setareh, and Li et al. On the other hand, these studies have shown that social media has had a positive impact on increasing the number of citations to articles.
Although there was a reasonable and moderate association between Altmetrics indicators and citations, the quality of the data obtained from the Altmetrics and their effective dimensions which could be reported accurately were not clear; and Altmetrics was still immature. Nonetheless, researchers of the present study believed that the indicators of article and Altmetrics could be used additional to the scientometric indicators in order to investigate the impact of scientific output and the performance of researchers in Iran. On the other hand, according to the results of this study and similar studies, it can be concluded that social media has had a positive impact on the increase of citations to articles. Therefore, policymakers of scientific journals need to provide facilities for the presence in social media in order to see more Iranian articles in social media.
Some limitations of the present research were limited coverage of data provided by the Altmetrics Institute and other Altmetrics service providers, as well as the inability to access certain social media within the country due to filtering; these limitations could affect the results.
Suggestions for further studies
Although in this research it was tried to fully review medical journals of three validated databases, it is suggested that this research should be repeated in different times and the process of changes in the Altmetrics indicators should be considered. In the present research, we examined the medical journals regardless of their specialty. It is suggested that for further studies the articles in various disciplines should be reviewed and compared to each other.
Ethics committee approval
Ethics approval was obtained from the Semnan University of Medical Ethics Committee (IR. SEMUMS. REC.1397.044).
We would like to thank the Research Deputy of Semnan University of Medical Sciences for providing facilities to this work.
Financial support and sponsorship
Research reported in this publication was supported by a grant [number: 1404] from the Semnan University of Medical Sciences, Semnan, Iran.
Conflicts of interest
The authors declare that they have no competing interests.
| References|| |
Priem J, Taraborelli D, Groth P, Neylon C. Altmetrics: A manifesto. 2010.
Thelwall M, Haustein S, Larivière V, Sugimoto CR. Do altmetrics work? Twitter and ten other social web services. PLoS One 2013;8:e64841.
Trueger NS, Thoma B, Hsu CH, Sullivan D, Peters L, Lin M. The altmetric score: A new measure for article-level dissemination and impact. Ann Emerg Med 2015;66:549-53.
Taylor M. Exploring the boundaries: How altmetrics can expand our vision of scholarly communication and social impact. Inf Stand Q 2013;25:27-32.
Lin J, Fenner M. Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Inf Stand Q 2013;25:20-6.
Bornmann L. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. J Inform 2014;8:895-903.
Costas R, Zahedi Z, Wouters P. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. J Assoc Inf Sci Technol 2015;66:2003-19.
Erfanmanesh M. The presence of Iranian Information science and library science articles in social media: An altmetric study. Iran J Inf Process Manag 2017;32:349-73.
Li X, Thelwall M, Giustini D. Validating online reference managers for scholarly impact measurement. Scientometrics 2012;91:461-71.
Salajegheh M, Diari S. The relationship between Altmetrics and SNIP, SJR, Eigenfactor and IF of medical science journals. J Nat Stud Librarianship Inf Organ 2016;27:167-80.
Goltaji M, Jowkar A. Presence of scientific outputs of medical informatics using social media: An altmetric study. Health Inf Manag 2017;14:71-7.
Erfanmanesh M. The relationship between altmetric activity and quality indicators of the library and information science journals in scopus. J Nat Stud Librarianship Inf Organ 2018;29:7-26.
Shirazi MS, Goltaji M. An altmetric study on scientific articles of “Health Literacy” in social media. Payesh (Health Monitor) 2018;17:249-56.
Zahedi Z, Costas R, Wouters P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of 'alternative metrics' in scientific publications. Scientometrics 2014;101:1491-513.
Cho J. A comparative study of the impact of Korean research articles in four academic fields using altmetrics. Perform Meas Metrics 2017;18:38-51.
Erfanmanesh MA. The relationship between altmetrics activity and qyality indicators of the library and information science journals in Scopus. J Nat Stud Librarianship Inf Organ 2018;29:7-26.
Mazov NA, Gureev VN. Alternative approaches to assessing scientific results. Her Russ Acad Sci 2015;85:26-32.
Mohammadi E, Thelwall M, Haustein S, Larivière V. Who reads research articles? An altmetrics analysis of M endeley user categories. J Assoc Inf Sci Technol 2015;66:1832-46.
Ebrahimy S, Setareh F. Research on alternative measures in the F1000 system with Google Scholar citation index. J Inf Process Manag 2016;31:891-909.
[Table 1], [Table 2], [Table 3], [Table 4]