Dr. Kim Holmberg: We are not yet ready to use altmetrics to measure research impact

Almost everybody uses some social networking platform, e.g., Twitter, LinkedIn, ResearchGate, Mendeley, in some way or other today. More and more people are talking about how social media interactions can be advantageous to researchers in several ways. In this interview, I talk to a researcher who is passionate about studying the online interactions of academics on social media and about understanding the potential of altmetrics to capture research impact. Meet Dr. Kim Holmberg, a postdoctoral researcher at the Research Unit for the Sociology of Education (RUSE) at the University of Turku. Dr. Kim Holmberg is also Honorary Research Fellow at the Statistical Cybermetrics Research Group at the University of Wolverhampton and an Associate Professor at the Department of Information Studies at Åbo Akademi University. His research involves altmetrics, webometrics, scientometrics, Web 2.0, Library 2.0, and virtual worlds in education. He also uses quantitative methods to study various phenomena on the web in general and in social media in particular. Dr. Holmberg has authored several journal articles, book chapters, and conference proceedings. He has also published books on topics close to his heart – altmetrics and information exchange in the social media age.

In this interview, Dr. Holmberg talks about how social media platforms play a pivotal role in facilitating useful networks and opportunities for academics. The highlight of this conversation is Dr. Holmberg’s explanation of how the scholarly community is at a nascent stage of understanding and exploring altmetrics and how it is too early to assume that altmetrics are definitive indicators of the societal impact of research. Dr. Holmberg also talks about the main problems with citation-based indicators and mentions some rookie mistakes researchers tend to make when using social media.

Your research interests are quite fascinating. Could you tell us more about each of them?

My research interests are all related to two things; the web and quantitative research methods. I could basically say that I’m keen on understanding a variety of different online phenomena that can be investigated using mainly quantitative methods. The choice of quantitative methods is quite natural due to the enormous volumes of data that the research and publishing community is producing and sharing online. More specifically, I’m currently interested in altmetrics, that is investigating online scholarly communication and how research output (articles, data) is being discussed, shared, and mentioned online and what that could perhaps tell us about the impact of that research or the attention it may have received from a wider audience beyond academia.

Based on your observations, what are the most distinct characteristics of the way in which researchers use Twitter?

I’d say that there aren’t any distinct characteristics in how researchers use Twitter. Some use Twitter professionally while others use it more personally, but probably, most use it in a way for both professional and personal reasons. This duality also makes it difficult to interpret anything from how researchers use Twitter.

Are there any disciplinary differences in researchers’ use of social media?

Yes, in our paper Disciplinary differences in Twitter scholarly communication, we found some evidence on how researchers from certain disciplines use Twitter more for scholarly communication than others. Even though our analysis did not include all disciplines, our results showed clear differences in researchers’ use of Twitter across disciplines. We also took this to suggest that even altmetrics on Twitter should be field-normalized. [Holmberg, K. & Thelwall, M. (2014). Disciplinary differences in Twitter scholarly communication. Scientometrics, vol. 101, no. 2, pp. 1027-1042. DOI:10.1007/s11192-014-1229-3.]

What role do social media play in facilitating and/or enabling collaborative research today?

Social media has an important role in both facilitating and enabling collaborative research. Through academic social networks or social reference managers researchers can find other researchers with similar research interests or the algorithms these platforms use can point users to articles that are similar to those they have authored, thus connecting researchers with each other. Social media can also enable collaboration, for instance, through open discussions on various social media platforms or collaborative authoring.

What are some common rookie mistakes researchers make during online interactions on social media platforms?

A good piece of advice for any researcher using social media platforms is to remember that these platforms (most of them by default) are open to everybody, so anyone can see what you write or share. Although you can create closed groups or chose which of your connections sees what you write on some platforms, it might be difficult for you to constantly keep track of the selected groups you’re sharing your information with. For this reason, I keep my Facebook profile completely open to public viewing. When I know it’s open and never even try to share anything with a closed group, I always keep in mind that anyone can read what I write. This helps me choose what I am saying and ensure that I am communicating what I want to. This also increases the effectiveness of that platform for my purposes.

Most discussions about social media usage focus on researchers. How can other members of the scholarly publishing community use social media to their advantage, e.g., academic institutions, libraries, funders, journals, publishers, etc.?

I do think a lot of these already use social media very innovatively. Libraries for instance have been at the forefront of using social media. In fact, the Library 2.0 phenomenon that came to be widely discussed a few years ago was just about that - about how libraries use social media to connect with their patrons and to promote their services. And that was even before the concept of altmetrics was introduced. Libraries can use social media platforms to promote academic work (articles and other types of academic output), to share information about academic events (academic conferences, guest lectures, public talks, seminars, etc.), and to interact with their patrons, customers, audiences, or stakeholders.

What are some of the current altmetric indicators used to evaluate research impact? How effective are they?

I’d say that no altmetric indicators should yet be used to evaluate research impact because I feel that we’re not yet ready for it. Our understanding of alternative metrics as research impact indicators is limited, even though a lot of research work on this topic is being undertaken in institutions in Montreal in Canada, Leiden in The Netherlands, Wolverhampton in the UK, Bloomington in the US, and Turku in Finland. We still need to resolve a lot of issues with data quality and standards of impact measurement. There are questions about normalizations of the altmetrics data. And to fully understand what the altmetrics (e.g., mentions of a research article on Twitter, Reddit, or Facebook) reflect we need to devise a reliable system to understand more about the people who generated the altmetric event through their interactions with the research output (e.g., sharing an article or mentioning it in a tweet) and in what context they have shared it. I think that, currently, altmetrics are best used for showcasing specific research that has received a lot of attention online. Altmetrics can also point to research that has received a lot of attention outside academia but that doesn’t typically appear on the radar when using citations only to assess research impact. And altmetrics can be used to create narratives around specific research, i.e., how the research has been shared and discussed and by whom. But more research needs to be done before we can talk about reliable altmetric indicators or before we can use them for research evaluation.

Today, altmetrics are becoming increasingly popular. Yet, several researchers are hesitant to use them. Based on your observations, what are some of the common misconceptions researchers have about using altmetrics to measure research impact? How can these be dealt with?

I think one problem is that “altmetrics” as a term includes a very wide variety of different platforms and sources where we can trace and collect mentions of research products. Some of these are promising sources for research evaluation purposes, while others should probably not be used at all. It is important to know the distinction between these different platforms and sources. It is also important to understand that while Twitter is a big data source for altmetric events, it is not the only source and certainly not the most reliable source.

What are some of the problems with citation based indicators? Conversely, what are some of the weaknesses of alternative indicators?

There are basically three well-known major problems with citation-based indicators. The first problem is that they focus only on research articles. In today’s digital age, research products can take many forms, from datasets to presentations and computer code to research articles. All of these research products should be considered as contributions to science, yet only research articles are considered as “important”. Also, there are great disciplinary differences in publishing and citation traditions, because in some disciplines publishing in books or conference proceedings is more frequent than in scientific journals.

The second problem has to do with time. The actual act of scholarly communication can span over a long period - from formulating a research idea to creating a formally published research article that can be cited by others. When using citations to evaluate research impact we can either wait for a few years and wait for citations to accumulate or we can judge an article based on in which journal it has been published in. The first option is thus an evaluation of past success, while the second is a tentative prediction of the future potential impact of the article.

The third problem with citation-based evaluation is that citations are created only by other researchers, thus showing how the scientific community has used the published research. This then gives an indication only of the scientific impact of the research. At the same time, there’s increasing demand to showcase wider societal impact of research, such as economic impact, impact on health, cultural impact, educational impact, or impact on the environment.

The weaknesses of alternative indicators or altmetrics currently are mainly connected to data quality. This has been worked on, for instance by NISO (National Information Standards Organization), but there’s still a lot of work to be done in this area. Another weakness is that we don’t always know the context in which the altmetric event has been created, i.e., whether is it an endorsement of the research or a criticism of it.

What is the best approach to measure research impact?

Peer review. Only through peer review can the impact of specific research be evaluated. However, peer review cannot be performed at a very large scale, or perhaps it can, but that would be very expensive! Therefore, we need other methods as well. Citation-based indicators can help and support peer-review, while altmetrics (currently) can highlight interesting research or research that deserves more attention by the reviewers.

Which online platforms do you think are more effective for researchers – broad platforms like Twitter or researcher-specific forums like ResearchGate?

This depends on what you mean by “effective.” Effective in reaching a wide audience? Then I would say Twitter. Effective in enabling you to share your own work with other researchers or to get in touch with other researchers working on similar projects? Then I would vote for an academic social networking site or social reference manager. So it all depends on what you would want to accomplish, and as we have seen, researchers use social media for a wide variety of different purposes.

How do you see the relationship between social media and the research community evolving over the coming years?

Social media has come to stay and we may as well accept it. But nothing will remain constant. While certain platforms may be discontinued, others will take their place. In the near future, I see more and more researchers beginning to use social media platforms (for different purposes) as they see the benefits these interactions offer.

Thank you, Dr. Holmberg, for sharing such interesting perspectives on this newly emerging topic of discussion in scholarly communication circles!