Video: The Future of Peer Review is AI Plus Human with Gareth Dyke

If you are still resisting the use of AI for scholarly publishing, know that you’re not alone. Many scientists are unsure of how AI is impacting the research community and do not feel comfortable using AI-powered tools. But Gareth Dyke (Academic Director, Reviewer Credits) is all for AI being incorporated in academia—ethically of course! Watch this video to discover how peer review can be reshaped by authors, editors, and journal publishers in the AI era. 

Q: Where do you see the greatest potential for AI to support peer reviewers today?
A: Okay, so I see that AI has huge potential both to support peer reviewers and editors at journals, books, and publishing companies. So, in particular, AI can be used to quickly evaluate manuscripts when they first come in to a journal workflow or when they’re first provided to peer reviewers for an evaluation, for example. Is this work good? What’s the standard like? What’s the level of paradigm shift, for example. What’s the level of importance that this work has? So to give editors, journal editors, and peer reviewers an overview of the research article itself, perhaps a translation into their own native language, a summary of the article in Mandarin or in Hindi, for example, so that experts from countries where English is not the first language can quickly also gain an overview of a research article. And then also a number of other kinds of checks such as on statistics, citations, spelling, the formatting of the paper to give editors and peer reviewers like a head start into the peer review process. But, as we’ll talk about in a moment, I firmly believe that the actual evaluation of the science of the research of the content within a research article should be done by an expert human peer reviewer. 

Q: What are the limits of AI when it comes to evaluating manuscripts? Can it handle nuance, ethics, or context?
A: Well, I would say at the moment, not really. I mean, with the tools that we have available to us right now, it’s very difficult for us to gain a view of the context and the nuance of a research paper using AI. But that’s not to say that these approaches, these tools won’t become available in the very near future. We already have tools that effectively do evaluate research in the context of other research articles in that field, in that vertical. So I would expect that at the moment we can’t give a view on nuance, on ethics particularly of a research article using AI apart from running a series of standard checks. And there are tools that do that. For example, tools like Eliza, tools like XPier, tools like REVIEWER Three; just to name 3 of the more than 12, 13, 14 AI-powered peer review tools that we currently have available to us as journal editors and peer reviewers. But yeah, I would say that this nuance and the ethical, real, proper evaluation of the article can’t yet at the moment be done using an AI tool. But that’s not to say that people aren’t doing it. So be careful as an editor. If you get peer reviews, there’s a good chance that they may have at least partially been generated using AI. 

Q: Have you seen any real-world examples where AI tools helped—or hurt—the peer review process?
A: Well, yeah. I mean, we’re all aware of a number of recent examples where AI tools have hurt the peer review process because these have been reported. So tests, for example, where a series of papers have been evaluated using AI versus human peer reviewers. And these tests have shown that really it’s very difficult for editors to be able to determine if a paper was evaluated entirely using AI or whether a human reviewer was involved in the process. So, I would say that’s an example of AI potentially hurting the peer review process because at the moment, we just can’t tell if a paper that we get at the editorial office was evaluated completely using AI.  

And of course, there are numerous examples of how AI has helped the peer review process. I’m a huge advocate of the use of AI tools, ethically of course, we all are! But also AI tools leveling the playing field for researchers for whom English is not their native language in particular. So I would say there are lots of examples of how AI has hurt the peer review process. Those get reported when we learn about them. But day-to-day researchers doing peer review for journals, for books are using AI tools to help them. And those situations don’t really go reported so much. But nevertheless, we’re living in a fantastic time for the use of AI tools. 

Q: Do you think disclosing the use of AI in peer review will become standard practice in the future? Why or why not?
A: Yes, I do. But this has to be carefully discussed across the publishing industry because AI in my opinion is just a tool, right? And we don’t necessarily have to disclose all of the tools that we use as researchers writing research articles. I use software to help me write papers. I use a computer, for example. These are also tools, but I’m not expected to disclose those when I’m writing a paper or performing a peer review. “Hey! I wish to disclose that I wrote this peer review using a typewriter rather than a computer!” It’s just not expected. And that’s the situation that I really expect that we’re going to get to with the use of AI tools in our workflows as editors, as journal and book publishing academics, and also as researchers. So, at the moment, I think that the publishers are kind of freaking out a little bit about this and worrying that they’re going to publish content that contains AI-generated text and images as well as peer review without being aware of that. But I do think that we’re going to get to a point, I hope, in the publishing industry where everybody gets together and decides and discusses and agrees on a standard set of ethics and criteria. And which industry bodies do we have to oversee that? Well, we don’t really have an effective industry body to oversee that. So, it’s going to be a really interesting couple of years.  

Want to know if your paper is ready for peer review? Get your manuscript evaluated by expert reviewers using our Pre-Submission Peer Review Service.