Why peer review should be fun, but can make you cry

The life of an academic is very much focused on and surrounded by the process of peer review.

With peer review, I mean making and receiving comments on manuscripts that we have written or have been asked to read and assess for scientific journals.

Most of us review papers for scientific journals in the hope that we can support a rigorous scientific process in producing knowledge that has been considered from all angles.

I am yet to go through a peer review process that did not add value and improve my own writing and communication of my research.

But it is a tricky process in many ways and just having gone through it again, I wanted to reflect here on why we do peer review and how each of us could perhaps structure feedback in ways that are constructive and has the potential to increase the quality of work that is being assessed.

The challenges of conducting peer review

Peer review has a tremendously important role in keeping science honest and to make sure that what is being published is an adequate account of an issue under scrutiny.

But sometimes peer review and, in particular, getting review comments back is far from fun. Most academics winch when they open the dreaded email and know that the time has come to read what someone thinks about their research.

The opinions can indeed be widely divergent. For example, in a recent peer review of a journal article I am writing with colleagues, we received 11 pages of comments from 3 different reviewers.

Many of these comments were helpful as these sought to clarify our terminology, analysis, and presentation of the results.

However, as we have to respond to each comment, the 11-page document turned into 25 pages, and this is not including the changes we made in the revised manuscript.

This time around I felt that I was writing two new papers: revising the actual manuscript and then writing responses to the reviewer comments.

One author noted that our conclusions were the best piece of the manuscript, whereas the other argued that the conclusions were merely sweeping generic statements that lacked the insights presented in the paper.

One reviewer was very happy with the terminology whereas two others wanted us to use similar terms as they would use.

In the peer review process, we have a double-blind peer review: neither the author(s) nor the reviewer know each other’s identity, or blind* peer review where the author(s) name is shown but reviewer names stay hidden.

Both of these have been put in place in order to maintain some level of integrity and reduce bias.

Although, at times one can easily guess who the author and the reviewer are: just count the most number of publications starting with the same last name.

But anonymity can also lead to overly negative comments and sometimes to even outright manoeuvring, especially when people work in the same area of research and might be competing for the same grants.

My first cry and peer review

When I was submitting my very first journal paper, I had worked hard in condensing my Master’s thesis research into a journal article that was submitted as part of a special issue.

By the time I got my first review comments back, I was holding back tears when a professor at the department read them out to me.

Apparently my work added no new value to the field and was not publishable.

All the other submissions in the special issue moved into revision (opportunity to revise before final submission and publication).

Except mine.

I was devastated. At the time I did not even know how to submit a paper to another journal (I was fresh out of Master’s degree and didn’t even know what the process looked like).

So I just let it go.

My next attempt was a journal paper based on a project I had been leading in Africa.

The peer review comments were conflicting: one reviewer thought the paper was very good, another thought that the literature review was excellent but the results needed more structure, but the third… The third reviewer said my paper was basically rubbish and there was nothing new or valuable in it.

The editor often intervenes in such cases and points out which comments the author needs to address. Here, this was not the case.

But with the support of my professor, I persisted despite the comments, re-wrote what needed rewriting, and learned the submission process by heart.

And alas, my first paper was shortly published after I started my PhD.

I was so full of joy and pride that I had managed to publish my first paper.

But part of the experience also left me upset again. I could not understand how my paper did manage to see the light of day given the very conflicting comments.

With more experience, I do now understand the process better however and I actually find reviewing papers exciting and interesting, and quite fun.

Why reviewing is rewarding

There are several reasons why reviewing is actually a good thing to do (and we do it for free btw**).

The good thing about reviewing other scientists’ work is that you get to see what is novel and new in the field and the kinds of submissions that are in the field at that point in time.

This helps in distinguishing your own thoughts, often about which directions are emerging in a field of research, and keeps you on your toes as well as to what a robust scientific paper should look like.

I find that reviewing papers is actually a good way to keep up to date with the literature and also some of the more innovative methods that authors have been testing in the field.

I encourage my students to review when they can as this also gives them access on how to situate their own research in the field and what they could be thinking about.

Also, engaging in a robust and helpful review of others’ work helps you to become established as an expert and people might seek your advice in the future when it comes to seeking particular experts for committees or initiatives.

At the same time, when I review someone’s work, I also have the responsibility of making sure that whatever is published contributes to what has already been discovered before.

This, of course, is a grey area in many ways because novelty can be claimed in many ways: for example; a new method, a new angle, a new geographic region or sector.

But the main aim is to increase the quality of science.

How to be a helpful reviewer

Although this post has focused on the scientific peer review process, many of these tips relate to the basic principles of giving feedback on reports, which are essential skills in leadership and management as well.

In reviewing, I always look at several things that can improve the manuscript:

1. Clarity of writing/ideas: Is the writer explaining his or her logic clearly? Are there concepts and sentence structures such that they make it easy to understand what the main aim of the manuscript is? Do the methods, results, and discussion contribute to the main aim of the article?
2. Novelty of research/ideas: How is the manuscript contributing to knowledge in this area? Are they citing the main sources and building upon what is already known? Are the methods novel and insights different from what has been published before?
3. Structure of the manuscript/report: Are there specific elements missing from the structure? Is the flow between sections good?
4. Recommendations: What kinds of recommendations emerge based on the research? How could these be implemented innovatively? What are the next steps/knowledge gaps identified?

On top of these, the tips I would give for someone starting out in academia is to understand the field (who has published what, which are the major debates you wish to contribute to) as this helps you to situate your own research in a niche or an area that has been overlooked.

This is particularly why PhDs are often required to spend most of their first year just going through literature and finding the gaps. In some areas this is easier and in others it is not.

Mind you, although I know all of this, it does not mean that all of this comes naturally to me. I find it even more difficult to judge my own writing but I do try to keep these guidelines in mind when I start drafting articles and reports.

The point is that, contrary to science being portrayed as a mere rational, non-emotional process, peer review in particular often raises lots of emotions (not necessarily always tears, mind you).

And that’s ok because it also signals how much our work and that of others means to us and that we are putting our hearts and souls into the papers we write and science that we produce.

*We assume that the author is referring to single-blind peer review here.

**By the way

Dr Johanna Nalau is an ARC Discovery Early Career Research Fellow (2019-2021), and Managing Editor of Climate Risk Management. This blog post was published on Dr. Nalau’ blog (available here), and has been republished here with her permission.