Innovations in peer review and the way forward: Interview with Helen Buckley Woods
The scholarly publishing system is in a state of flux, more so now than ever before. We are seeing exciting new technologies, promising experiments with publishing processes and models, worrying patterns of unethical practices, and increased emphasis on fairness, rigor, and sustainability.
What does this mean for peer review, then? It’s no mean task to conceptualize improvements and adaptations to a process on which the reliability of scholarly literature has hinged for decades. That’s why the theme for this year’s Peer Review Week—Peer Review and the Future of Publishing—is so timely and apt.
What makes discussions on this topic both fascinating and complex is the diversity of approaches, requirements, and philosophies that already exist in the industry. And that’s why, as we celebrate Peer Review Week this year, we decided to speak with a researcher whose work has focused on understanding exactly this complex diversity.
Meet Dr Helen Buckley Woods, Research Fellow, Research on Research Institute. Helen is a social scientist with a doctorate in higher education studies. Her thesis “Knowledge production and disciplinary practices in a British University: A qualitative cross-disciplinary case study” explored researchers’ views about knowledge production, evaluation, and dissemination in British higher education. She joined the Research on Research Institute (RoRI) in 2019. Before this, she worked as a librarian and information specialist, including ten years at the Sheffield Centre for Health and Related Research (SCHARR).
What, according to your research, are some of the biggest problems that have led to the common belief that peer review is broken?
In researching this area of practice, through my doctorate and with colleagues at RoRI, I’ve repeatedly found that peer review means something very different across fields and disciplines, due to the diverse types of research knowledge that are being produced and different cultures of peer review.
I would say the most prevalent problem cited by those working in the research system is reviewer burden—whether that is finding reviewers, or reviewing, or training others to do so. Another factor well charted in the literature is the challenge of inconsistent and discriminatory results produced by peer review. Human judgment is unique in its affordances, which inevitably leads to both desirable and unwanted outcomes. This means that although there are multifaceted problems with peer review, it is irreplaceable as the foundation of research practice.
Going back to how peer review is enacted along disciplinary lines, in interviews with researchers in “hard pure” disciplines such as fundamental sciences, they spoke about peer review being typically conducted within a close and incrementally expanding circle of peers, within a disciplinary community. In other contexts, particularly applied or transdisciplinary research, peer review was entirely different, beginning with a close group of peers and then expanding to include both a variety of expertise and wider engagement, predominantly as the research topics were relevant to a wider group of “publics” than more basic research areas.
What are some of the most noteworthy innovations in peer review that have attempted to address existing challenges? What can be learned from the outcomes so far?
In RoRI’s recent research on peer review, we identified and compared many approaches to peer review in scholarly publishing, and the innovations being introduced. In our literature overview of six review articles, we extracted information on approaches to peer review (such as open/masked peer review; collaboration and decoupling), reviewer-focused initiatives (such as incentives and support), and technology to support peer review (with reference to future potential models).
In our survey of scholarly communication organizations, we received 54 responses and identified 95 innovations. We considered what, why, who, and how of these initiatives and their level of openness. For example, in biomedical science, where patient involvement is an intrinsic part of the research process, we received two examples where patients were peer reviewers. In these examples, the category of “peers” was broadened to reflect the importance of patients’ perspectives in this type of research. One might also label additional reviews separately indicating the type of expertise utilized.
In our literature review, we used a broad definition of peer review to reflect the range of actors and processes covered in the literature we selected. For example, one sub-theme of innovation—focused and specialized review—captured approaches to peer review that focus on only one aspect or section of a publication, such as soundness, plagiarism, or use of statistics or images. This work is done by various individuals, such as editorial or journal staff, and researchers, and sometimes with the support of specialist software and AI tools.
We also reflected on the clearly established difficulties with peer review and the solutions that have been developed around open access, respectful and fair treatment of others’ work, and increasing efficiency. In this regard, the well-established F1000 model goes a long way as a multi-coverage solution, in that it uses preprints, allows researchers to suggest peer reviewers, has an open peer review, and provides editorial support; moreover, the final articles are curated and indexed by established journals. Clearly this speeds up processes, allows access to a greater pool of reviewers, gives credit for peer review by making it visible, and provides wider access to research knowledge.
However, to take one part of this model, open peer review is not always desirable. And there is evidence that double-anonymous peer review (where reviewer and author identities are concealed) is preferred by some authors and increases acceptance rates, as the work of IOP publishing shows.
In grant peer review, the use of partial randomization of applications has been piloted by several funding bodies in conjunction with peer review. In our research in this area, we found the primary driver for organizations was to increase fairness in decision-making, ensure the diversity of projects and individuals funded, strengthen the perception of how fair their processes were, and comply with legal requirements.
Other motivations included breaking the deadlock when peer review has run its course and overcoming bias against applications presenting “cold” topics or high-risk projects. That is, allowing unfashionable, or outlying topics a chance for proper consideration, without being discarded too quickly, and counteracting risk-averse decision making, based too heavily on applicant’s past performance. In time, with continued use, it will be possible to establish significant conclusions on the effectiveness of this method.
In the short term, numerous pilots have been conducted and new trials are being initiated in this area. We are tracking these initiatives through our work with funders at RoRI.
In a paper that you and your co-authors published earlier this year, you describe four schools of thought on how peer review can be improved (the Quality & Reproducibility school, Democracy & Transparency school, Equity & Inclusion school, and Efficiency & Incentives school). This year’s theme for Peer Review Week will encourage conversations from all these schools of thought. When re-imagining peer review, which factors will be important in ensuring a greater synergy between these schools?
To concentrate on the commonalities rather than the differences between people’s perspectives. Most people have sympathy with all the schools, they just prioritize or feel that one is more important in their context. Facilitating conversations to learn how things are done in different research communities will help to promote the heterogeneity of ways in which peer review can be done, that fit with different knowledge communities.
Also, it’s important for new initiatives to link across and align with developments in other parts of the research system such as in grant peer review, in EDI (equity, diversity, and inclusion) initiatives to decolonize research, and in efforts to improve research integrity.
A lot has changed in scholarly publishing in the last two decades, in terms of absolute volumes of research published, level of interdisciplinarity, geographical trends in authorship, types of ethical challenges, and so on. Many of the problems that plague peer review are due to the process not being sufficiently adaptable, scalable, and sustainable. What do you envision the peer review of the future will be like? Will peer review the way we’ve known it even exist, say, 10 years from now?
I think the innovations will continue to develop along disciplinary/field lines to create an even greater plurality of peer review forms and practices. This is in step with the expansion of approaches such as team science (and the science of team science), transdisciplinary and co-produced research, citizen science, and other engaged models of research which encompass different ways of knowing and multiple perspectives.
Comments
You're looking to give wings to your academic career and publication journey. We like that!
Why don't we give you complete access! Create a free account and get unlimited access to all resources & a vibrant researcher community.
