Often when we talk about peer review, we focus on the various forms of journal submissions that undergo close scrutiny. The second day of the International Congress on Peer Review and Scientific Publication included a session on the less discussed topic of funding or grant application reviews. The session, moderated by Trish Groves (Director of academic outreach and advocacy, BMJ), included four presentations on various aspects of grant application.
First up was João Martins (Swiss National Science Foundation) who presented the results of his attempt to examine peer reviewed grant applications submitted to the Swiss National Science Foundation for any potential geographic and/or gender bias. Martins found that local grant reviewers (Swiss-based) were more likely to give lower scores (they were harsh reviewers) and that female applicants received lower scores. Martins advised grant reviewers to be cognizant of these potential biases.
The next two studies presented by Mary Ann Guadagno and Richard Nakamura were focused on and presented by experts from the National Institutions of Health’s (NIH) Center for Scientific Review. Guadagno shared results of a guided evaluation study that sought to know whether CSR’s grant review process helps achieve its mission and to identify areas of success and improvement in CSR’s quality of grant peer review. The study resulted in some useful suggestions for improving best practices for peer review by stakeholders in real time. Nakamura discussed the testing of two application ranking approaches at CSR. Following a change in the NIH scoring system in 2009, there were concerns about the effectiveness of the new scale, which is functionally cut in half for the 50% of applications that are considered competitive. Nakamura’s study found favorable results for the half point scoring system.
Finally, Laura Forsythe (Patient-Centered Outcomes Research Institute, Washington) discussed the roles played by scientists, patients, and stakeholders in the review of research applications in the Patient-Centered Outcomes Research Institute (PCORI) approach to research funding. The results of this study indicated that the perspectives of non-scientists can play a critical role in the review, scoring, and funding of research applications.
The idea behind this well-planned session was to get the Congress attendees to think about different aspects of peer review beyond journal articles. By focusing on the first stage of research – the funding /grant application – this session introduced a new understanding of the peer review process and its role in ensuring the dissemination of high-quality science.
Research integrity and misconduct in science: A report from the 2017 Peer Review Congress
Quality of reporting in scientific publications: A report from the 2017 Peer Review Congress