{"id":46968,"date":"2026-05-14T09:27:00","date_gmt":"2026-05-14T03:57:00","guid":{"rendered":"https:\/\/www.editage.com\/insights\/?p=46968"},"modified":"2026-05-13T16:42:09","modified_gmt":"2026-05-13T11:12:09","slug":"testing-ai-for-peer-review-responses","status":"publish","type":"post","link":"https:\/\/www.editage.com\/insights\/testing-ai-for-peer-review-responses","title":{"rendered":"How AI Tools\u00a0Handle Peer Review Responses: A Case Study\u00a0"},"content":{"rendered":"\n

This report presents an evaluation of four AI tools (Perplexity, GPT, Claude, and Gemini) in assessing whether author responses to peer reviewer comments adequately address the concerns raised. The underlying materials used for this evaluation, including the abstracts, hypothetical reviewer comments, corresponding author responses, and a human expert\u2019s assessment of issues in those responses, are available in the table here<\/strong> <\/a>(you can also download it from the end of the blog); readers are encouraged to review this before proceeding as the case numbers described subsequently pertain to the data summarized in the table.  <\/p>\n\n\n\n

Details of the Experiment <\/a><\/p>\n\n\n\n

What AI Did Well <\/a><\/p>\n\n\n\n

Where AI Faltered <\/a><\/p>\n\n\n\n

AI Tends to Agree with the Input <\/a><\/p>\n\n\n\n

AI tends to hallucinate <\/a><\/p>\n\n\n\n

AI tends to contradict itself <\/a><\/p>\n\n\n\n

Conclusion<\/a><\/p>\n\n\n\n

Details of the Experiment<\/strong> <\/h2>\n\n\n\n

For each reviewer comment, at least three distinct issues were identified, resulting in a total of 15 evaluation points across five cases (60 observations across four tools, excluding repetitions\/iterative prompts used to check for consistency and drift, which would further increase this number). <\/p>\n\n\n\n

The primary objective of this exercise is to demonstrate the continued importance of human intervention in response letter crosschecks, particularly in identifying gaps, misinterpretations, or incomplete justifications in author replies. <\/p>\n\n\n\n

Editage\u2019s Response Letter Check<\/strong><\/a> (RLC) service is designed to support authors in strengthening their revision submissions by systematically reviewing whether responses adequately address reviewer comments and whether corresponding changes have been appropriately incorporated into the manuscript. <\/p>\n\n\n\n

For this evaluation, each AI tool was prompted to assess the adequacy of author responses, with inputs consisting of the abstract, the reviewer comment, and the author\u2019s response. <\/p>\n\n\n\n

Note:<\/strong> The tools were NOT ASKED TO GENERATE RESPONSES themselves. This mirrors the approach followed in Editage\u2019s RLC service, where authors provide their own responses. Several aspects of a revision require direct author input, and neither humans nor AI should assume how a reviewer\u2019s concern has been addressed. For instance, if a reviewer questions a methodological choice, the author may respond by revising the method itself or by acknowledging the limitation; these are distinct approaches, and it is not appropriate to infer one in the absence of explicit information. <\/p>\n\n\n\n

What AI Did Well<\/strong> <\/h2>\n\n\n\n