{"id":46959,"date":"2026-05-14T09:30:00","date_gmt":"2026-05-14T04:00:00","guid":{"rendered":"https:\/\/www.editage.com\/insights\/?p=46959"},"modified":"2026-05-13T16:43:30","modified_gmt":"2026-05-13T11:13:30","slug":"how-ai-handles-peer-review-responses","status":"publish","type":"post","link":"https:\/\/www.editage.com\/insights\/how-ai-handles-peer-review-responses","title":{"rendered":"Can AI Handle Peer Review Responses? We Tested It:\u00a0Here\u2019s What We Found\u00a0"},"content":{"rendered":"\n

The Tempting Shortcut<\/strong> <\/p>\n\n\n\n

You\u2019ve received reviewer comments. Some are straightforward, others less so. You draft your responses, but you\u2019re unsure: have you addressed everything clearly? Is the tone right? Are you missing something? <\/p>\n\n\n\n

It\u2019s increasingly common at this stage to turn to generative AI tools, pasting in the reviewer comment and your response, and asking the tool to refine or validate it. The output is often polished, structured, and reassuring. But how reliable is that reassurance? To explore this, we ran a structured test to evaluate how well AI tools assess the adequacy of responses to peer review comments.<\/p>\n\n\n\n

Our AI Experiment<\/a><\/p>\n\n\n\n

What AI Does Well<\/a><\/p>\n\n\n\n

What AI Does Not Do Well<\/a><\/p>\n\n\n\n

Why Human-Driven Response Letter Cross Check is Important<\/a><\/p>\n\n\n\n

Our AI Experiment<\/strong> <\/h2>\n\n\n\n

We evaluated four widely used AI tools, Perplexity, GPT, Claude, and Gemini, on their ability to assess whether author responses adequately address peer reviewer comments. <\/p>\n\n\n\n

We created five representative cases, each consisting of an abstract, a peer reviewer comment, and an author response. Each case included at least three distinct issues identified by the peer reviewer, resulting in 15 evaluation points across the dataset. Across the four tools, this amounted to 60 observations (excluding repeated prompts used to assess consistency and drift). <\/p>\n\n\n\n

Each AI tool was prompted to evaluate the adequacy of the author\u2019s response based on the abstract and reviewer comment.\u00a0Here is the full\u00a0report<\/a> <\/strong>including screenshots of AI outputs as well as the abstracts, reviewer comments, and responses tested.\u00a0<\/p>\n\n\n\n

What AI Does Well<\/strong> <\/h2>\n\n\n\n

Before examining the limitations, it\u2019s worth noting areas where AI tools performed reasonably well. Across cases, AI tools were able to improve the clarity and readability of responses, summarize and restate author arguments in a structured way, identify some surface-level gaps (e.g., when a reviewer point was not addressed at all), and provide organized feedback formats that can help authors revise responses more efficiently.  <\/p>\n\n\n\n

These strengths make AI useful as a language and structuring aid, particularly in the early stages of drafting or refining responses. However, as the analysis below shows, adequacy in peer review responses requires more than clarity and structure. <\/p>\n\n\n\n

What AI Does Not Do Well<\/strong> <\/h2>\n\n\n\n

1. Misinterpretation of Reviewer\u2019s Intent<\/strong> <\/h3>\n\n\n\n

A recurring issue with the AI evaluations of author responses was misalignment between reviewer intent and AI validation of the response. For instance, when a reviewer questioned what novel contribution was offered by focusing on banks in a knowledge economy study given that banks are already established as knowledge-intensive institutions (by many previous studies), the author shifted to explaining why banks are knowledge-intensive and provided descriptive examples of digital transformation in banking. The reviewer\u2019s concern, however, was about incremental contribution and positioning of the study<\/em>, not whether banks belong to the knowledge economy, which they already agreed with. AI tools nevertheless endorsed the response as appropriate.  <\/p>\n\n\n\n

Similarly, when the reviewer raised a methodological concern about asset turnover being potentially inflated due to its denominator, AI described the author\u2019s switch to an alternative ratio (working capital turnover) as a meaningful or \u201csubstantive\u201d improvement, without engaging with the fact that the reviewer had commented on the divisor (the denominator), whereas the author\u2019s justification for the change focused on the numerator, thus not addressing the reviewer\u2019s concern. <\/p>\n\n\n\n

2. Misjudging the Level of Clarification Provided by Authors<\/strong> <\/h3>\n\n\n\n

Another pattern was reflected in a case where a reviewer\u2019s remark questioning the use of a \u201cself-reported survey\u201d was interpreted by the author as a need to define the term, leading to a descriptive explanation of what self-reports are. However, the reviewer\u2019s phrasing functioned as a methodological prompt questioning why subjective self-reporting was appropriate for the construct<\/em>, not what the term meant. AI tools nevertheless judged the clarification as sufficient and appropriate.  <\/p>\n\n\n\n

In another instance, when contextual variables were removed in response to reviewer concerns about their inclusion, AI endorsed this as a satisfactory resolution, without engaging with the alternative and arguably more appropriate solution of repositioning them as background rather than eliminating them entirely. <\/p>\n\n\n\n

3. Inconsistent Reasoning<\/strong> <\/h3>\n\n\n\n

A third issue was internal inconsistency in AI reasoning across related outputs. In one case, AI treated banks as a distinct category of knowledge-intensive firms and elsewhere framed them as a standard representative case within the broader knowledge economy in response to a reviewer\u2019s concern about why the authors had considered banks. In another case involving GPS-based positional tracking, AI responses varied significantly: it alternately assumed consistency in device usage by the researcher, dismissed the relevance of device variation altogether by excluding this information from its outputs, and elsewhere framed the use of multiple devices as a methodological strength (even though it called this a weakness in several other instances). <\/p>\n\n\n\n

4. AI Hallucinations<\/strong> <\/h3>\n\n\n\n

Finally, hallucinations were observed in some cases, where AI introduced unsupported institutional or geopolitical framing. For example, in one response evaluation involving a study on space and national security, the AI expanded the author\u2019s mention of European countries (Spain, France, Italy, Germany, and Greece) into a broader geopolitical narrative. It incorrectly grouped the European Space Agency (ESA) and EU space programmes such as Galileo under a single \u201cEU bloc policy framework,\u201d despite ESA being an intergovernmental organization that includes non-EU members and operates independently of EU governance structures. It further justified the country selection by linking these states to \u201cBlack Sea regional tensions\u201d and broader NATO\u2013Russia dynamics, even though several of the listed countries (e.g., Spain and Germany) have no direct geographical or strategic connection to the Black Sea region. These additions were not present in the original input: they were neither requested by the reviewer nor constituted a part of the author\u2019s response. <\/p>\n\n\n\n

Why Human-Driven Response Letter Cross Check is Important <\/strong> <\/h2>\n\n\n\n

Alignment between Responses and Manuscript<\/strong> <\/h3>\n\n\n\n

Responding to reviewer comments is only one part of the revision process. Equally important is ensuring that the manuscript itself reflects the claimed changes.  <\/p>\n\n\n\n