What Do Medical Reviewers Notice in AI-Assisted Manuscripts?

Generative AI can make medical manuscripts more polished, but does it make them more scientifically robust? We explore what medical editors and peer reviewers look for, where AI-assisted writing can go wrong, and what authors should verify before submitting their work.

The Changing Landscape of Medical Publishing

Artificial intelligence (AI) has changed the way researchers prepare manuscripts for publication. Today, you will hardly run into a researcher in medicine or the life sciences who has not used AI. Understandably so, since the scale at which one can do science – and make new discoveries is astounding.

For the medical researcher, generative AI not only provides speed, it also gives the researcher an expansion of their capabilities into areas which they may not have the expertise in – across disciplines – as well as the opportunity to generate new information. This comes with its’ own set of opportunities and challenges.

Medical journals are adapting rapidly changing editorial policies, publishers are developing guidelines for responsible AI use, and researchers are navigating increasingly complex expectations around disclosure and accountability.

A recent article in Annals of Internal Medicine, “How AI Is Affecting Medical Journals,” highlights how generative AI is reshaping medical publishing and the challenges journals face in responding to its growing use.

But there is another dimension to this discussion that deserves attention: What happens when an AI-assisted manuscript reaches peer review?

“Generative AI can certainly “polish” scientific writing, but it cannot elevate its impact. It thus decreases the burden at peer review by making it easier to vet the content for what it is really worth.” – Dr. Chhavi Chauhan, CEO & Founder, Samast AI

For reviewers, the challenge is not simply determining whether AI was used. It also transcends simply polishing the writing. The true test is in evaluating whether the manuscript presents accurate, methodologically sound, and clinically meaningful research. A manuscript may be grammatically flawless and professionally structured, yet contain weaknesses in its research rationale, statistical interpretation, or clinical conclusions.

We explore some of the areas where medical reviewers may look more closely when evaluating AI-assisted manuscripts.

1. When polished language masks weak scientific reasoning

One of the most visible contributions of generative AI is its ability to improve academic writing. Researchers can use AI tools to simplify complex sentences, improve transitions, organize arguments, and make their manuscripts more readable.

These applications can be particularly helpful for researchers writing in English as an additional language. However, improved language does not necessarily translate into improved scientific reasoning.

Medical reviewers assess whether the authors demonstrate a genuine understanding of their research question and its significance.

“I think broadly speaking, AI is fine for polishing English, checking grammar and things like that. The journal guidelines are changing on a monthly or even a weekly basis.” – David Geveaux, Chief Editor, IOP Publishing.

2. When the methodology and reporting don’t add up

Methodological accuracy is particularly important in medical research because errors in study design, data interpretation, or statistical reporting can affect how findings are understood and applied.

AI tools can help researchers describe established research methods and improve the presentation of methodological information. However, they do not necessarily know what happened in the laboratory, clinic, or research setting unless the authors provide accurate information. This creates a potential problem: a manuscript may contain a methodologically plausible description that does not accurately reflect the procedures followed in the actual study.

Other areas requiring careful scrutiny include:

• Whether the stated study design is consistent with the research question and methods.
• Whether the sampling strategy and inclusion and exclusion criteria are accurately described.
• Whether statistical tests are appropriate for the data and study design.
• Whether the reported sample size, participant numbers, and analysis populations are consistent throughout the manuscript.
• Whether potential sources of bias and confounding have been adequately considered.

3. Do the results support the conclusions?

The interpretation of research findings is another area where human scientific judgment remains essential. In medicine, overstated conclusions can have implications for clinical practice and patient care.

Other potential problems include:

• Treating statistically significant findings as necessarily clinically meaningful.
• Generalizing results beyond the population studied.
• Presenting exploratory subgroup analyses as definitive evidence.
• Overlooking non-significant findings or contradictory results.
• Making clinical recommendations that go beyond the study’s findings.

Reviewers must determine whether the conclusions are proportionate to the strength and limitations of the evidence. This requires an in-depth understanding of the study design, clinical context, statistical analysis, and implications. While it may be useful to use AI tools to facilitate the review process, it can not and should not be employed to replace human judgement.

“We are now seeing a big ground swell that AI should become a part of the review process and actually assist reviewers in their peer review reports. We have to be very aware. We have to be very thoughtful about what the boundaries are and always make sure there is a human in the loop.” – Zoe Mullen, Editor-in-Chief, The Lancet Global Health.

4. Can reviewers trust the references and supporting evidence?

References are another important area of concern when generative AI is used in manuscript preparation. By now, most researchers and reviewers are aware of the problem of hallucination. Yet, papers continue to be submitted – and often published as well – that include references that are not relevant or may not even exist.

Reviewers need to look beyond the presence of citations and assess whether the supporting evidence is accurate and appropriately represented.

5. Where clinical nuance and human judgment matter most

Medical research requires an understanding of clinical context that extends beyond the ability to generate technically correct sentences. Allowing AI to conclude or make clinical recommendations is an absolute red flag. Clinical interpretation depends on years of experience and training of the author as well as various other factors such as patient characteristics, disease severity, comorbidities, treatment settings, and differences in clinical practice.

“I am a strong proponent of using AI to evaluate statistical rigor, scope fit, and formatting, but at submission QC, not peer review. Let machines clear the basics so that the human reviewers can focus their expertise on the science and novelty.” – Dr. Chhavi Chauhan, CEO & Founder, Samast AI

For medical reviewers, evaluating these aspects requires domain-specific expertise and an understanding of how research findings relate to actual clinical practice. Reviewers may find this is the place that AI-generated knowledge becomes most evident. A generic or vaguely worded conclusion or discussion section could signal deeper issues within a manuscript.