The Human Gap in AI-assisted Research: 10 Critical Checks Before You Submit
TL; DR
- AI is not reliable for verifying the technical accuracy, ethical compliance, scientific soundness, and how compelling a manuscript is for a journal’s readership.
- Check research question clarity, with appropriate alignment between the study objectives, hypotheses, methods, findings, and conclusions.
- Manually review the journal’s aims and scope, recently published articles, preferred article type, word limit requirements, target readership, and accepted subject areas.
- Evaluate the accuracy of statistical tests and interpretations, reproducibility of research methodology, and assess if your claims align with the evidence presented.
Artificial Intelligence (AI) continues to prove its dominance in various fields, including the research landscape. Plenty of students, authors, and researchers are gradually acclimatizing themselves to this new era of digital evolution. Yet, they don’t wish to let go of the humanness that makes their research manuscript unique.
Here are ten critical post-AI checks that every author must perform before submitting the research manuscript to a journal.
2. Discipline-specific Terminology
3. Compliance with Reporting Guidelines
9. Citation Accuracy and Relevance
1. Technical Accuracy Checks
AI tools can perform surface-level checks for inconsistencies in a manuscript. But when it comes to technical accuracy and scientific soundness, a human review becomes critical. For example, if you decide to brainstorm research methodology or study design for your investigation, an AI tool can provide suggestions. But the suitability of the suggested methodology to your research, whether it is a qualitative or quantitative or a mixed-methods approach, must be evaluated by you and not the tool.
Technical accuracy also entails checks for experiments abiding the required scientific standards and protocols, verifying the accuracy of data analysis, validating consistency in numerical results, confirming factual correctness, ensuring that AI did not distort the intended scientific meaning when editing, and validating precise presentation of information throughout the manuscript.
2. Discipline-specific Terminology
General AI chatbots like ChatGPT are not trained for academic use. So, if you rely on them to check and enhance the writing in your research manuscript, you are likely to end up with incorrect phrases mixed into your sentences. Here’s a case study outlining the ramifications of a misguided automated translation of a website’s content that referred to “free cooling” of HVAC systems as “free refreshment” [1]. Now imagine such mistakes occurring in research papers!
Which is why authors should always review their manuscripts for correctness of the terminology used. Take these examples.
1. A psychology paper exploring behavioral traits should not mix-up “antisocial” and “asocial” to describe a person’s behavior; the former indicates sociopathic or harmful actions against society while the latter indicates introverted characteristics.
2. If your research covers aspects of data privacy, know whether you are referring to anonymized data (cannot be unmasked) or pseudonymized data (reversible unmasking is possible).
Pay close attention to such jargon and ensure that you use accurate terminology to properly convey the intended meaning.
3. Compliance with Reporting Guidelines
Your findings reported should conform to standardized reporting guidelines like CONSORT, PRISMA, STROBE, ARRIVE, or others depending on your study design [2]. Asking an AI tool to verify this is not ideal.
The checklist that your manuscript should be validated against is often customized and cannot be verified by a generic tool. This is because some journals require you to submit the official checklist of the guideline as a supplemental file, clearly denoting whether each item is complete, absent, or partial. You are expected to indicate the specific section of the manuscript that contains the corresponding information along with line/page number details. This compliance check should be strictly human driven.
4. Data Interpretation Checks
Statistics, data, and numbers. The cornerstone of any research. While AI tools can consolidate, streamline, and analyze data for you, their interpretation must be your undertaking. Are the correct statistical steps implemented for the data and study deign? The result may be statistically significant, but is it scientifically important? These aspects must be determined by a human checker and not an AI tool.
Moreover, there are instances of findings being negative or contradictory, which AI tools may not prioritize when reporting. In fact, they may completely avoid reporting them. But the discretion of tactically reporting these findings without weakening your research argument lies with the authors. You may even utilize expert statistical analysis services to ensure that statistical errors are identified and fixed before journal submission.
5. Journal Scope Match
Matching with journal scope can seem like a job for an AI tool. When there are tools that easily shortlist journals, why can’t they also check the perfect journal fit? Here’s the catch. AI tools can give you a list of journals based on a broad analysis of factors. But it’s up to the authors to decide the best fit.
For example, if your manuscript is reporting a novel diagnostic technique for cervical cancer diagnosis, an AI tool could recommend journals related to gynecology or oncology. But there’s a possibility for a gynecology journal to find the study topic highly specialized and therefore reject the submission citing journal scope mismatch as the reason. Similarly, oncology journals may not find the paper suited for their target audience. What authors need to recognize here is that the study investigates a specific diagnostic technique with an overlap of two subject areas (gynecology and oncology). So, the most suitable journals would be those focusing on cancer of the female reproductive system.
TIP: Perform a manual review of the journal’s aims and scope, preferred article type, recently published papers, target readership, and accepted subject areas to determine the journal fit.
6. Ethical Disclosures
Considering that most publishers allow use of AI tools to a certain extent, disclosing their use, in addition to any other ethical disclosures, becomes necessary. The AI use guidelines clarify that only the authors are responsible for any unethical research practices. This means the human-driven check should include:
- Have all participants provided informed consent
- Are the relevant permissions from ethics committee and institutional review board obtained for human/animal trials
- Is the trial registered following the necessary protocols
- Have proper permissions been obtained to reproduce copyrighted information
- Are all confidential data protected
- Is the AI disclosure statement complete, accurate, and abides by the journal policy
- Are details like author names, affiliations, funding source, author contributions, conflicts of interest provided as required
7. Review the Figures
In 2024, a paper on cell biology published in an open access Frontiers journal contained an AI-generated image with inaccurate representation of a rat’s anatomy, garbled figure labeling, and other misrepresented scientific figures [3], creating quite the stir in the academic publishing landscape. Two years on, a study has determined that such instances are not isolated. A systematic review of 4,700+ published medical materials (papers in peer-reviewed journals and online anatomy courses) identified over 10.8% of AI-generated images with significant anatomical errors [4].
The use of AI for creating, enhancing, and fixing images has been gradually normalized. But that does not take the responsibility of reviewing the figures for accuracy, especially in scientific publishing, away from humans.
8. Reference Verification
Paper mill activity and fabricated references by generative AI tools are major concerns in scientific publishing [5]. So if you’re using AI tools for literature review, ensure that the tool is designed specifically for academia, check whether the references indeed exist, and also make sure that the referenced papers are not retracted.
Keep a watch on paper retractions from time to time. Have those works been cited in your previous publications? Do they need a relook? It’s a common tendency to develop your research from your own previous works. And if retracted papers have been cited in those, do not make the mistake of carrying them forward.
9. Citation Accuracy and Relevance
As you check references, you must also check whether the citations are accurate and relevant. No, it’s not only verifying that the numbering is correct or that the citations actually exist. A more important question is: Does the citation genuinely support the claim you are making? This can only be verified if you delve into the cited work and the evidence you’ve presented to establish a strong relation between the two.
When it comes to relevance, check whether the works are fairly recent. You do not want outdated research methodology, which are no longer entirely relevant or suitable for analysis, to be the basis of your research.
10. Submission Compliance
Overall, check whether all submission requirements have been satisfied. Some journals even provide their own checklists to help you determine if all the necessary steps have been followed. Use them. Manually ticking things off a list is not something AI tools can do! So, use checklists, sometimes even your own customized ones, to make sure every task related to journal submission is completed:
- The research question is right: not too broad, not too specific, and aligns with the study objective
- The research methodology is reproducible. Check factors like inclusion/exclusion criteria, sample selection, equipment details, description of experimental procedures, instrument names, software versions, etc.
- The evidence is strong enough to support your claims and the implications do not overstate the results.
- Balance between findings, interpretations, and limitations in the Discussion section.
- Integrity checks. Is your manuscript under consideration elsewhere? Has the abstract been submitted at a conference? Is there significant overlap with previously published papers? These issues could be problematic.
- Journal-specific checks. Are the supplementary files included? Is the cover letter provided? Are all files submitted in the format required by the journal? Have you specified the word count? Is the paper anonymized? Manually checking these is important because the requirements vary across journals.
If AI helped build your manuscript, it’s time to check what it may have missed! Get Editage’s Post-AI Risk Assessment service before journal submission.
References
1. From translation chaos to an efficient B2B lead machine https://weventure.de/en/case-studies/website-relaunch-ics
2. Equator Network https://www.equator-network.org/reporting-guidelines/
3. Scientific journal publishes AI-generated rat with gigantic penis in worrying incident https://www.vice.com/en/article/scientific-journal-frontiers-publishes-ai-generated-rat-with-gigantic-penis-in-worrying-incident/
4. A Inaccurate Anatomy: Prevalence of Artificial Intelligence–Generated Illustration Errors in Peer-reviewed Aesthetic Medicine Publications https://pmc.ncbi.nlm.nih.gov/articles/PMC13108651/
5. Fabricated citations: an audit across 2.5 million biomedical papers https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(26)00603-3/fulltext
