AI for Systematic Reviews: How to Use AI in the Systematic Review Process

Key Takeaways:

  • AI can save time and effort in the mechanical stages of a review: deduplication, screening prioritization, and first-pass data extraction. It does not replace reviewer judgment at any stage.
  • Marketers of AI tools for systematic reviews report screening workload reductions of roughly 30-70%, but don’t consider the additional time and effort needed to verify AI output.
  • Every AI-assisted step needs human verification and AI assistance must be described in full: tool name, model version, date accessed, exact prompts, and shareable links to output.
  • General-purpose chatbots like ChatGPT are not bibliographic databases. Use them to draft and translate search strings, never to find the studies themselves.

Table of Contents

What Is an AI Systematic Review?

An AI systematic review is a systematic review in which machine learning, natural language processing, or large language models assist 1 or more stages of the PRISMA workflow. The AI does not conduct the review; it only reduces the mechanical labor inside it. Every eligibility decision, quality rating, and interpretive claim remains attributable to a named human reviewer.

Which Stages of the Review Process Can Be Automated?

Automation readiness varies sharply across the 9 core PRISMA stages. The table below maps where AI is dependable, where it needs heavy supervision, and where it should not be used at all.

Review stage Automation readiness What AI does today
Protocol design and question framing Not recommended Nothing substantive. Scope, PICO elements, and eligibility criteria are methodological judgments.
Search strategy development Partial Drafts Boolean strings, suggests MeSH and Emtree terms, translates syntax between databases.
Deduplication High Matches records across databases using fuzzy title, DOI, author, and year logic.
Title and abstract screening High Ranks records by predicted relevance so eligible studies surface early in the queue.
Full-text retrieval and eligibility Partial Locates PDFs, flags probable exclusions and proposes a reason for each.
Data extraction Partial Populates structured fields from full text. Requires 100% human verification against the source.
Risk of bias assessment Partial Finds candidate text for each domain. Does not produce the final rating.
Statistical synthesis Partial Writes and checks analysis code; converts between effect measures.
Interpretation, GRADE, conclusions Not recommended Nothing. Certainty ratings and implications require domain and contextual reasoning.

How to Choose an AI Systematic Review Tool

Choose an AI systematic review tool that you can defend to a peer reviewer and reproduce your search 2 years later.

Core selection criteria:

  • Published validation: recall or sensitivity figures on recognized benchmark datasets, ideally reported as WSS@95 (Work Saved over Sampling at 95% recall).
  • Every AI suggestion, human override, and timestamp can be exported as a file. This makes the systematic review process recordable and checkable.
  • PRISMA flow export: automatic record counts at identification, screening, eligibility, and inclusion.
  • Model transparency: the tool tells you which algorithm or model version produced each output.
  • You have full understanding of where full texts are stored, whether your inputs train the vendor model, and whether your institution has approved the processor.
  • The tool allows RIS, BibTeX, and EndNote import and clean export to RevMan, Covidence, or Rayyan.

Red flags for an AI systematic review tool:

Don’t use a tool with the following:

  • No published recall figures on any benchmark dataset.
  • Confidence scores with no documented scale or meaning.
  • Screening decisions that cannot be exported in full.
  • No record of which model version generated an output.
  • Marketing that promises a complete systematic review in 24 hours.

Step-by-Step: Running an AI Systematic Literature Review

The workflow below follows PRISMA order. 1 rule governs all of it: every AI-assisted step gets verified by a human, and only human-verified output (not the raw AI output) is what enters the review.

Step 1: Protocol Registration and Question Framing (Human-Led)

  • Frame the question using PICO, SPIDER, or an equivalent structure before touching any tool.
  • Register the protocol on PROSPERO or Open Science Framework before screening begins.
  • Pre-specify AI use in the protocol: which stages, which tool, and what human verification applies.

Step 2: Using ChatGPT to Develop a Search Strategy

This is the stage where a general-purpose LLM adds the most value with the least risk, because every output is verifiable against a database thesaurus in minutes.

  • Generate synonym sets and spelling variants for each PICO concept, including British and US forms.
  • Draft candidate MeSH, Emtree, and CINAHL headings, then confirm each one exists in the actual thesaurus.
  • Translate a validated MEDLINE strategy into Embase, Scopus, Web of Science, and CINAHL syntax, including proximity operator differences.
  • Ask ChatGPT to suggest terms an experienced information specialist would add for this topic.

How to verify AI output

Run every proposed heading against the database itself, then pilot the assembled string against 5-10 known eligible studies. If the string does not retrieve all of them, it is not ready.

Step 3: Deduplication and Record Management

  • Automated deduplication is the safest use of automation in the entire workflow; DOI and fuzzy-title matching are reliable at scale.
  • Review the borderline matches manually. Errata, retractions, and reprints legitimately share metadata with the original.
  • Handle preprint and published pairs by rule: keep the published version, record the link, and note the decision.
  • Keep conference abstracts as separate records unless your protocol excludes them outright.
  • Export the pre-deduplication and post-deduplication counts immediately; PRISMA requires both.

Step 4: Screening Titles and Abstracts in a Systematic Review With AI

Screening is where AI saves the most time but also poses the highest risk. Here’s what you should do:

  • Calibrate first: screen 200-300 randomly selected records manually and measure how the tool ranks your known “includes”.
  • Set the stopping rule before you start, for example 200 consecutive irrelevant records after the last include, and record it in the protocol.
  • Never stop at an arbitrary percentage of the list.
  • Retain 2 independent human screeners for Cochrane-standard reviews. Current guidelines for systematic reviews don’t allow AI as a substitute for a second human screener.
  • If AI acts as a safety net rather than a screener, run it over the excluded pile and re-examine anything it flags.
  • Report inter-rater agreement, including agreement between the AI ranking and human decisions, using Cohen kappa.

Step 5: Full-Text Review and Data Extraction

AI is also efficient at extracting data as follows:

  • Define the extraction template first, with 1 field per variable and explicit units. Vague fields produce vague AI output.
  • Have the model populate the template per study. Require AI to quote the source sentence and page for each value it extracts.
  • Instruct AI to report “not found” for values it can’t populate rather than guess at these values on its own.
  • Verify 100% of extracted numeric values against the PDF. This is not optional because the risk of hallucinations is high here.
  • Expect AI to be less reliable when extracting data from tables, figures, appendices, and supplementary files, especially if these are embedded images. Extract these manually.
  • If you’re running a AI-assisted meta-analysis as well, you need 2 human verifiers for outcome data because a single hallucinated number can drastically alter a pooled estimate.
  • Log every field the model could not populate.

Step 6: Risk of Bias and Quality Appraisal

At this stage, AI can only play a supporting role:

  • Use AI only to locate candidate text: randomization statements, blinding descriptions, attrition figures, and trial registration numbers.
  • Keep the judgment human. RoB 2 and ROBINS-I domains require inference about what authors did not report, which models handle poorly.
  • AI must always quote the exact sentences from which it has picked information.
  • In case of disagreement between AI-flagged signals and reviewer ratings, re-read the paper yourself and don’t write it off as noise.

Step 7: Synthesis, Meta-Analysis, and Write-Up

AI is genuinely useful for analysis code (R packages such as metafor and meta, or Python equivalents, plus effect-size conversions).

What are the RAISE Recommendations for Systematic Reviews?

RAISE (Responsible use of AI in evidence SynthEsis) governs whether and how you use AI in evidence synthesis. It is maintained as a living framework on OSF, with version 3 released on 13 March 2026.

The guidance comes in 3 documents:

  • RAISE 1: role-specific recommendations for 8 groups across the evidence synthesis ecosystem, including synthesists, methodologists, tool development teams, producing organizations, publishers, funders, users, and trainers.
  • RAISE 2: how to build, validate, and evaluate AI tools, including which performance metrics to report.
  • RAISE 3: how to assess, select, and use a tool for a specific synthesis, covering ethical, legal, and regulatory considerations.

Why this matters before you register a protocol: Cochrane, the Campbell Collaboration, JBI, and the Collaboration for Environmental Evidence issued a joint 2025 position statement supporting RAISE, holding synthesists responsible for the decision to use AI, requiring human oversight, and requiring full reporting of any AI use that makes or suggests a judgment.

Benefits and Limitations of an AI Systematic Review

Using AI makes screening easier and faster BUT also requires an additional verification workflow. Whether that trade off is worth making depends on the stage and on how rigorously you check the output.

Aspect What AI improves What it puts at risk
Screening volume Reported workload reductions of roughly 30-70% in published evaluations Missed eligible studies if the stopping rule is not calibrated
Consistency No fatigue at record 4,000; criteria applied identically throughout Systematic, repeated error if the model learned a biased pattern early
Timeline Compresses a 12-18 month review, mainly at screening and extraction Time savings partly consumed by mandatory verification
Data extraction Fast first-pass population of structured fields Hallucinated sample sizes, effect estimates, and confidence intervals
Search development Faster synonym expansion and cross-database translation Invented MeSH headings that silently return 0 results
Reproducibility Open-source models can be versioned and rerun exactly Proprietary model updates make the same prompt irreproducible later

Common Pitfalls in an AI Systematic Literature Review

Pitfall Why it happens How to prevent it
Using an LLM as a search engine Chat interfaces feel like search and answer confidently Search only in indexed databases; use AI-assisted literature search on dedicated tools as a backup
Accepting AI-generated reference lists Fabricated citations look structurally perfect Verify every DOI and PMID against the source record
No pilot test of prompts Pressure to start screening immediately Test on 30-50 records with known decisions before scaling up
No agreement statistic Kappa is not calculated automatically by most tools Compute Cohen kappa between AI and human decisions and report it
Prompt drift across a session Long contexts degrade consistency Restart the session per batch and reuse a fixed prompt file
Sycophantic agreement Models tend to confirm the reviewer framing in the prompt Ask for reasons to exclude and to include in the same prompt
Undocumented model version Tools update silently Record model name, version, and date at every session

Reporting and Transparency: Documenting a Systematic Review With AI

Reviewers and editors increasingly reject AI-assisted reviews not because AI was used, but because its use was described too vaguely to evaluate. That’s where the PRISMA-trAIce guidelines come in.

PRISMA-trAIce Checklist

PRISMA-trAIce (Transparent Reporting of Artificial Intelligence in Comprehensive Evidence Synthesis) was published by Holst et al. (2025). It is a proposed discipline-agnostic extension to the PRISMA 2020 statement, built by extracting and thematically synthesizing items from established AI reporting guidelines such as CONSORT-AI and TRIPOD-AI.

The gap it fills is specific and worth understanding before you adopt it:

  • Existing guidelines such as PRISMA-AI treat AI as the subject of the research being reviewed, not as an instrument used to conduct the review.
  • PRISMA-trAIce addresses the opposite case: you used an AI tool to screen, extract, or search, and readers need to judge whether that process was sound.
  • The checklist spans the full review structure, from title through discussion, and grades each item as mandatory, highly recommended, recommended, or optional.

Status caveat:

PRISMA-trAIce is currently considered a living guideline, maintained on GitHub, with EQUATOR Network registration and Delphi validation planned in the future. Cite the version you used.

How to apply PRISMA-trAIce in practice

  • Read M1 to M10 before registering your protocol, not before submission. M1 cannot be satisfied retrospectively.
  • Archive prompts, parameters, and model versions during the review; many items cannot be reconstructed from memory 8 months later.
  • Build the split flow diagram counts as you screen, since separating AI-handled from human-handled records afterward is usually impossible.
  • Cite both Holst et al.’s paper (doi: 10.2196/80247) and the specific GitHub version, because the checklist is expected to change.

 

Why you shouldn’t use ChatGPT, Claude, etc. for systematic reviews

General-purpose chatbots like ChatGPT or Gemini are capable assistants for text tasks and poor substitutes for purpose-built review platforms. Even the best prompts do not make them suitable for systematic reviews. If you’re not able to access a purpose-built AI systematic review tool, it’s best that you conduct a traditional systematic review or use a chatbot while keeping in mind these (severe) limitations.

Limitation Practical consequence Mitigation
No native database access Cannot retrieve or verify indexed records Search in MEDLINE, Embase, and similar; import results into a reference manager like Endnote or Zotero
Context window limits Cannot hold hundreds of full texts at once Process 1 study per session with a fixed template
Non-determinism The same prompt returns different output on reruns Set temperature to 0 where possible to limit how “creative” the chatbot can get
Hallucinated detail Invented sample sizes, DOIs, and outcomes Require a verbatim source quotation for every extracted value
No audit trail Decisions are not logged in exportable form Export the full transcript and archive it with the protocol
Data privacy exposure Licensed or unpublished full texts leave your environment Use an enterprise or institutional tool that doesn’t retain your input as training data
No dual-screening workflow Conflicts and kappa are not tracked Keep screening inside Rayyan, Covidence, or an equivalent platform

Frequently Asked Questions

Can you use AI to write a systematic review?

You can use AI to assist a systematic review; you cannot use it to create one from scratch. Assistance with screening, extraction drafting, code, and language editing is broadly acceptable when disclosed and properly verified. Generating conclusions or eligibility decisions without human verification is not, and AI cannot be credited as an author under ICMJE rules.

How much time does AI save in systematic review screening?

Many marketers of AI tools claim that their tools save roughly 30-70% of researchers’ time at title and abstract screening, with the largest gains on searches returning more than 5,000 records. Net saving across the whole review is smaller, because verification of AI literature search output consumes quite a bit of time, yet is essential.

Can AI replace the second reviewer in title and abstract screening?

Current Cochrane and JBI guidance does not support this for full systematic reviews. You have to use 2 human screeners for a systematic review and run AI over the excluded records as a safety net, reporting anything it recovers. Rapid reviews may justify a looser standard as their design itself allows having just 1 reviewer. But even in a rapid review, you will need to verify the accuracy of any AI output and document your verification process just as rigorously.

Does Cochrane allow AI in systematic reviews?

Cochrane permits AI assistance in defined stages with mandatory disclosure and human accountability for every judgment. Policies are updated frequently, so check the current Cochrane guidance and your Review Group requirements before registering the protocol rather than at submission. Current Cochrane guidelines say:

Our authors can use AI if it upholds the principles of research integrity. AI and automation must be disclosed, have human oversight, and authors are accountable for the final content. Authors must also justify why they’re using AI and demonstrate that it will not compromise methodological rigor or integrity.

How do you cite ChatGPT in a systematic review?

Most style guides treat it as software rather than a source. Give the developer, model name and version, year, and date accessed, and describe the use in the methods section. Cite the underlying literature separately. Even though style guides like APA offer guidelines for citing AI output, remember that AI output is a much weaker and less-credible source than peer-reviewed literature.

Can AI extract data from full-text PDFs accurately?

AI has acceptable accuracy for narrative fields such as setting, design, and population description, and but AI tools are unreliable for numeric outcomes embedded in tables, figures, and supplementary files. Treat AI extraction as a first draft, verify 100% of numeric values against the source, and extract image data manually.

How do you write a data availability statement for an AI-assisted systematic review?

A defensible data availability statement for an AI-assisted review names 4 things:

  • The screening decision log, including AI relevance scores and every human override, deposited in a repository such as OSF, Zenodo, or Dryad with a DOI.
  • The full prompt set and model parameters, matching PRISMA-trAIce item M6, as a plain text or JSON file rather than as screenshots.
  • The extraction file in both AI-drafted and human-verified versions, so readers can see what verification changed.
  • The calibration sample used to measure recall, with the human decisions attached.

Where full texts cannot be shared for copyright reasons, say so explicitly and share the record identifiers instead. A data availability statement that mentions only the search strategy now reads as incomplete for any review that used AI.

Will journals reject a systematic review that looks AI-written?

Increasingly, yes, though the trigger is rarely the prose itself. Editors screen for signs of AI writing as a proxy for undisclosed AI use in the methodology, and once a manuscript is flagged, the reviewers start questioning the screening and extraction too.

Common signs of AI writing that draw attention:

  • Uniform paragraph length and sentence rhythm across a manuscript with 5 co-authors.
  • Hedged, non-committal phrasing where the results warrant a clear statement.
  • Generic transitions and summary sentences that restate the previous paragraph.
  • Citations that support a general claim but not the specific number attached to them.

The safe position is to disclose any AI assistance as required by your target journal, keep the analytical writing your own, and read the final manuscript aloud once. AI editing and proofreading is accepted by most publishers. Undisclosed AI use combined with a fabricated reference is what leads to rejection.

 

Can you use AI images or AI-generated figures in a systematic review?

For data figures, no. Forest plots, funnel plots, and PRISMA flow diagrams must be generated from your actual dataset by analysis software, because a generated image is not evidence of anything.

Journals and publishers have tighter guidelines about acceptable AI use in images and figures:

  • Conceptual or explanatory graphics, such as a framework diagram, where the figure illustrates rather than reports.
  • Layout and formatting assistance for figures you built from real data.

If AI images appear anywhere in the manuscript, label them in the figure caption with the tool and model version, and check the journal policy first. Several publishers now prohibit generated images outright, and most require declaration. Never use AI to redraw, clean up, or upscale a figure taken from an included study; that alters primary evidence and raises an integrity issue on top of the copyright problem.

Do I need to verify AI output if I’ve use AI just for editing and proofreading?

Yes, though the verification is simpler and much faster than for screening or extraction.

The risk with AI editing and proofreading is not fabrication but meaning drift: AI alters your intention or removes cautious, hedged wording while trying to make your text more polished and fluent.

If you’re using AI for language assistance, keep the pre-edit version of your paper, run a compare-merge in MS Word, and review all edits made by AI before accepting them. For a 6,000-word manuscript that takes 15 to 20 minutes, versus 2 hours of rereading.

 

Summarize this Blog with AI

Comment

There are no comment yet.

TOP