- Inconsistent AI outputs can often be prevented by using specific, narrow prompts.
- Also make a gold-standard test set of task outputs and periodically check your AI outputs against them.
- Always record the exact prompts you used and AI model, for journal peer reviewers.
AI Use in Research and Unpredictability
Large language models (LLMs) like ChatGPT have become useful research assistants. They speed up your literature search, code qualitative responses, find research gaps, support systematic reviews and meta-analyses, and help you draft an entire research paper in hours instead of weeks. However, if you ask the same question twice, you may get two different answers. This is because LLMs are designed with a layer of randomness in their responses. They generate text by predicting likely next tokens, and are supposed to give different answers each time in order to create a “customized” feel for the user.
To get similar or functionally equivalent output for every run of the AI tool you’re using, it’s important to use the right prompts.
What is a prompt?
A prompt is the input you give an AI model (instructions, questions, context, or examples) that tells it what to produce. Its wording, clarity, and structure heavily determine the quality and consistency of the model’s response.
How to Create Stable AI Prompts
A stable prompt produces the same (or functionally equivalent) output when given the same input across repeated runs. Building one starts with a few core principles.
Remove Ambiguity from Every Instruction
Ambiguity is the single biggest cause of inconsistent outputs. When a prompt says “summarize the key findings,” you’re also telling AI to decide what counts as “key,” how long a summary should be, and whether to include limitations. AI can decide that a summary can be 200 words or 500 words, whether p values should be included or not, etc.
Example
Compare these two instructions:
| Vague Prompt | Specific Prompt |
| “Summarize this paper.” | “Summarize this paper in exactly 3 bullet points: (1) the research question, (2) the main method, (3) the primary finding with its effect size if reported.” |
| “Describe the sample” | “Extract sample size, mean age, ethnicity breakdown, income breakdown, and percentage female. If a value is not reported, write NOT REPORTED. Do not estimate.” |
Specific prompts leave fewer decisions to the model, which means fewer opportunities for AI to do as it pleases.
Define Your Terms and Categories
Research relies on operational definitions, and your prompts should too. Let’s look at an example where the AI extracts anxiety prevalence from study abstracts.
The prompt (poor definition):
Read this abstract and extract the percentage of participants with anxiety. Return a single number.
Abstract of Study 1:
“Among 400 university students, mean GAD-7 score was 7.2 (SD 4.1). 38% scored ≥5, indicating at least mild anxiety symptoms, and 14% scored ≥10. Additionally, 9% reported a prior physician diagnosis of an anxiety disorder.”
What AI considers
AI output can range from 9% to 38%. “9%” is chosen when AI prioritizes physician diagnosis. “38%” is chosen when AI assumes that mild anxiety symptoms means having anxiety. The actual, correct answer is 14% as 10 is the cutoff for the GAD-7.
Better prompt
Extract the prevalence of anxiety disorder, defined as the percentage of participants meeting or exceeding the following screening cutoffs:
- GAD-7 ≥ 10
- HADS-A ≥ 11
- BAI ≥ 16
If multiple cutoffs are reported, use only the one listed above. If the study uses a different instrument or cutoff, return OTHER_MEASURE and quote the reported value. If only diagnoses are reported, return DIAGNOSIS_ONLY. If no prevalence is reported, return NOT_REPORTED.
Specify the Output Format Precisely
When AI gives its answers as free text, there’s a lot more variation. Ask AI to provide structured output wherever possible; ask for a fixed format such as JSON, a table, or a labeled list, and describe every field.
For example:
Return your answer as JSON with these fields only: “label” (one of POSITIVE, NEGATIVE, MIXED), “confidence” (LOW, MEDIUM, or HIGH), and “evidence” (a direct quote from the text of 25 words or fewer supporting the label). Do not include any other text.
Structured outputs have a second benefit: they’re machine-readable, which makes large-scale analysis and consistency checks far easier.
Tell the Model What to Do When Uncertain
Many inconsistencies arise when the input doesn’t fit neatly into your instructions. If you don’t tell the model how to handle these cases, it will improvise differently each time. Add explicit fallback rules:
- “If the text does not mention a sample size, return NOT REPORTED.”
- “If the item could fit two categories, choose the one listed first in the codebook and set confidence to LOW.”
- “If the text is not in English, return NOT ENGLISH. Don’t translate it.”
Techniques for AI Prompt Optimization: Refining Your Prompts
Here are further tips to make sure you’re prompting AI correctly.
Use Examples Strategically
Providing worked examples (known as few-shot prompting) is one of the most effective ways to get AI output that meets your expectations. Examples show the model what you mean and what you want.
The best examples you should include are as follows:
- Cover the edge cases. Include examples of tricky inputs, not only obvious ones.
- Balance your categories. If all examples are POSITIVE, the model may lean toward that label.
- Keep the order fixed. Example order can influence outputs, so use the same sequence every run.
- Match the real format. Examples should look like the actual inputs the model will see.
Assign AI a Role and Context
Framing the task with relevant context helps the model adopt consistent standards. For example:
You are assisting with a systematic review of randomized controlled trials in cardiology. Your job is to extract data exactly as reported in each abstract, without inference or interpretation.
This context tells the model what level of rigor is expected and discourages it from hallucinating.
Ask for Reasoning Before the Answer
For complex judgments, asking the model to reason step by step before giving its final answer often improves accuracy and consistency. This means you can also check the accuracy of your prompt and refine it further. For example, let’s say you’re screening article titles while conducting a literature review and instruct AI to “only include longitudinal studies”. But AI might miss a paper like “Amoxicillin for uncomplicated acute appendicitis in adult patients: A 10-year follow-up” simply because the title doesn’t include the word “longitudinal”.
Break Complex Tasks into Smaller Steps
If you tell AI at one go to read a paper, assess its quality, extract ten variables, and write a summary, you’re just asking for trouble. Each additional demand increases the chance of inconsistency. Instead, you should make a chain of smaller prompts to
- Extract study characteristics.
- Assess study quality using a defined rubric.
- Extract outcome data.
- Generate a summary from the extracted fields.
Each output is easier to verify on its own.
Control Your Tool’s Settings
| Temperature is a setting that controls how random an AI model’s word choices are: low values make outputs more focused and repeatable, while high values make them more varied and creative. If your tool allows you to, lowering the temperature reduces randomness in word selection. If you’re using AI for classifying or extracting data, a low or zero temperature is best. But keep in mind that even low temperature doesn’t guarantee perfectly identical outputs on every run nor does it replace the need to verify AI output for hallucinations. |
AI Prompt Testing: Measuring Consistency
You can’t claim a prompt is reliable without testing it. AI prompt testing borrows directly from measurement practices researchers already know, such as test-retest reliability and inter-rater agreement.
Build a Gold-Standard Test Set
Start with a sample of inputs that have been labeled or extracted by human experts. This becomes your benchmark. A good test set should:
- Include somewhere around 50 to 200 items, depending on task complexity.
- Represent the full range of inputs, including difficult and ambiguous cases.
- Be labeled independently by at least two humans, with disagreements resolved.
- Stay separate from the examples used in your prompt, to avoid inflated results.
Run Repeated Trials
Run the same prompt on the same test set multiple times (5-10 runs) and compare the outputs. This reveals how much the model varies with itself.
Then compare the model’s outputs against the gold standard to measure accuracy. A prompt can be highly consistent yet consistently wrong, so you need both measures.
Analyze the Failures
Review every item where runs disagree or where the model diverges from the gold standard, and look for patterns:
- Are errors concentrated in one category?
- Do long inputs cause more mistakes than short ones?
- Are certain phrasings consistently misread?
- Is the model ignoring a specific instruction?
Each pattern points to a specific prompt revision.
Iterate One Change at a Time
When revising prompts, change one element per iteration and retest. If you rewrite the definitions, add examples, and restructure the output format all at once, you won’t know which change helped or hurt. Keep a log of each version and its scores.
Maintaining Stable AI Prompts Over Time
A prompt that works today may not work in six months because AI models and tools frequently get updated. Keeping prompts stable across a long project requires deliberate maintenance.
Version Control
Treat prompts like code. Store each version with a unique identifier, a date, and notes on what changed. Many researchers keep prompts in a Git repository alongside their analysis scripts. At a minimum, keep a prompt log with these fields:
| Field | Example |
| Prompt ID | SENTIMENT-v3.2 |
| Date | 2026-09-15 |
| Model and version | The exact model name and version string used |
| Settings | Temperature 0, max output length 500 |
| Change summary | Added exclusion criteria for sarcasm |
| Test results | Kappa vs. gold standard 0.84; self-consistency 97% |
Note Down the Model Version
Model providers update their systems regularly, and even minor updates can change behavior. Where your tools allow it, specify an exact model version rather than a general name like “Fable” or “Sonnet”. Record the version in your methods section.
Re-Validate After Any Change
If your AI tool updates mid-project, rerun your test set before continuing. Compare the new results to your baseline. If agreement drops meaningfully, you may need to re-process earlier data with the new setup.
Monitor Ongoing Performance
For long projects, periodically spot-check a random sample of outputs against human judgment. A small audit (say, 5% of items) can catch deteriorations in AI performance early, before your entire dataset is compromised.
Documenting Your Prompts for Reproducibility
Journals and publishers increasingly expect clear disclosure of AI use, and good documentation protects the integrity of your work.
Your methods section or supplementary materials should include:
- The full prompt text, including system instructions and examples.
- Model name, version, and access date.
- All settings, such as temperature and output length limits.
- Validation procedures, including the size and construction of your test set.
- Reliability results, reported with the same statistics you would use for human coders.
- Human oversight steps, describing how outputs were reviewed or corrected.
- Known limitations, such as categories where performance was weaker.
Sharing prompts publicly, through a repository or supplementary file, allows other researchers to replicate, critique, and build on your approach.
Comment