AI Bias in Research: Types, Examples, and How to Reduce Bias

TL;DR

• AI tools come with their own biases. These biases can get amplified in your research.
• AI bias can arise from incomplete training data, prejudices in the training data, the kind of decisions made while designing algorithms, or the way the researcher prompts AI.
• You can reduce AI bias in your study by changing the way you feed AI training data and what prompts you use.

What is bias in research?

Bias in research is any systematic error that distorts a study’s results or conclusions. The key word here is systematic. Bias doesn’t happen by chance or because your RA dropped a test tube.

We’ve discussed the different types of bias in a previous article, but suffice it to say that there are a variety of biases in research, and they’re also there in qualitative research and even in peer review.

What is AI bias in research?

Research is susceptible to AI bias when researchers use AI tools and inherit the biases in them. An AI algorithm or model produces systematic errors that result in unfair, skewed, or discriminatory research data or results.

Example: ChatGPT bias

In an interesting experiment, researchers fed ChatGPT some sentences involving the words “doctor” and “nurse” and asked ChatGPT to interpret them. ChatGPT assumed that the doctor was male and the nurse was female.

Causes of research bias in AI

Incomplete or skewed training data

The data used to train the AI model could itself be incomplete, missing minority groups and with overcoverage or undercoverage of certain ages, genders, skin tones, or races.

Another issue could arise when the training data has more data from sicker patients, just because those patients have more and longer electronic health records (EHRs).

Example

Suppose an AI model is trained to predict which patients with diabetes will develop kidney disease. The training dataset contains far more records from very sick patients because they have frequent hospital visits, multiple diagnoses, and years of detailed electronic health records. The model therefore learns patterns that are common among these highly monitored patients.

When a researcher uses the model on the general public, it may overpredict kidney disease risk. For instance, it might interpret mild abnormalities in CBC tests as strong indicators of impending kidney disease. CBC tests appeared to be strong predictors in the training data partly because the sickest patients generated much more data, not because of their actual predictive value.

Historical prejudices in the training data

The training data could contain outdated labels or biases in those studies that feed into the AI model. For example, a large number of older clinical trials had much more men than women in their samples, and a 2022 study suggested that this sex bias has only reduced but is not eliminated. Now, when another researcher uses that AI model, a sex bias could creep in

• During the literature review: the model returns male-focused evidence and male-focused research summaries

• Research question generation: the model may overlook sex-specific issues when suggesting new research questions

• Diagnostics: the AI model does not realize that symptoms or behaviors or treatment responses may differ by sex and attribute these symptoms, responses, and behaviors to something else.

• Data analysis: the AI may incorrectly pool data from male and female subjects or may ignore sex as a potential confounder, mediator, or moderator.

Example

A researcher developing a new screening system for chronic kidney diseases asks an AI model about which symptoms to prioritize. The AI returns a list that are based on studies where more than 70% of the sample are White and male. The new screening system is thus more likely to miss CKD in women or BIPOC populations.

Choices Made by Humans in AI Model Design/Algorithms

An AI model doesn’t have to be explicitly programmed to favor one group. Human choices about what the model should prioritize and how it should make decisions can unintentionally produce different outcomes for different groups. While developing an AI model, developers make subjective programming choices, such as setting specific optimization goals, feature weighting, or decision thresholds. And these can inadvertently favor majority groups over others.

Example:

Imagine researchers develop an AI model to identify communities that are at high risk of an infectious disease outbreak. They might decide that number of hospital admissions should be a strong predictor.

But suppose the training data contains much more information from large, well-resourced cities? Just because these areas have better disease surveillance and more complete health records. So when researchers then use that AI model, it might underestimate risk in rural communities, where hospital admissions are low because hospitals are far away or inconvenient to access. The researchers could mistakenly conclude that these communities have a lower disease burden when in reality, the algorithm prioritized the wrong thing.

Human Judgment in Prompts

When a researcher is using AI, they choose prompts, label data, select parameters, and interpret outputs in a way that their own unconscious cultural and social biases enter the system.

For instance, a researcher asks Claude “Why do students from low-income families struggle with mathematics?” This question actually hints to Claude to base the answer on family income. Claude’s answer will thus focus more on factors like parental education and family resources, and less on factors like curriculum, school funding, or teaching practices. Now if the researcher uses Claude’s output to formulate a research question or design a questionnaire or interview protocol, the results will again focus on family-income factors.

Researchers should be aware of AI sycophancy, which means that AI models are designed to “please” the user by providing responses that agree with them. So a prompt like “Explain how a flexible work policy improves retention and cite real journal articles” will ignore any research that shows that flexible work doesn’t improve or has no effect on retention.

AI bias mitigation: How researchers can prevent bias in AI systems they use

The following steps can mitigate the risk of bias when you use AI in your research:

• Use neutral prompts. For example, use “has an influence” instead of “improve” or “reduce”, which convey the direction of the effect.

• Write prompts that could return an answer you’d hate, prompts with adversarial framing, and prompts with no stated hypothesis, and log all the output, not just the one you kept.

• Look at the AI’s training data or corpus. For instance, choose a literature search tool that covers non-English sources over one that covers only English sources.

• Set up a diverse research team. Having collaborators from different countries, races, ethnicities, and genders can help you spot the WEIRD (Western, Industrialized, Rich, Democratic) bias in any AI output.

• Preprocess and clean the training data you feed AI, by balancing datasets or using weights.

References

• Cross JL, et al. Bias in medical AI: Implications for clinical decision-making. doi: 10.1371/journal.pdig.0000651
• Casey Laizure S. Caution: ChatGPT Doesn’t Know What You Are Asking and Doesn’t Know What It Is Saying. doi: 10.5863/1551-6776-29.5.558
• Abbott EE, et al. Understanding and Addressing Bias in Artificial Intelligence Systems: A Primer for the Emergency Medicine Physician. doi: 10.1016/j.acepjo.2025.100311
• Barlek MH, et al. The Persistence of Sex Bias in High-Impact Clinical Research. doi: 10.1016/j.jss.2022.04.077