# Bias in moderation models: why innocent messages with foreign names get flagged

> Moderation models can flag “Mohammed missed the train” as hate. Where this bias comes from, how to measure it and how to remove it.

Source: https://www.vigimod.com/en/learn/bias-in-moderation-models · VigiMod (https://www.vigimod.com) · updated 2026-11-11 · 6 min

## In short

- Models learn shortcuts: if a word appears mostly in harmful examples, the word itself starts to look harmful.
- Names, nationalities, religions and identities are the classic victims.
- You measure bias with counterfactual tests: same sentence, only the name changes.
- It is fixed with data: many innocent examples with those words, plus checks per group.

## The shortcut problem

A model learns from examples. If, in its training data, the name of a nationality or a religion appears mostly inside hateful messages, the model can learn a **shortcut**: “this word ⇒ hate”. Then a perfectly innocent message like “Mohammed missed the train this morning” gets a high hate score, only because of the name.

This is called **bias**, and it hurts exactly the people moderation is supposed to protect.

## Where it comes from

- **Unbalanced data:** hateful messages mention identities often; innocent everyday messages with those identities are rarer in public datasets.
- **Annotator bias:** people labelling data can be harsher with dialects or slang of some groups.
- **Reclaimed words:** a group using a word about itself is not the same as an attack.

## How to measure it: counterfactual tests

Take sentence templates and fill them with names from different origins:

| Template | Names |
|---|---|
| “{name} missed the train this morning” | Giulia, Mohammed, Wei, Kwame, Dmitri, Priya… |
| “I had dinner with {name} and her brother” | … |
| “{name}, you are an idiot” | … |

Then check two things for every group:

1. **Innocent sentences** must not be flagged, and the rate must be the same for every origin.
2. **Harmful sentences** must still be caught, again equally for every origin.

A model can pass the first test by simply never flagging anything with a name: that is why the second test is essential.


## What you can do as a user of a moderation API

- **Test with your community's names and languages** before going live.
- **Look at false positives by group,** not only overall.
- **Report systematic errors** to the provider: they are the most useful feedback there is.

Measuring a model fairly is covered in [precision, recall and F1](https://www.vigimod.com/en/learn/precision-recall-f1); why labels matter in [training data and labels](https://www.vigimod.com/en/learn/training-data-and-labels).

## Quiz

1. Why can a model flag “Mohammed missed the train” as hate?
   Answer: It learned a shortcut from data where the name appeared mostly in hateful messages. Unbalanced data teaches shortcuts.
2. What is a counterfactual test?
   Answer: Same sentence, only the name or identity changes. Changing only the name isolates the bias.
3. Why must harmful sentences with names also be tested?
   Answer: A model could “pass” by never flagging anything with a name. Removing bias must not create blind spots.