Bias in moderation models: why innocent messages with foreign names get flagged

The shortcut problem

A model learns from examples. If, in its training data, the name of a nationality or a religion appears mostly inside hateful messages, the model can learn a shortcut: “this word ⇒ hate”. Then a perfectly innocent message like “Mohammed missed the train this morning” gets a high hate score, only because of the name.

This is called bias, and it hurts exactly the people moderation is supposed to protect.

Where it comes from

• Unbalanced data: hateful messages mention identities often; innocent everyday messages with those identities are rarer in public datasets.
• Annotator bias: people labelling data can be harsher with dialects or slang of some groups.
• Reclaimed words: a group using a word about itself is not the same as an attack.

How to measure it: counterfactual tests

Take sentence templates and fill them with names from different origins:

Template	Names
“{name} missed the train this morning”	Giulia, Mohammed, Wei, Kwame, Dmitri, Priya…
“I had dinner with {name} and her brother”	…
“{name}, you are an idiot”	…

Then check two things for every group:

1. Innocent sentences must not be flagged, and the rate must be the same for every origin.
2. Harmful sentences must still be caught, again equally for every origin.

A model can pass the first test by simply never flagging anything with a name: that is why the second test is essential.

What you can do as a user of a moderation API

• Test with your community's names and languages before going live.
• Look at false positives by group, not only overall.
• Report systematic errors to the provider: they are the most useful feedback there is.

Measuring a model fairly is covered in precision, recall and F1; why labels matter in training data and labels.