False positives and false negatives in content moderation

Two ways to be wrong

A moderation system can make exactly two kinds of mistakes:

• A false positive flags something innocent: “I'm going to kill it at the exam tomorrow” treated as a threat, a quote of an insult treated as an insult.
• A false negative lets something harmful through: a veiled threat, an insult with creative spelling, a hateful message written politely.

What they cost

Mistake	Cost for the community
False positive	annoyed users, the feeling of being watched, moderators' time, sometimes people leaving
False negative	harm to the target, a toxic atmosphere, legal and reputational risk

The costs are not symmetric and depend on the community. In a space for minors, a missed grooming attempt is far worse than ten false alarms. In a gaming server, flagging every bit of friendly trash talk kills the fun.

Why you cannot remove both by turning one knob

The threshold moves mistakes from one column to the other: lower it and false negatives turn into true positives, but more false positives appear. With a given model, there is always a trade-off.

Where the common false positives come from

• Missing context: jokes between friends, replies, spoilers (see context in moderation).
• Hyperbole: “I want to die, I have three tests tomorrow 😭” is stress, not self-harm.
• Quotes and counter-speech: reporting or condemning an insult contains the insult.
• Identity words: names, nationalities or religions that were rare in innocent training examples (bias).

Where the common false negatives come from

• Obfuscation: “1d10t”, extra spaces, emojis instead of words.
• Indirect harm: threats without violent words, harassment spread over many polite messages.
• New slang and languages the model saw little of.

How to keep both low

1. A better model on the right data: the only way to improve both at once.
2. Context: send the previous messages, not only the last one.
3. Graded actions: remove only clear cases automatically, send the doubtful ones to a person.
4. Feedback: keep examples of mistakes; they are the most valuable data for the next version.