# Training data and labels: why a model is only as good as its examples

> Where moderation training data comes from, why labels are often wrong, what synthetic data and teacher models are, and why licences matter.

Source: https://www.vigimod.com/en/learn/training-data-and-labels · VigiMod (https://www.vigimod.com) · updated 2026-11-11 · 6 min

## In short

- A model learns from labelled examples: the labels define what “toxic” means for it.
- People disagree on labels, especially on hate and irony; some labels are simply wrong.
- Large models can generate and label examples (teacher → student), but need checks.
- Every dataset has a licence: not all can be used for a commercial product.

## Labels are the definition

When a model is trained, it never sees a dictionary definition of “insult”. It sees thousands of messages that people marked as insults or not. **The labels are the definition.** If they are inconsistent, the model learns an inconsistent idea.

## People disagree

On clear cases everyone agrees. On hate speech, irony, reclaimed slurs or political insults, annotators often split. Studies on hate-speech datasets regularly find only partial agreement between people labelling the same messages. Common fixes:

- several annotators per message, and a majority vote;
- detailed guidelines with examples;
- keeping the “disputed” cases apart instead of forcing a label.

## Some labels are just wrong

In large datasets a share of labels is simply a mistake: a clearly toxic sentence marked as clean, a joke marked as a threat. When we checked the messages our own model flagged in a human-labelled test set, about half of the “false alarms” were actually messages the dataset had labelled wrong. Two consequences:

1. **Training:** the model learns some errors.
2. **Measuring:** even a perfect model looks worse than it is. Serious evaluations build a cleaned “gold” test set.

## Real data, synthetic data, teachers

- **Real messages** written by people are the most valuable: real slang, real mistakes, real context.
- **Synthetic data** is written by a large language model on purpose, to cover what is missing: conversations where context changes the meaning, rare languages, tricky rules.
- **Teacher → student:** a large model (the teacher) labels or writes examples, a small fast model (the student) learns from them. The student can become almost as good as the teacher on that task while being hundreds of times cheaper to run. It also inherits the teacher's mistakes, so samples must be checked by people.

## Licences matter

Public datasets come with licences. Some allow commercial use (CC0, CC-BY, MIT, Apache), others forbid it (“non-commercial”), and some require derived work to use the same licence (“share-alike”). Model outputs can also come with terms of use. A moderation product must check every source before training on it.

## Forgetting

When a model is trained again on new data only, it can **forget** what it knew. The usual fix is a **replay**: mixing a share of the old training data with the new one. It is how a model can learn something new (for example, [not flagging innocent sentences with foreign names](https://www.vigimod.com/en/learn/bias-in-moderation-models)) without getting worse at everything else.

## Quiz

1. For a model, what defines what “insult” means?
   Answer: The labels in its training data. The model only sees labelled examples.
2. What is a “gold” test set?
   Answer: A test set whose labels were carefully re-checked. Re-checked labels make measurements trustworthy.
3. In teacher → student training…
   Answer: a small model learns from examples written or labelled by a large one. The student becomes fast and cheap, but inherits the teacher's mistakes.
4. What is “replay” used for?
   Answer: Avoiding that the model forgets old knowledge when trained on new data. Mixing old data with new data prevents forgetting.