What is a text classifier, and how does it decide?

The idea in one sentence

A text classifier is a program that reads a piece of text and says which categories it belongs to. A spam filter is a classifier with two categories (spam, not spam). A moderation model is a classifier with several: toxic, insult, hate, threat, sexual, self-harm and so on.

From rules to learning

The oldest classifiers were lists of rules: “if the message contains this word, block it”. They are fast and easy to explain, but they fail in two directions:

• they miss everything that is not on the list (a new insult, a typo, another language);
• they block innocent messages that happen to contain a listed word (“the killer whale”, a quote, a joke between friends).

A machine-learning classifier is not given rules. It is given thousands of examples with the right answer, written by people, and it learns by itself which patterns matter. That is why it can recognise an insult it has never seen, in a sentence written in a way nobody listed.

What comes out: a score, not a verdict

For every category the classifier returns a probability between 0 and 1:

Message	toxic	insult	threat
“what a great game tonight”	0.01	0.00	0.00
“you are an idiot”	0.98	0.97	0.01
“I’m going to destroy you in the final 😄”	0.12	0.03	0.08

A score is not a decision. To decide, you compare it with a threshold: above it the message is flagged, below it is not. Choosing the threshold is a trade-off between missing bad messages and flagging good ones, and it deserves its own guide.

Several categories at once

A message can be an insult and a threat at the same time. Moderation models are usually multi-label: each category gets its own independent score, instead of picking a single winner. In VigiMod, for example, an insult, a hate attack or a threat is also counted as toxic.

How it learns

Training works roughly like this:

1. The model reads an example and produces scores.
2. Its scores are compared with the label given by people.
3. Its internal numbers (the weights) are nudged so that next time the answer is a little closer.
4. Repeat over hundreds of thousands of examples.

At the end, the model has not memorised the examples: it has built an internal representation of what makes a message insulting, threatening or harmless. Modern classifiers start from a language model that already understands text in general (see how language models read text) and are then fine-tuned on moderation examples.

Where classifiers go wrong

• Data they never saw. A model trained mostly on English chats is weaker in Turkish or Korean.
• Biased examples. If innocent sentences with certain names or identities were rare in training, the model may associate those names with harm. Bias has a guide of its own.
• Missing context. “He dies at the end” is harmless alone and a spoiler in a chat about a TV series. Classifiers that read the conversation do better.
• Wrong labels. If the training examples are mislabelled, the model learns the mistakes.

In practice

A good classifier is a fast, consistent first reader. It scores every message in milliseconds, so people only have to look at the doubtful ones. Treat its scores as evidence, choose thresholds that fit your community, and keep a human in the loop for serious actions like bans.