Precision, recall and F1, explained with moderation examples

Why accuracy is not enough

Imagine 1,000 chat messages, 20 of which are insults. A model that never flags anything is right 980 times out of 1,000: 98% accuracy, and completely useless. When the thing you look for is rare, accuracy hides the problem. That is why moderation is measured with precision and recall.

The four boxes

Every decision falls in one of four boxes:

	Really harmful	Really harmless
Flagged	true positive (TP)	false positive (FP)
Not flagged	false negative (FN)	true negative (TN)

A false positive is an innocent message flagged by mistake; a false negative is a harmful message that slipped through. They have different costs, explained in false positives and false negatives.

Precision

Precision = TP / (TP + FP)

Of everything the model flagged, how much was really harmful? Low precision means many false alarms: your moderators waste time and your users get annoyed.

Recall

Recall = TP / (TP + FN)

Of all the harmful messages, how many did the model catch? Low recall means many misses: harmful content stays in the chat.

A worked example

Out of 1,000 messages, 20 are insults. The model flags 25 messages: 18 are real insults, 7 are innocent.

• TP = 18, FP = 7, FN = 20 − 18 = 2
• Precision = 18 / 25 = 0.72
• Recall = 18 / 20 = 0.90

The model catches almost every insult, but about one flag in four is a false alarm.

F1: one number for both

F1 = 2 × precision × recall / (precision + recall)

F1 is the harmonic mean of the two: it is high only when both are high. In the example, F1 = 2 × 0.72 × 0.90 / 1.62 ≈ 0.80. A model with precision 1.0 and recall 0.1 has an F1 of only 0.18, even if the average of the two is 0.55.

Which one matters for you?

• A children's game may want very high recall: missing a grooming attempt is far worse than a few false alarms reviewed by a person.
• A gaming community full of banter may want high precision: flagging every friendly “you're trash at this game” would drive people away.

You move between the two with the threshold: lower it and recall goes up, precision goes down; raise it and the opposite happens.

Average precision (AP)

F1 depends on the threshold you chose. Average precision summarises the model at every threshold at once: it tells you how well the scores separate harmful from harmless messages, independently of where you put the line. It is useful to compare two models fairly.

A note on the “truth”

All these numbers compare the model with labels written by people, and people disagree, especially on hate speech and irony. If the test labels are wrong, even a perfect model gets a lower score. That is why serious evaluations re-check their test sets (see training data and labels).