Run on · vigimod-1.2.1
Multilingual hate speech benchmark: VigiMod vs Detoxify
600 test cases per language from HateCheck and Multilingual HateCheck, written by experts to catch the usual mistakes of hate speech models: hateful messages in many forms, and hard harmless ones like counter-speech, quotes, negations, profanity and insults aimed at no protected group. VigiMod was called through its public API, exactly as customers get it; it never trained on these tests.
- VigiMod macro F1, 6 languages
- 93.8
- F1 points ahead of the best Detoxify setting
- +14.9
- Cases scored per system
- 3,592
F1 per language
F1 on the hateful class (higher is better). Bold: the best per language.
| System | English | Italian | Spanish | French | German | Portuguese | Macro* |
|---|---|---|---|---|---|---|---|
| VigiMod (vigimod-1.2.1, hate) | 95.2 | 93.9 | 93.8 | 91.1 | 94.7 | 93.8 | 93.8 |
| Detoxify multilingual (identity_attack ≥ 0.5) | 65.1 | 55.6 | 53.4 | 55.3 | n/s | 57.0 | 57.3 |
| Detoxify multilingual (toxicity ≥ 0.5) | 80.4 | 77.2 | 79.2 | 78.9 | n/s | 78.8 | 78.9 |
n/s: Detoxify's multilingual model does not cover German (its languages are English, French, Spanish, Italian, Portuguese, Turkish and Russian); *Macro: the average over the languages each system supports, so Detoxify is not penalised for German.
Precision, recall and false positives
Precision: how many flagged messages were hateful. Recall: how many hateful messages were flagged. False positive rate: how many harmless messages were flagged, the number moderators feel most.
VigiMod (vigimod-1.2.1, hate)
| Language | Precision | Recall | False pos. |
|---|---|---|---|
| English | 95.2 | 95.2 | 10.8% |
| Italian | 92.8 | 95.0 | 17.1% |
| Spanish | 93.3 | 94.2 | 15.9% |
| French | 92.2 | 90.0 | 17.8% |
| German | 93.7 | 95.7 | 14.8% |
| Portuguese | 93.4 | 94.3 | 15.5% |
Detoxify multilingual (identity_attack ≥ 0.5)
| Language | Precision | Recall | False pos. |
|---|---|---|---|
| English | 84.6 | 52.9 | 21.5% |
| Italian | 88.5 | 40.6 | 12.2% |
| Spanish | 83.6 | 39.2 | 18.2% |
| French | 85.9 | 40.7 | 15.6% |
| German (n/s) | 100.0 | 2.2 | 0.0% |
| Portuguese | 86.4 | 42.5 | 15.5% |
Detoxify multilingual (toxicity ≥ 0.5)
| Language | Precision | Recall | False pos. |
|---|---|---|---|
| English | 77.0 | 84.1 | 55.9% |
| Italian | 77.1 | 77.3 | 53.0% |
| Spanish | 77.8 | 80.8 | 54.5% |
| French | 79.0 | 78.8 | 48.9% |
| German (n/s) | 82.1 | 5.5 | 2.7% |
| Portuguese | 78.6 | 79.0 | 49.7% |
Method
Data. HateCheck (Röttger et al., 2021) for English and Multilingual HateCheck (Röttger et al., 2022) for Italian, Spanish, French, German and Portuguese, both CC BY 4.0. Per language, up to 600 cases with the same share of every test and label (seed 42); the same cases for every system.
Systems. VigiMod: the hate flag of POST /moderate/batch with its default threshold, on the public API. Detoxify: the open-source multilingual model (Unitary), run locally, with its identity_attack score and, separately, its toxicity score at 0.5.
What is missing. Google's Perspective API has taken no new users since February 2026 and shuts down on 31 December 2026, so it could not be run. HateCheck tests hate speech only: other categories (threats, sexual content, self-harm, spam) and conversation context are not measured here.
Reproduce it. The cases, every prediction and the per-test accuracy are below. Texts are from HateCheck and Multilingual HateCheck, CC BY 4.0.
Try it on your own messages
Free checker without an account, or a free API key during the beta.