How language models read text: tokens, embeddings and attention

Step 1: tokens

A model cannot read letters the way we do. First the text is cut into tokens by a tokenizer. Common words are a single token; rare words are split into pieces:

Text	Possible tokens
“moderation”	moder · ation
“ciao”	ciao
“unbelievable!!”	un · believ · able · !!

Multilingual models share one vocabulary (often 100–250 thousand tokens) across many languages, so “insulto”, “insult” and “Insulte” can end up close to each other. The number of tokens also decides the cost: a model has a maximum length (for example 256 or 512 tokens) and the time it takes grows with it.

Step 2: embeddings

Each token is turned into a long list of numbers, called an embedding (for example 1,024 numbers). You can imagine it as coordinates in a space of meanings: tokens with similar meaning sit close together. “Idiot” and “stupid” are neighbours; “idiot” and “tomato” are far apart.

Step 3: attention

A word alone is ambiguous. In “I sat on the bank of the river” and “I opened a bank account”, the same token means two different things. Attention solves this: at every layer each token looks at all the others and mixes in the information that matters. After a few layers the representation of “bank” is different in the two sentences.

This is the core of the transformer, the architecture behind almost every modern language model. A model is a stack of these layers (12, 24, 36 or more): the first ones capture grammar, the later ones meaning, tone and intent.

Encoders and decoders

There are two big families:

• Encoders (BERT, XLM-RoBERTa, mmBERT) read the whole text at once, in both directions. They produce a representation of the full sentence, perfect for classification: is this an insult? Is this a spoiler? They are small and fast. VigiMod is built on a multilingual encoder.
• Decoders (GPT, Llama, Qwen, Mistral) read left to right and generate text one token at a time. They are great at writing and reasoning, but generating is slow, especially on a CPU.

To classify with a decoder you can avoid generation and read the probability of the first answer token (“yes”/“no”), or attach a small classification head to its internal representation.

From reading to a decision

A classifier adds a small final layer on top of the model: it takes the representation of the text and turns it into a score per category. Training adjusts both this layer and, usually, the model below it (see what is a text classifier).

Why size matters (and why it is not everything)

More layers and more numbers per token mean a model can capture subtler meaning: irony, quotes, context. They also mean more computation for every message. A 0.5-billion-parameter encoder answers in a few tens of milliseconds on a CPU; a 20-billion-parameter decoder that writes its reasoning can take seconds. For moderation in a live chat, a smaller model trained on the right data is often the better choice. How models are made fast on ordinary computers is explained in running AI on a CPU.