# How language models read text: tokens, embeddings and attention

> Computers do not read words. How a language model turns a sentence into tokens and numbers, and how attention lets it understand meaning.

Source: https://www.vigimod.com/en/learn/how-language-models-read-text · VigiMod (https://www.vigimod.com) · updated 2026-11-11 · 7 min

## In short

- Text is cut into tokens: whole words, pieces of words or single characters.
- Each token becomes a list of numbers (an embedding) that captures its meaning.
- Attention lets every token look at the others, so “bank” means a river or money depending on the sentence.
- Encoders read a whole text at once and are ideal for classification; decoders generate text one token at a time.

## Step 1: tokens

A model cannot read letters the way we do. First the text is cut into **tokens** by a *tokenizer*. Common words are a single token; rare words are split into pieces:

| Text | Possible tokens |
|---|---|
| “moderation” | moder · ation |
| “ciao” | ciao |
| “unbelievable!!” | un · believ · able · !! |

Multilingual models share one vocabulary (often 100–250 thousand tokens) across many languages, so “insulto”, “insult” and “Insulte” can end up close to each other. The number of tokens also decides the cost: a model has a maximum length (for example 256 or 512 tokens) and the time it takes grows with it.

## Step 2: embeddings

Each token is turned into a long list of numbers, called an **embedding** (for example 1,024 numbers). You can imagine it as coordinates in a space of meanings: tokens with similar meaning sit close together. “Idiot” and “stupid” are neighbours; “idiot” and “tomato” are far apart.

## Step 3: attention

A word alone is ambiguous. In “I sat on the bank of the river” and “I opened a bank account”, the same token means two different things. **Attention** solves this: at every layer each token looks at all the others and mixes in the information that matters. After a few layers the representation of “bank” is different in the two sentences.

This is the core of the **transformer**, the architecture behind almost every modern language model. A model is a stack of these layers (12, 24, 36 or more): the first ones capture grammar, the later ones meaning, tone and intent.

## Encoders and decoders

There are two big families:

- **Encoders** (BERT, XLM-RoBERTa, mmBERT) read the whole text at once, in both directions. They produce a representation of the full sentence, perfect for **classification**: is this an insult? Is this a spoiler? They are small and fast. VigiMod is built on a multilingual encoder.
- **Decoders** (GPT, Llama, Qwen, Mistral) read left to right and **generate** text one token at a time. They are great at writing and reasoning, but generating is slow, especially on a CPU.

To classify with a decoder you can avoid generation and read the probability of the first answer token (“yes”/“no”), or attach a small classification head to its internal representation.

## From reading to a decision

A classifier adds a small final layer on top of the model: it takes the representation of the text and turns it into a score per category. Training adjusts both this layer and, usually, the model below it (see [what is a text classifier](https://www.vigimod.com/en/learn/what-is-a-text-classifier)).

## Why size matters (and why it is not everything)

More layers and more numbers per token mean a model can capture subtler meaning: irony, quotes, context. They also mean more computation for every message. A 0.5-billion-parameter encoder answers in a few tens of milliseconds on a CPU; a 20-billion-parameter decoder that writes its reasoning can take seconds. For moderation in a live chat, a smaller model trained on the right data is often the better choice. How models are made fast on ordinary computers is explained in [running AI on a CPU](https://www.vigimod.com/en/learn/running-ai-on-a-cpu).

## Quiz

1. What is a token?
   Answer: A piece of text the model reads: a word, part of a word or a character. Text is cut into tokens before the model reads it.
2. What does an embedding capture?
   Answer: The meaning of a token as a list of numbers. Tokens with similar meaning have embeddings close to each other.
3. What problem does attention solve?
   Answer: It lets each word take its meaning from the rest of the sentence. “Bank” means different things depending on the other words.
4. Which family is usually best for fast classification?
   Answer: Encoders. Encoders read the whole text at once and do not need to generate anything.