Running AI on a CPU: quantization, ONNX and why small models win

Why speed matters

A chat moderation API must answer while the user is still looking at the screen: a few hundred milliseconds at most. GPUs are fast but expensive; ordinary servers have CPUs. Making a model fast on a CPU is what makes a moderation service affordable.

What a model computes

A neural network is mostly matrix multiplications: the numbers representing the text are multiplied by the model's weights, layer after layer. Speed depends on how many multiplications there are (the size of the model and the length of the text) and on how fast each one is.

Quantization

Weights are normally stored as 32-bit floating point numbers. Quantization stores them as 8-bit integers (int8), with a scale factor per row or channel:

Format	Size of a 560-million-parameter model	Typical speed on CPU
float32	about 2.2 GB	baseline
int8	about 0.6 GB	often 2–4× faster

Modern CPUs have instructions that multiply many 8-bit numbers at once (VNNI, AVX-512, Intel AMX), which is where the speed comes from.

The cost is a tiny loss of precision. With dynamic quantization per channel, a good classifier usually loses almost nothing measurable, but scores shift slightly. That is why the thresholds must be matched again after quantizing: each threshold is moved so the quantized model flags the same share of messages as the original on a calibration set.

ONNX and ONNX Runtime

ONNX is an open format to save a model as a graph of operations, independent of the framework it was trained with. ONNX Runtime executes that graph and is optimised for CPUs: it fuses operations, uses all cores and picks the fastest instructions available.

A real example

VigiMod 1.2.1 has about 560 million parameters. On 4 CPU cores:

Version	Time per message–category pair
Full precision	about 113 ms
int8 ONNX	about 32 ms (3.5× faster)

After matching the thresholds, the quality loss measured on validation data was below 0.005 F1, within the noise.

Other tricks

• Shorter inputs: limit the context to the last messages that matter.
• Batching: score several messages at once when they arrive together.
• Caching the fixed part of the input (for example a long policy) in models that support it.
• Smaller model, better data: a small model trained on the right examples often beats a large general one, at a fraction of the cost.

When you do need a GPU

Training, and running very large models that generate long answers. A common design is a cascade: a fast small model on every message, a larger model only on the few uncertain ones.