# Running AI on a CPU: quantization, ONNX and why small models win

> You do not always need a GPU. How quantization to int8, ONNX Runtime and the right model size make a moderation model answer in milliseconds on a CPU.

Source: https://www.vigimod.com/en/learn/running-ai-on-a-cpu · VigiMod (https://www.vigimod.com) · updated 2026-11-11 · 7 min

## In short

- A model is mostly multiplications of numbers; fewer and smaller numbers mean more speed.
- Quantization stores weights as 8-bit integers instead of 32-bit floats: about 4× smaller, often 2–4× faster.
- ONNX Runtime runs models efficiently on CPUs using their vector instructions (AVX-512, AMX).
- After quantizing, thresholds must be re-checked: scores shift a little.

## Why speed matters

A chat moderation API must answer while the user is still looking at the screen: a few hundred milliseconds at most. GPUs are fast but expensive; ordinary servers have CPUs. Making a model fast on a CPU is what makes a moderation service affordable.

## What a model computes

A neural network is mostly **matrix multiplications**: the numbers representing the text are multiplied by the model's weights, layer after layer. Speed depends on how many multiplications there are (the size of the model and the length of the text) and on how fast each one is.

## Quantization

Weights are normally stored as 32-bit floating point numbers. **Quantization** stores them as 8-bit integers (int8), with a scale factor per row or channel:

| Format | Size of a 560-million-parameter model | Typical speed on CPU |
|---|---|---|
| float32 | about 2.2 GB | baseline |
| int8 | about 0.6 GB | often 2–4× faster |

Modern CPUs have instructions that multiply many 8-bit numbers at once (VNNI, AVX-512, Intel AMX), which is where the speed comes from.

The cost is a tiny loss of precision. With **dynamic quantization per channel**, a good classifier usually loses almost nothing measurable, but scores shift slightly. That is why the thresholds must be **matched again** after quantizing: each threshold is moved so the quantized model flags the same share of messages as the original on a calibration set.

## ONNX and ONNX Runtime

**ONNX** is an open format to save a model as a graph of operations, independent of the framework it was trained with. **ONNX Runtime** executes that graph and is optimised for CPUs: it fuses operations, uses all cores and picks the fastest instructions available.

## A real example

VigiMod 1.2.1 has about 560 million parameters. On 4 CPU cores:

| Version | Time per message–category pair |
|---|---|
| Full precision | about 113 ms |
| int8 ONNX | **about 32 ms** (3.5× faster) |

After matching the thresholds, the quality loss measured on validation data was below 0.005 F1, within the noise.

## Other tricks

- **Shorter inputs:** limit the context to the last messages that matter.
- **Batching:** score several messages at once when they arrive together.
- **Caching the fixed part** of the input (for example a long policy) in models that support it.
- **Smaller model, better data:** a small model trained on the right examples often beats a large general one, at a fraction of the cost.

## When you do need a GPU

Training, and running very large models that generate long answers. A common design is a **cascade**: a fast small model on every message, a larger model only on the few uncertain ones.

## Quiz

1. What does int8 quantization change?
   Answer: Weights are stored as 8-bit integers instead of 32-bit floats. Smaller numbers mean a smaller, faster model.
2. Why must thresholds be re-checked after quantizing?
   Answer: Scores shift slightly. Matching keeps the same share of flagged messages as the original.
3. What is ONNX Runtime?
   Answer: An engine that runs models efficiently, including on CPUs. It executes the model graph using the fastest CPU instructions.
4. What is a cascade?
   Answer: A fast model on every message, a large one only on uncertain cases. It keeps most requests fast and cheap.