Quantization

At its core, an AI model consists of billions of individual numbers. The model set these numbers itself while learning from example data, and they determine how it responds. How precisely such a number is stored can be chosen: very finely, with many decimal places, or more coarsely, with fewer. Quantization is the process of converting all these numbers from a fine to a coarser representation. A comparison: instead of “3.14159265” you simply write “3.14”. The number is still usable, but takes up much less space.

Why large models don’t fit on normal hardware without it

A model’s memory requirements can be roughly calculated. A model with 70 billion numbers takes up around 140 gigabytes in the usual fine representation. That doesn’t fit on any single graphics card you can buy. If you store the same numbers more coarsely, using a quarter of the space, the model shrinks to about 35 gigabytes. Suddenly it runs on a single powerful machine.

The second benefit is speed. When generating a response, the processor has to shuttle all of the model’s numbers through the lines to the compute unit. These lines are often the bottleneck, not the computation itself. Less data therefore directly means shorter waiting times and lower electricity costs.

That’s why quantization is an economic topic, not just a technical one. Anyone running a chatbot with millions of requests per day saves real money with coarser numbers. And for phones and laptops, it is often the only way to run a model locally at all, that is, without a connection to a data center.

How fine numbers become a grid

Technically, a grid is laid over the range in which the numbers lie. Suppose the values of a group lie between minus two and plus two. With 8 bits, 256 possible levels are available. Every real number is rounded to the nearest level, and only the level number is stored. In addition, the program remembers how wide a level was, so it can later convert back.

The rounding errors that arise in this process are called quantization errors. At 8 bits, they are usually so small that you barely notice anything in the result. At 4 bits, that is, only 16 levels, things become considerably trickier. Modern methods therefore don’t treat all numbers the same: particularly sensitive spots in the model remain fine, while uncritical ones are simplified heavily.

Two approaches are common. In one, you quantize the fully trained model afterward, which is quick and doesn’t require large amounts of data. In the other, the coarseness is already factored in during training, so the model can adapt to it. The second approach delivers better results but costs computing time. Quantization should not be confused with distillation: there, a smaller model is trained following the example of a large one, rather than simply storing the numbers of the existing model more coarsely.

How to recognize quantized models in downloads and reports

Anyone downloading freely available AI models immediately runs into abbreviations like Q4, Q8, INT8, or FP16. The number indicates how many bits are used per value. A dozen or so such variants often exist for a single model, ranging from very small and imprecise to large and precise. You choose based on your own graphics card’s memory.

Quantization is also in use everywhere, unnoticed. Speech recognition on phones, face recognition in photo galleries, and the translation feature in browsers use quantized models. Without it, these features would be too slow or would drain the battery.

In financial and tech news, the term usually appears in connection with costs. When a provider reports that it has halved the price per request, quantization is often behind it. A common misconception, by the way, is that a quantized model is a different, smaller model. It is the same model, just with numbers noted less precisely.