Pruning

An AI model consists of many millions or billions of individual numbers. These numbers are adjusted during training until the model masters its task. In the end, it turns out that a large portion of these numbers is tiny and contributes almost nothing to the result. Pruning means throwing out exactly these unimportant parts. The English term comes from gardening and means “to trim,” just as one removes dead branches from a fruit tree. The goal is a model that needs less memory and computes faster, but can do practically the same as before.

Why large models need to slim down

Modern language models are huge. A model with 70 billion numbers takes up over a hundred gigabytes even in economical storage. Something like that only runs on expensive specialized chips in a data center. On a smartphone or in a car, that’s impossible. Pruning is one of the ways to bring a model down to a size that fits on normal hardware.

Then there’s the cost factor. Every single request to an AI model consumes electricity and computing time. With millions of users per day, this adds up to considerable amounts. A model that is thirty percent smaller can accordingly be run more cheaply. For companies, this is often more important than the last few percentage points of accuracy.

A second advantage is response time. Someone using speech recognition in a car doesn’t want to wait two seconds. Smaller models respond faster and can run directly on the device without a detour via the internet. This also protects private data, since it never leaves the device.

What exactly gets removed

The simplest approach is removing individual numbers. All values are sorted by size, and the smallest ones are set to zero. Removing ten percent usually goes unnoticed. Removing seventy percent risks noticeably worse answers. Experts call this variant unstructured pruning.

The catch: scattered zeros bring hardly any speed gain on normal graphics cards. The chip still stubbornly computes through all positions regardless. That’s why structured pruning exists. Here, entire contiguous building blocks are removed, such as a complete layer or a whole row of connections. This saves real computing time, but costs more quality.

Pruning is almost always followed by retraining. The shrunken model is run over sample data once more so that the remaining numbers adapt to the gaps. Without this step, performance often drops noticeably. It’s important to distinguish this from a related technique: with quantization, all building blocks are retained, they are just stored more coarsely. Pruning, by contrast, actually throws something away. Both techniques can be combined.

Pruning in products and headlines

When a provider introduces a “compact” or “mobile” version of its model, it’s usually a mix of pruning, quantization, and knowledge transfer to a smaller model behind the scenes. Voice assistants on smartphones, translation without an internet connection, and image recognition in cameras use such downsized models.

In the business press, the term comes up when discussing data center operating costs. Chip manufacturers now advertise hardware that actually skips scattered zeros and thus benefits from unstructured pruning. A common misconception, by the way, is that pruning makes a model dumber but more economical. Up to a certain point, that’s not true: the removed parts really were superfluous. Only when you cut too far does quality tip over, and often suddenly rather than gradually.