Distillation

Large AI systems like chatbots consist of huge computational models. Such models give good answers, but they require a great deal of computing power and electricity. Distillation is a trick for shrinking that overhead. A large, capable system is made to solve many example tasks. A smaller system then practices delivering the same results. In the end you have a lean program that behaves almost like the large one. The English term literally means distillation: you extract the essential part and leave the rest behind.

Why it matters

Every answer an AI gives costs money, because data centers consume chips and electricity for it. A model that is only a tenth as large costs correspondingly less to run. That’s why distillation is a direct lever on margins for companies.

But it’s not just about costs. Small models fit on devices that don’t need an internet connection. Voice control in cars or text suggestions on phones often run locally. That’s faster, and the data never leaves the device.

A third point makes the technique politically sensitive. Anyone who uses a foreign top-tier model as a teacher can partially skim off its capabilities without paying the billions needed for training. This is exactly what providers regularly argue about, since their terms of use usually forbid it.

How it works

Two roles are distinguished: teacher and student. The teacher is the large, fully trained model. The student is a significantly smaller model that can’t yet do anything.

Now the teacher is asked a great many questions, and its answers are collected. These answers are the practice material for the student. It is adjusted until it answers as similarly as possible to the same questions.

The real trick lies in what the teacher reveals. A language model doesn’t simply decide on one word, but assigns a probability to many possible words. Asked about the capital of Australia, it might name Canberra with high probability, Sydney with medium probability, and Berlin with almost none. This gradation reveals how confident the model is and which alternatives it even considers. The student learns more from this than from a mere model answer.

The comparison with classroom teaching fits well here. A student who only memorizes the correct answer understands little. If, however, they see the typical mistakes and the reasoning behind them, they learn faster. Similarly, the small model uses the teacher’s uncertainties as additional information.

This isn’t free, though. The student generally remains somewhat weaker than the teacher, especially on difficult tasks. And it also inherits the teacher’s errors and biases.

Where you encounter the term

Almost all major providers sell their models in several sizes. The small variants are often called Mini, Flash, Nano, or Lite. They are usually distilled versions of the large sibling model.

In the news, the term comes up especially in disputes. When the Chinese company DeepSeek unveiled a surprisingly strong and cheap model in early 2025, OpenAI accused the provider of having used its systems as a teacher. Such accusations are hard to prove, because answers can barely be traced back unambiguously.

The technique is also present in everyday life without being named. Offline translation, dictation functions, and camera filters are often based on shrunken models. So when providers advertise falling AI prices, distillation is often behind the savings.