Speculative Decoding

Programs like ChatGPT write their answers piece by piece. Each new word requires a complete computational pass through a very large program. That costs time, and the user waits. With Speculative Decoding, a second, much smaller program helps out. It guesses several upcoming words in a flash. The large program then checks these suggestions together in a single pass. If they are correct, several words are done at once. If they are not correct, correction happens starting at the first wrong spot.

Why this matters

With large AI models, the computation itself isn’t the bottleneck, but rather reloading the model values from memory. For every single word, billions of stored numbers have to travel through the graphics card. Whether you check one word or five words at the same time makes hardly any difference. Speculative Decoding exploits exactly this gap.

The result: answers often appear two to three times faster. Quality remains identical, since the large model always has the final say. For providers, this means lower costs per answer. For users, it means less waiting time. In applications like voice assistants or coding aids, speed is a genuine selling point.

How it works

You need two models. The large one is called the target model, the small one is called the draft model. Both work with the same language and the same vocabulary. The draft model is often twenty times smaller and correspondingly faster.

The process has three steps. First, the draft model writes a short suggestion, typically four to eight words. Second, the target model is presented with this suggestion in full and calculates what it itself would have written at each position. Third, the two sequences are compared. All the words at the beginning where both agree are adopted. At the first disagreement, the rest is discarded, and the target model inserts a word of its own there. Afterward, the next round begins.

A comparison helps: An intern types up a paragraph in advance, the boss reads through it once. Whatever fits stays as is. Starting from the first mistake, she crosses it out and continues writing herself. Because reading is faster than writing, this saves time overall — even if the intern is sometimes off the mark.

What matters is the draft model’s hit rate. If it guesses well, almost all suggestions are accepted and the gain is large. If it guesses poorly, the work was in vain, and the procedure even becomes minimally slower than without it. That’s why the draft model is chosen carefully, often a smaller version from the same model family. With predictable text such as program code, the trick works especially well, because continuations there are often unambiguous.

Where you encounter the term

In products, you don’t see the procedure, you only feel it: answers appear more quickly. Almost all major providers use it in their data centers. The common programs for running AI models also support it by default, such as vLLM or llama.cpp.

In the news, the term comes up when it’s about the cost and speed of AI. Chip manufacturers and cloud providers like to cite Speculative Decoding as a reason for better numbers. Anyone running AI models on their own laptop will also come across it in the settings. Related approaches carry names like Medusa or EAGLE. They dispense with the second model and let the main model itself suggest several words.