Grouped-Query Attention

Large language models like ChatGPT generate text word by word. With each new word, they look back at everything already written before it. So they don’t have to recompute this constantly, they store notes for every previous word in an intermediate cache. This cache grows with every word and, for long texts, is the biggest cost factor. Grouped-Query Attention is a design principle that shrinks this cache: multiple parallel search processes inside the model access the same set of notes instead of each having its own. Almost all current models are built this way today.

Why the intermediate cache became a bottleneck

A language model doesn’t internally work with a single look back, but with many simultaneously. These parallel looks are called heads. A large model often has 32 or 64 of these per layer, and there are dozens of layers. If every head stores its own notes for every previous word, a very large amount of data builds up very quickly.

Concretely: for a model with 70 billion parameters and a text of 100,000 words, this intermediate cache can reach double-digit gigabyte amounts. That is more than a single graphics card has left over for the model itself. And because this cache is created separately per user, the problem multiplies a hundredfold with a hundred simultaneous requests.

The bottleneck here is not computing power, but shuffling data back and forth between memory and the compute unit. Grouped-Query Attention targets exactly this. Fewer notes mean less data traffic, and less data traffic means faster answers and more users per graphics card. That’s why the technique is less an improvement in intelligence than one in economic efficiency.

From one note sheet per head to one per group

When looking back, each head poses a kind of question: which earlier words are relevant to me right now? This question is called the query. The notes on the earlier words consist of two parts, key and value. The key is the keyword by which the head recognizes whether something fits. The value is the content that is then passed on. Only keys and values need to be stored, since only they are reused.

The classic design gives each head its own keys and values. Grouped-Query Attention instead divides the heads into groups. For example, 32 heads become 8 groups of 4 heads each. Each group has only one shared set of keys and values. The four heads of a group continue to ask their own, different questions, but look up the answers in the same notebook. This reduces the intermediate cache here to a quarter.

One can push this principle to the extreme: with only a single group, all heads share one notebook. This technique existed before Grouped-Query Attention and is called Multi-Query Attention. It saves the most, but costs measurable quality. Grouping is the compromise in between and, in practice, the point where the quality loss almost disappears.

Where GQA appears in models and reports

The technique became well known in 2023 through a research paper by Google and shortly afterward through Meta's Llama 2. Since then it has been built into practically every large language model one can use, for instance Mistral, Qwen, or the Llama successors. In technical model specs, one then finds figures like “32 attention heads, 8 key-value heads”. The ratio of these two numbers is exactly the group size.

In everyday use, one notices the technique indirectly. The fact that chatbots today can process very long documents without usage becoming unaffordable is due, among other things, to this. The fact that smaller models run on a laptop or phone is also barely feasible without this memory savings.

A common misconception: Grouped-Query Attention doesn’t make a model smarter, nor does it shrink it. The number of parameters stays almost the same; only the intermediate cache during answering shrinks. It also shouldn’t be confused with Mixture of Experts, where the model activates only parts of its building blocks per request. Both techniques save resources, but at completely different points, and are often used together.