Multi-Query Attention

Programs like ChatGPT generate text word by word. With each new word, they look back at everything already written. So that this look-back doesn’t have to be recalculated every time, the program keeps notes on each previous word. These notes consume memory, and the longer the text gets, the more memory they consume. Multi-Query Attention is a trick that drastically reduces the amount of these notes. Instead of each of the many parallel look-back channels keeping its own notes, all channels share a single set of notes.

Why long texts otherwise become too expensive

The memory for these notes is technically called the KV cache. In large models, it is the actual bottleneck. It’s not computing power that slows things down, but the shuffling of data back and forth between memory and processing core. In a chat with several thousand words of history, this intermediate memory can occupy several gigabytes — per user.

This is exactly where Multi-Query Attention comes in. It can shrink the KV cache by a factor equal to the number of channels. With 32 channels, roughly one thirty-second of it remains. That sounds like a detail for engineers, but it determines how many people a provider can serve at the same time.

The length of the conversation also depends on this. Context windows of 100,000 words or more would be practically unaffordable with the old method. The fact that models today can read entire books in one go is largely due to memory tricks like this one.

One set of notes for all heads

The look-back at previous words is technically called attention. A model doesn’t do this just once, but in many parallel passes. These passes are called heads. One head might attend to sentence structure, another to content-related connections. What exactly a head ends up focusing on is not defined by anyone; it emerges during training.

Each head works with three quantities: a query into the past, plus a key and a value for each previous word. The key and value are the notes that need to be stored. In the classic method, Multi-Head Attention, each head has its own key and its own value.

Multi-Query Attention keeps the many queries but eliminates the diversity in keys and values. All heads access the same key-value set. Picture it this way: thirty experts read the same file note but each ask it different questions. An intermediate approach is called Grouped-Query Attention: there, small groups of heads each share one note. This is the more common compromise today, because pure MQA can slightly degrade answer quality.

Which models contain this trick

Multi-Query Attention is never visible. At most, you notice it in the fact that a chatbot answers quickly and can process long documents. The method was first described in 2019 at Google, but it was only widely adopted with the large language models from 2023 onward.

In technical model descriptions, the abbreviations MQA and GQA appear regularly. Anyone reading the spec sheets of open models, such as the Llama or Mistral series, will usually find Grouped-Query Attention there. The term also comes up in trade news about data centers, since it is directly linked to GPU memory and operating costs.

A common misconception: Multi-Query Attention does not make a model smarter, nor does it shrink it. The number of learned values in the model stays almost the same. The savings apply only to the intermediate memory used while generating answers.