Multi-Head Attention

A program that processes language must understand which words in a sentence belong together. In the sentence “Anna gave Lisa her book back,” it is unclear whose book “her” refers to. Humans resolve this from context. A computer needs a computational procedure for this, one that compares every word with all other words in the sentence and weights the important relationships more heavily. This is exactly what attention does. Multi-Head Attention is the standard extended version of this: the comparison doesn’t run just once, but eight, sixteen, or a hundred times in parallel. Each of these passes is called a head, and each attends to a different kind of relationship.

Why one perspective per word is not enough

Between two words, there isn’t just one relationship, but many at once. “Book” is grammatically the object of “gave.” In terms of content, it relates to “read” and “library.” And it stands in relation to “her.” If a single computational pass had to compress all of that into one number, most of it would be lost. Multiple heads can divide up these relationships.

Studies of finished models show exactly this. Some heads respond almost exclusively to the immediately preceding word. Others connect verbs with their subjects, while still others find the referent of a pronoun. This division of roles isn’t programmed in — it emerges on its own during training.

Practically important as well: all heads compute independently of one another, and thus simultaneously. Graphics cards are built for exactly this kind of parallel work. Older language models had to go through a sentence word by word and couldn’t do this. Multi-Head Attention is one of the reasons very large models can be trained at all within a reasonable amount of time.

Query, key, and value in each head

Each word is first converted into a long list of numbers describing its meaning in context. From this list, each head generates three shorter versions: a query, a key, and a value. You can think of this like a search. The query is the question “Which words are relevant to me right now?” The keys are the profiles with which all other words apply.

The model now compares the query of one word with the keys of all words in the text. If a key matches well, the comparison yields a high number. These numbers are converted into weights that together add up to 100 percent. The head then blends the values of all words together according to these weights. For the word “her,” the information from “Lisa” might make up 70 percent of the mix.

Because each head computes its own queries, keys, and values, each arrives at a different result. In the end, the results of all heads are concatenated and merged into a new list of numbers by an additional computational layer. The arithmetic behind this matters: the heads share a fixed budget. With 512 numbers per word and 8 heads, each head works with 64 numbers. More heads therefore don’t automatically mean more computational effort, but rather a finer subdivision.

In ChatGPT, translators, and image models

Multi-Head Attention is the centerpiece of the Transformer architecture, the blueprint underlying virtually all of today’s language models. A large model contains dozens of layers, and each of them has its own Multi-Head Attention. When you use ChatGPT, Gemini, or Claude, thousands of such comparisons take place for every single word. Translators like DeepL and image generators work this way too.

In trade news, you usually encounter this term when the discussion turns to cost. Comparing every word with every other word becomes expensive for long texts: doubling the text length quadruples the computational effort. That’s why companies cut corners. Techniques with names like Multi-Query Attention or Grouped-Query Attention let several heads share the same keys and values. This saves memory and speeds up responses, but costs a bit of accuracy.

A common misconception is to understand the heads as subject-matter specialists, say, one head for sports and one for politics. That role is played by the experts in a Mixture-of-Experts model, which is something entirely different. Attention heads deal with relationships between words, not subject areas. And they are never switched off individually: in every layer, all heads are always at work.