Multi-Head Latent Attention
Multi-Head Latent Attention is a trick that lets language models use significantly less working memory while processing a long text. Instead of fully storing all intermediate results for every previous word, the model only saves a heavily compressed summary.
A text program like ChatGPT writes its answer word by word. For every new word, it looks at everything that has already been written. So that it doesn’t have to recompute this every time, it stores a few intermediate results for each previous word. With long texts, this intermediate cache becomes huge and fills up the graphics card's working memory. Multi-Head Latent Attention is a design that shrinks this cache. Instead of storing complete intermediate results for each word, it stores a compressed short version. From this short version, the model recalculates the required values whenever needed.
Why this matters
The working memory of graphics cards is the bottleneck when running large language models. It is usually scarcer than raw computing power. The larger the cache per user, the fewer users a server can serve at the same time. That drives up the cost per answer.
Multi-Head Latent Attention reduces this memory requirement by a large factor, depending on the implementation by 80 to over 90 percent. As a result, more requests fit in parallel on the same hardware. It also makes very long inputs practical, such as an entire book or a large codebase. The Chinese lab DeepSeek made the method widely known and used it to explain why its models are so cheap to run.
How it works
Language models work with a mechanism called attention. In it, the model generates several vectors, i.e. long lists of numbers, for each word. Two of these must be kept so that later words can access them. These two are called Key and Value; the memory for them is called the KV cache.
In addition, each layer of the model has several parallel attention channels, called heads. Each head pays attention to different relationships in the sentence. Classically, each head gets its own Key and Value. So memory grows with the number of heads, the number of layers, and the length of the text.
Multi-Head Latent Attention compresses these vectors beforehand. For each word, only one small shared vector is created, the latent vector. Only this one ends up in memory. If a head later needs its Key or Value, it is reconstructed from the latent vector via multiplication. You can think of it like a compressed image file: what gets stored is a small file, but what gets displayed is still a full image. Some information is lost in the process. However, the method is trained in from the very beginning, so the model learns to pack what’s important into the short version. In tests, quality remains practically unchanged as a result.
Where you encounter the term
In everyday use, you don’t notice the technology directly. You only notice the consequences: low prices per request and models that can read very long documents in one go. In trade news, the term mainly comes up with the DeepSeek models, where it is mentioned together with Mixture of Experts. Mixture of Experts saves computing time, Multi-Head Latent Attention saves memory. Together, both explain the strikingly low operating costs of these models.
Anyone reading technical descriptions of models will often find the abbreviation MLA. Alongside it, related methods like Multi-Query Attention and Grouped-Query Attention are frequently mentioned. They pursue the same goal but simply share the Keys and Values among several heads instead of compressing them. For investors, this point is interesting because such memory-saving methods influence how many graphics cards a provider actually has to buy.