Linear Attention
Linear attention is a resource-efficient computation method in language models: instead of comparing every word to every other word, the model maintains a running summary of the text. As a result, computational effort and cost for long texts grow only steadily instead of exploding.
Programs like ChatGPT process text by checking which words in the text belong together. In the standard procedure, the program compares every word with every other word. With 100 words, that’s 10,000 comparisons; with 1,000 words, already a million. The effort thus grows quadratically: twice as much text means four times as much computational work. Linear attention is a modified variant of this procedure. It dispenses with the complete comparison table and instead works with a continuously updated summary of the text so far. Twice as much text then costs only twice as much computational work.
Why this matters
The length of text a model can survey at once is one of the most important properties of modern AI systems. Users want to paste in entire books, legal texts, or months-long chat histories. This is exactly where quadratic growth becomes a problem.
A model that handles 2,000 words effortlessly chokes on the cost at 200,000 words. The computational work is then not a hundred times as much, but ten thousand times as much. On top of that comes the memory requirement: the usual procedure has to store intermediate results for every previous word. Memory keeps growing with every new word.
Linear attention tackles both problems. Computational effort grows steadily, and memory requirements even stay constant. This is what makes very long inputs economically feasible in the first place.
How it works
A comparison helps. Imagine you have to answer a question about a 500-page novel. The usual procedure flips through the entire novel again for every question and marks all the relevant passages. That’s thorough, but slow.
Linear attention proceeds differently. It reads the novel once and keeps a notepad while doing so. After every page, the notepad is updated. For the answer, the model only looks at this notepad, not at the entire text anymore.
Technically, this notepad is a fixed-size table of numbers, often called the “state.” Every new word changes the numbers in it a little. This is made possible by a mathematical trick: the order of the computational steps is rearranged so that the large comparison table never has to fully come into existence.
The price for this is accuracy. A fixed-size notepad can’t retain everything. It has to compress, and in the process, details get lost. Pure linear attention is therefore weaker at tasks that require exact recall — such as a number that was mentioned once 50,000 words earlier. For this reason, most systems today rely on hybrid forms: part of the model’s layers compute linearly and cheaply, while a smaller part continues to use the full comparisons.
Where you encounter the term
Linear attention is never directly visible. You notice it through product features. When a provider advertises that its model handles a million words of context, such a technique is often behind it. Falling prices per request for long inputs are also partly attributable to this.
In trade news, the names of specific methods come up: Mamba, RWKV, RetNet, or “gated linear attention.” Chinese labs such as MiniMax and Alibaba have released models that mix such layers with classical ones. The technique is also interesting for on-device AI on phones, because the constant memory requirement suits small devices well.
For investors, the question is relevant because it touches on data center demand. If computation per word becomes cheaper, hardware requirements per request decrease. So far, however, the savings have mostly been invested into longer texts and more usage, rather than into smaller data centers.