Attention

When a computer program processes a sentence, it has to interpret every word in context. In the sentence “The dog chases the cat because it is fast,” it’s unclear who “it” refers to. A human resolves this by looking back at the fitting words in the sentence. Attention is exactly this looking-back, translated into computational steps. The program calculates, for each word, how much every other word contributes to it. Words with a high value flow strongly into the interpretation, unimportant ones barely at all. The English term means “Aufmerksamkeit” in German, and in German it is also called the attention mechanism (Aufmerksamkeitsmechanismus).

Why language models don’t work without attention

Before 2017, language programs processed texts strictly from left to right, word by word. In doing so, they maintained a kind of memory note that got overwritten after each word. In long texts, the beginning got lost in this process. A reference spanning twenty words or more was barely maintainable.

Attention solves this problem because every word is allowed to directly access every other word. There is no detour through a shrinking memory. A reference at the end of a paragraph reaches the first word in a single computational step. That’s why today’s models can keep entire book chapters in view.

The second advantage is of a practical nature. All words can be calculated simultaneously, instead of one after another. This is exactly what graphics cards are so good at, and this is precisely why training huge models became affordable at all. The field dates the beginning of today’s AI boom to the paper “Attention Is All You Need,” which introduced this method.

How the weighting of a word is calculated

In the model, every word is represented as a long list of numbers. From this list, the model generates three new lists, each with a different task. The first describes what the word is looking for. The second describes what the word itself has to offer. The third contains the content that is later passed on. In technical terms, these are called Query, Key, and Value.

You can imagine this like a search in a library. The search query is compared against the labels of all the books. The better the query and the label match, the higher the score. These scores are then converted so that together they add up to 100 percent. The result is a blend of the content in which fitting words dominate.

Modern models don’t do this just once, but in parallel across several separate passes. These passes are called heads. One head might attend to grammatical relationships, another to content similarity. No one sets this by hand; it emerges during learning. One drawback remains: the computational cost grows quadratically with text length. Twice as much text means four times as much work.

Attention in chatbots, image generators, and stock market reports

Every chatbot in use today calculates attention for every word. Translation services, automatic subtitles, and coding assistants also rely on it. Image generators use it to link text descriptions and image regions. Even programs for predicting protein structures use the same principle.

In the news, the term is usually encountered indirectly. When a company advertises a model with “a million tokens of context,” this concerns how much text attention can still handle affordably. Reports about new methods like Flash Attention or Sparse Attention concern precisely this cost issue. They save memory or deliberately leave out words that are presumably unimportant.

A common misconception is that attention is a kind of understanding or even consciousness. It is pure computation with numbers and weights. It should also not be confused with the Transformer, the overall architecture of such models. Attention is its most important component, but still just one component among several.