KV Cache

A chatbot doesn’t write its answer all at once, but word by word. For each new word, the program looks at everything that already stands before it: the user’s question and the text generated so far. Without any aids, it would have to completely recompute this entire text for every single word. The KV cache prevents exactly that. It is a temporary store in which the computed results for every word already processed are kept. For the next word, the program simply retrieves these results instead of computing them a second time.

Why responses would be agonizingly slow without it

The savings grow with the length of the text. At the tenth word, you save little; at the thousandth word, you save enormously. Without a cache, the computational effort for a long response increases roughly quadratically: double the length means roughly quadruple the work. With the cache, the effort grows only linearly with the length.

In practice, this determines whether a product is usable at all. An assistant that writes a page of text in two seconds feels smooth and fluid. The same assistant without a cache would need many times that amount of time and electricity. Since providers pay for every single response, the KV cache is also a pure cost factor.

The price for this is memory space. The cache sits in the fast memory of the graphics card, and it keeps growing with every word. In very long conversations, it can occupy several gigabytes. That’s why only a limited number of users often fit on one graphics card at the same time, even though the model itself has long since been loaded.

What exactly ends up in memory

Language models work with a mechanism called attention. For each word, the model computes three lists of numbers. One of these is called Query, i.e., request; the other two are Key and Value, i.e., key and value. Put simply, the query asks: which earlier words are relevant to me right now? The keys of the earlier words serve as address labels, the values as the actual content.

The crucial point: the key and value of a word no longer change once they are computed. They depend only on that word and the text before it, not on what comes afterward. So they can be kept. Exactly these two lists move into memory, hence the name key-value cache, or KV cache for short.

You can picture this like a card index. Every word processed gets a card with a label and content. When a new word is added, only a single new card is created. The old cards are read, but never rewritten. A common misconception is that the cache stores answers to entire questions. It does not: it stores intermediate results within a response that is currently being generated.

Context windows, price lists, and prompt caching

The KV cache lies behind many figures you read in product announcements. When a provider advertises a large context window, meaning the amount of text a model can take into account at once, the memory needed for the cache is the actual technical hurdle. A window of one million words sounds harmless, but costs a great deal of graphics card memory.

Some providers even sell the cache directly as a feature. Under names like prompt caching, the keys and values of a long, always-identical preamble remain stored across multiple requests. Anyone who, for example, sends along the same company handbook every time then pays significantly less for that portion. Price lists have their own, cheaper rates for this.

The term also appears regularly in research announcements. Methods like Multi-Query Attention or PagedAttention have one main goal above all: making the cache smaller or better managed. So when people talk about faster and cheaper models, surprisingly often it’s not about intelligence but about memory management.