PagedAttention

A program that writes text remembers everything it has said so far while writing. These notes sit in the memory of a graphics card, and space there is expensive and scarce. In the past, such programs reserved a large, contiguous block of memory for each request right from the start. After all, you didn’t know whether the answer would be ten words or a thousand words long, so you planned generously. PagedAttention does it differently: it splits the notes into many small blocks and only ever allocates as many blocks as are actually needed at that moment. The blocks don’t have to sit next to each other; a table keeps track of where each block is located.

Why scarce graphics memory determines the price

Operating a language model costs money every time someone asks a question. This operation is called inference, as opposed to the one-time training of the model. How expensive a single answer is depends mainly on how many requests a graphics card can process in parallel. If you double that number, the price per answer is halved.

This is exactly where the problem with the old methods lay. Because every request reserved a lot of memory as a precaution, a large portion of the memory was occupied but empty. Measurements showed that in some cases only 20 to 40 percent of the reserved space was actually used. The rest lay idle, and the graphics card could serve fewer users than it actually could.

PagedAttention raises this utilization to over 90 percent. In practice, depending on the situation, this means two to four times more simultaneous requests on the same hardware. This doesn’t make the model smarter, it just makes it answer more cheaply. For chatbot providers, this has a direct effect on the bill.

Blocks instead of one large chunk

The idea is borrowed from operating systems. When you have many programs open on your computer, Windows or Linux also divides memory into equally sized pages. No program gets one contiguous giant block, but rather many scattered pages. A translation table keeps track of which page belongs to which program. The name PagedAttention comes from these pages.

Applied to a language model, this means: the notes for a text are broken down into blocks of, say, 16 positions. If the answer gets longer, the system simply appends another block, regardless of where in memory space happens to be free. A block table keeps track of the order in which the blocks belong to a given request. The actual computation, the attention operation, is rewritten so that it can handle scattered blocks.

A second advantage arises almost automatically. Multiple requests with the same beginning, for example the same system instruction, can share the matching blocks. The text then only needs to be stored once in memory. Only when the answers differ is a block copied. Experts call this copy-on-write, meaning copying only happens at the point of writing.

vLLM and everyday life behind chatbots

PagedAttention was developed in 2023 at the University of California, Berkeley, and is the core of the vLLM software. vLLM is a server that provides language models to many users simultaneously. It is now one of the most widely used tools of this kind and is used by cloud providers, startups, and research groups. Anyone who runs an open model themselves will almost inevitably come across it.

The technology is not directly visible to users. You only notice it indirectly: in the fact that responses arrive promptly even with many simultaneous users. In industry news, however, the term appears regularly, usually in the context of inference costs or data center utilization.

A common misconception is that PagedAttention makes the model itself smaller. It does not. The model’s weights remain unchanged in memory; the savings only occur in the cache for ongoing conversations. Other methods, such as quantization, address exactly the other point and shrink the model itself. Both can be combined, and in practice, they are.