Prompt Caching

Anyone working with an AI text system like ChatGPT sends it a text as a task. This text is called a prompt, roughly meaning input or instruction. The system must first process the entire prompt before it writes the first word of its answer. In practice, the beginning of this text is often identical every single time: the same behavior rules, the same manual, the same conversation log from before. Prompt caching means: this unchanging beginning is processed only once, and the intermediate result is kept on hand for the next requests. The second time around, the system skips the work and continues directly after the known part.

What this saves in time and money

Operating an AI model is billed per piece of text. These pieces of text are called tokens; a token is roughly a short word or a syllable. A company running a customer service bot often sends several thousand tokens of instructions along with every customer question. With a million requests per month, the same text is thus paid for a million times, even though it never changes.

This is exactly where caching comes in. The major providers charge only a fraction of the normal price for a reused prompt beginning, often around one-tenth. For long, fixed instructions, the costs for this part drop noticeably as a result. This is the reason why company reports and quarterly figures from cloud providers regularly mention it.

The second benefit is speed. The time until the first word of the answer depends heavily on how much text has to be processed beforehand. If this step is eliminated, the system responds noticeably faster. For very long instructions, the waiting time can be cut in half or reduced even more.

Why only the beginning is reusable

When processing a prompt, the model calculates intermediate values for each token. These values always depend on everything that comes before — never on what comes after. That’s why the intermediate values for the first 2,000 tokens stay the same, no matter which question follows afterward. It is precisely these values that get cached.

This property leads to a strict rule: the cache only takes effect if the text matches exactly from character one onward. An extra space at the beginning, an inserted date, a changed customer name — and the hit is already lost, and everything has to be recalculated. Anyone wanting to use prompt caching therefore deliberately structures their requests: first everything fixed, then everything variable.

A common misconception is that finished answers are stored here. This is not the case. Only intermediate processing steps are stored, not the result. The same question can therefore still yield a different answer with caching. Moreover, such a cache doesn’t last long: a few minutes is typical, hours for an extra fee. Storage space on the graphics chips is scarce and expensive.

Where the technology works in the background

As a user of a chat app, you never see prompt caching, but you benefit from it. In a long chat history, the entire conversation so far is sent along again with every new message. Without caching, every answer would be slower and more expensive than the previous one. With caching, the history stays in the cache, and only the new message is added.

The effect is especially pronounced with programming assistants. These tools often send several files of program code along as context, sometimes tens of thousands of tokens. The code barely changes between two questions, whereas the question itself does. Without caching, such assistants would hardly be affordable.

That’s why the price lists of OpenAI, Anthropic, and Google have their own lines for cached inputs, usually labeled “cached input.” For developers, this is a lever just like the choice of model: those who cleverly organize their prompts pay significantly less for the same performance.