Thinking Tokens

Thinking Tokens

Thinking tokens are invisible intermediate steps that an AI model computes internally before its actual answer, in order to better solve more difficult tasks. They make visible that modern language models don't just retrieve an answer, but actively compute something.

A language model — that is, an AI system that understands and generates text — normally outputs an answer immediately. With thinking tokens, it’s different. The model first produces a series of internal intermediate steps before it answers. These steps are called tokens because language models break text down into small units that they generate one after another. The “thinking” tokens are usually not visible to the user — they run in the background. Only afterward comes the actual answer. The more such intermediate steps the model goes through, the better it performs on complex tasks.

Why thinking tokens boost performance

A model that answers immediately has only a limited amount of computation available. For simple questions, that’s enough. For a mathematical proof or a multi-step logic problem, it isn’t.

Thinking tokens expand the available “computation space.” The model can jot down intermediate results, check assumptions, and discard wrong paths before formulating an answer. In benchmarks — standardized tests for AI capabilities — results for math and logic often improve substantially through thinking tokens. OpenAI’s model o3, for instance, achieved results in an olympiad-level math test that no AI system had reached before — largely due to this principle.

A common misconception is that the model is truly “thinking” like a human in the process. In fact, it continues to generate token by token — just more of them before the answer arrives. The difference is quantitative, not fundamentally new. Nevertheless, the practical effect is real and measurable.

How the internal computation works

The model generates the thinking tokens in a shielded area, often called a “scratchpad.” There it writes down considerations — for example: check an assumption, solve a sub-problem, plug in a result. These steps don’t follow any instructions written by hand by humans. They emerge because the model was trained to recognize that more extensive internal deliberation leads to better answers.

How long the thinking process is allowed to be can often be controlled. Developers can set a budget of thinking tokens. A higher budget costs more compute time and thus more money, but delivers better results on difficult tasks. For a simple question like “What is the capital of France?”, lengthy thinking would be pure waste.

Some systems show the thinking tokens to the user upon request. One then sees a kind of rough draft of the solution path — complete with detours, corrections, and self-doubt. This distinguishes thinking tokens from a normal answer, which always appears smooth and finished, even if the path to it was bumpy.

Thinking tokens in current AI products

The principle gained widespread attention when OpenAI released the o1 model series in late 2024. Since then, Anthropic with Claude and Google with Gemini have built in similar features. The term “extended thinking” is often used synonymously, particularly by Anthropic.

In news articles, the topic appears mainly in connection with AI performance on difficult tests — and with costs. Thinking tokens consume more computing capacity than a direct answer. Providers therefore charge more for them. Anyone who activates the feature pays a multiple of the normal price, depending on the length of the thinking process.

For users, the difference is most noticeable in tasks that require multiple steps: debugging code, weighing legal arguments, or solving math problems. For everyday queries, the gain is small. The skill lies in knowing when the expensive thinking pays off — and when a quick answer suffices.

Related Products

Latest News

Subscribe free. Unsubscribe the second it sucks.

High-signal news across AI, business, UX, and tech. Every morning.