Token

Computers calculate with numbers, not with letters. So for a program like ChatGPT to process a sentence, the text must first be broken down and converted into numbers. The pieces into which the text is broken down are called tokens. A token is often a whole short word, but sometimes only a syllable, part of a word, a space, or a punctuation mark. The word “house” is typically one token, whereas “improbability” is cut into several. As a rule of thumb for English texts: one token corresponds to roughly three-quarters of a word.

Tokens as currency and as ceiling

Tokens are the unit in which the AI industry calculates. Anyone using a language model via a programming interface doesn’t pay per question, but per token. Both are charged: the text you feed in, and the text the model outputs. Prices are usually given per million tokens and range, depending on the model, from a few cents to several euros.

Second, tokens limit how much a model can take in at once. This ceiling is called the context window. Once it’s reached, the beginning of the conversation drops away or the model refuses the input. Current models can handle several hundred thousand tokens, which is several hundred pages of a book. A few years ago, it was only a few thousand.

In practice, this means: anyone having a long document analyzed should be able to roughly estimate how many tokens it contains. A novel with 100,000 words has around 150,000 tokens. This determines whether a request is even possible at all and what it costs.

How a text is cut into pieces

A small dedicated program, the tokenizer, is responsible for the breakdown. It works with a fixed list of roughly 50,000 to 200,000 known text pieces. This list is generated automatically beforehand from huge amounts of text. Frequent character sequences get their own spot on the list, while rare ones are assembled from smaller parts. That’s why “the” is a single piece, whereas a made-up name might be a dozen.

Each piece on the list has a number. So the sentence becomes a series of numbers, and the model continues calculating with this series of numbers. When answering, it works in reverse: the model predicts, token by token, which piece comes next. The fact that answers appear word by word on the screen is exactly this process, visible live.

A common misconception is that tokens are the same as words. The difference has real consequences. Because the lists were formed predominantly from English texts, the same content requires more tokens in German than in English – and therefore costs more. And because the model doesn’t see words as sequences of letters, it struggles with tasks like “Count the R’s in strawberry.”

Where the number shows up in everyday life

In product announcements, the token count has become a marketing argument. When a provider advertises a context window of one million tokens, they mean: you can hand over an entire book or a large code collection in a single request. In pricing tables from OpenAI, Google, or Anthropic, costs are generally listed per million tokens, usually split between input and output.

Tokens also play a role in quarterly figures and stock market news. Large providers now state how many trillions of tokens they process per month. This figure is a measure of actual usage – similar to streaming hours for a music service. Analysts use it to gauge whether the business is growing and how much computing power is needed for it.

And finally, anyone building something themselves encounters the term. Anyone embedding AI into an app sees token counts instead of request counts on their bill. Anyone working with long prompts will eventually hit the token limit. Note: the term token means something completely different in IT security, namely a digital access key. In that case, it doesn’t refer to a piece of text, but to a kind of password.