Token Level

A computer program that processes language does not read sentences the way humans do. It first breaks down every text into small pieces. Such pieces are called tokens: a whole short word, part of a word, a punctuation mark, or a space. From “Fahrradkette” (bicycle chain), for example, three pieces can result: “Fahr”, “rad”, and “kette”. When something is said to happen at the token level, this means: it is decided or measured for each of these individual pieces, not for the sentence and not for the whole text. The token level is thus, in a sense, the finest resolution at which a language model perceives the world.

Why models must think in text building blocks

A model must be able to deal with words it has never seen before. Names, typos, technical terms, and foreign languages occur constantly. A fixed dictionary would fail at this. With word parts, however, almost anything can be assembled. Even a made-up word like “Blubberstuhl” can be represented from known building blocks.

The token level is also the unit in which billing and calculations are made. Providers of AI services charge their prices per thousand or per million tokens. The limit on how much text a model can take in at once is also given in tokens. Anyone reading these figures should roughly know: in German, a token corresponds on average to about three to four letters. A 300-page novel therefore amounts to several hundred thousand tokens.

A common misconception is equating tokens with words. This is often true for short English words, but rarely for German ones. Long compound words like “Krankenversicherungsbeitrag” (health insurance contribution) break down into many pieces. This is why identical texts in German are usually more expensive to process than in English.

One building block after another

A language model does not generate text as a finished block. It calculates a list of probabilities for each next building block. For all possible tokens, it indicates how well they fit at that particular point. Then one is selected, appended to the text, and the calculation starts over. It is exactly this loop that runs at the token level.

This is why measurement and intervention are also possible at this level. How confident a model was at a particular point can be read off building block by building block. If confidence drops sharply, the statement is often a guess. Filters for problematic content can also intervene here, suppressing individual tokens. Even a setting like temperature, which controls randomness in selection, acts at every single step.

Language can also be viewed at coarser levels. At the sentence level, whole statements are evaluated; at the document level, whole texts. These levels are more comprehensible to humans, but the model itself operates more finely. This gap explains some behavior. A model can begin a sentence cleanly and still end it incorrectly, because at the start it had no plan yet for the ending.

Where the token level shows up in news and products

It becomes most visible in the typing effect of a chatbot. The answer appears piece by piece because it is generated piece by piece. What pops up one after another there is essentially the individual tokens. Autocomplete in search fields and programming tools also works according to the same principle.

In finance and tech news, token counts appear as key figures. Companies quote prices per million tokens, context windows of 128,000 or more tokens, and output speeds in tokens per second. These figures are the currency in which the performance and cost of language models are compared. Anyone wanting to evaluate an offer must be able to read them.

One more word of caution: the word token means something completely different in the financial world. There, a token is a digital asset on a blockchain. This has nothing to do with the token level in language processing. On mixed news sites, it is worth briefly paying attention to the context.