Transformer

A Transformer is a particular design for computer programs that learn from examples. Such programs are called neural networks: they consist of many computational steps whose settings are not fixed by a human but adjust themselves during learning. The Transformer was introduced in 2017 by researchers at Google in a paper titled “Attention Is All You Need.” Its special feature is the way it handles text: it does not read word by word from left to right, but processes an entire passage of text all at once. In doing so, it calculates for each word which other words in the sentence currently matter to it. Almost all well-known AI systems for language – from ChatGPT to translation services – follow this blueprint. The “T” in GPT stands for exactly this.

The blueprint that triggered the AI wave

Before the Transformer, language programs worked step by step: they read one word, retained something of it, then read the next. This is slow, because one step has to wait for the previous one. Moreover, the memory of the beginning of the sentence faded as the text grew longer. The Transformer solves both problems at once.

Because it processes all words simultaneously, the computational work can be spread across thousands of graphics chips. This is precisely what made it possible in the first place to train models on enormous amounts of text. The boom since 2020 thus has less to do with a new idea about thinking than with a design that fits well with modern hardware. The fact that Nvidia was able to rise to become the world’s most valuable company is directly tied to this.

A second point is versatility. The same blueprint also works for images, music, audio recordings, and even for the structure of protein molecules. You just have to break the data down into a sequence of numbers. In the past, each task required its own specialized construction; today, one usually reaches for a Transformer.

Attention: how the model relates words to one another

First, the text is cut into small pieces called tokens. A token is often a word, sometimes just a syllable or a punctuation mark. Each token is translated into a long list of numbers that roughly describes its meaning. Similar words receive similar lists of numbers.

Then comes the actual core trick, self-attention. For each token, the model calculates how strongly it should attend to every other token in the text. In the sentence “The lawyer gave the client his file because he needed it,” the model has to work out who “he” refers to. To do this, it compares the lists of numbers and weights the matching words more heavily. You can picture this like a student who, in a long text, highlights the relevant passages with a marker instead of reading everything with equal attention.

This step is repeated many times in succession, in so-called layers. Large models have eighty or more such layers. With each layer, the representation of a token becomes more precise and takes in more context. In the end, the model predicts only one thing: which token is most likely to come next. A common misconception is that the Transformer plans sentences. It generates them token by token, with each new token taking the entire sequence so far into account.

From chatbots to weather forecasts

The most obvious place to encounter Transformers is in chatbots like ChatGPT, Claude, or Gemini. Translation in Google Translate, automatic subtitles on YouTube, and text suggestions in search bars are also based on it. Image generators use Transformers to understand the input prompt text.

In business news, the term often appears indirectly. When people talk about a “context window,” they mean how many tokens a Transformer can consider at once. When data centers cost billions, it’s because the attention computation rises sharply with text length: text twice as long roughly means four times the computational effort. Researchers are working on modified designs to address precisely this weakness.

It’s important to distinguish related terms. The Transformer is the architecture, i.e., the blueprint. A language model like GPT-5 is a concrete, fully trained building constructed according to this plan. And a product like ChatGPT is the interface through which one talks to this building.