GPU

Every computer contains a main processor. It works through tasks very quickly one after another, but always only a few at a time. A GPU is a second, very differently built chip. It consists of thousands of small computing units, all working at the same moment. Each individual one is slow and can only handle simple tasks. Together, however, they manage enormous amounts of simple calculations. The abbreviation stands for Graphics Processing Unit, in other words a graphics processor.

Why GPUs became the bottleneck of the AI industry

At their core, AI systems mostly compute with lists and tables of numbers. Millions of such values are multiplied and added. These calculations hardly depend on one another, so they can be carried out simultaneously. That is exactly what a GPU is built for. On a main processor, training a large language model would take decades. On thousands of GPUs, it takes weeks.

This has turned into an economic bottleneck. The US manufacturer Nvidia dominates this market and has thereby become one of the most valuable companies in the world. Individual specialized cards for data centers cost between 20,000 and 40,000 dollars, depending on the model. Large AI companies buy tens of thousands of them. Whoever cannot get any cannot compete in the development of large models.

That is why GPUs are also politically important. The US bans the export of its fastest AI chips to China. Computing power is thus treated like a strategic raw material. News about delivery times, export rules, or new chip generations regularly moves the stock markets.

Many slow computing units instead of a few fast ones

The difference can be compared to two ways of working. A main processor is like a single, very experienced craftsman. He can also solve complicated tasks, but only one after another. A GPU is like a hall with a thousand helpers, all performing the same simple hand movement. For a thousand identical parts, the hall is unbeatable. For a single tricky repair, it is useless.

For this to work, the task must be divisible. With images, this is obvious: every pixel is calculated independently. With AI models, the same applies to the many multiplications in a table of numbers. Tasks in which each step waits for the result of the previous one, on the other hand, gain no advantage.

A second point is memory. A GPU has its own, very fast connected memory directly on the card. However, it is limited, often 80 to 192 gigabytes per chip. A large model therefore does not fit on a single GPU. It is distributed across many cards, which work together via fast connections. Often it is not the computing power that is the bottleneck, but the reloading of data into memory.

From the game console to the data center

The first GPU in most people’s lives is found in a gaming device. In consoles, gaming PCs, and also in every smartphone, a graphics processor calculates the images on the display. These chips are smaller and more energy-efficient than the variants used for data centers, but they work on the same principle. Video editing, 3D programs, and image processing also make use of them.

Anyone working with AI themselves usually rents GPUs rather than buying them. Cloud providers rent out computing time by the hour, often for just a few euros for smaller cards. When chatting with an AI assistant, a GPU is also running in the background in a data center. All you notice of it is the roughly one-second wait time.

Related terms also crop up in the news. TPUs from Google and other AI accelerators are even more tailored to AI computations and can do fewer different things. A common misconception is that a GPU is inherently fast. Its advantage only arises with tasks that can be broken down into many similar parts.