CUDA

Every computer contains a main processor that works through instructions one after another, very quickly. Graphics cards are built differently: they contain thousands of small computing units that work simultaneously. Originally, this only served to calculate images for games. CUDA is a software collection from the company Nvidia, dating from 2007, that opens up this computing power for arbitrary tasks. With it, programmers can describe in a normal programming language what the graphics card should do. The name is an abbreviation for “Compute Unified Device Architecture,” but practically no one uses this full form.

CUDA as Nvidia’s moat

At their core, modern AI systems consist of huge numbers of multiplications and additions. These calculations do not depend on one another, so they can all be carried out in parallel. Graphics cards are built precisely for this. Without them, training a large language model would take centuries instead of weeks.

Nvidia sells the matching hardware, but the actual advantage lies in the software. CUDA has existed for over 15 years, and during this time researchers and companies have built upon it. Well-known AI tools like PyTorch or TensorFlow are optimized for CUDA. Anyone developing an AI therefore almost automatically reaches for an Nvidia card.

In the business press, something like this is called a moat: an advantage that competitors can only overcome with difficulty. Competing companies such as AMD do build powerful chips. But their software is considered less mature, and existing programs would need to be adapted at considerable effort. CUDA runs exclusively on Nvidia hardware, and this is intentional.

What happens when programming with CUDA

A CUDA program has two parts. One runs on the main processor and controls the flow. The other is a short function that runs on the graphics card, where it is executed simultaneously in thousands of copies. Each copy processes a different piece of data, for example a different number in a large table.

You can picture this like grading exam papers. A single teacher grades all the notebooks one after another. With CUDA, a thousand helpers each get a single task from a notebook and work in parallel. This only works if the tasks are independent and all follow the same rule.

A common misconception is that CUDA makes every program faster. The opposite can happen. Data must first be copied into the graphics card’s memory, and that costs time. Tasks with many branches or little parallelism run better on the main processor. CUDA pays off when there are very many similar calculations to be done.

CUDA in stock market news and on your own computer

In business news, the term usually comes up in connection with Nvidia’s market value. Analysts rarely justify the high valuation with the chips alone, but regularly cite CUDA as well. It also plays a role in the US export restrictions on AI chips to China, since the hardware is worth little without this software.

Anyone experimenting with AI themselves will quickly run into it. When installing image generators or language models on their own PC, the instructions call for a matching CUDA version. If it’s missing, the program keeps computing on the main processor, just ten to a hundred times slower.

Technical data sheets also mention CUDA cores. These refer to the individual computing units of an Nvidia graphics card, often several thousand per card. The number is a rough indicator of performance, but not a measure that allows meaningful comparison of cards from different generations.