GPU Kernel
A GPU kernel is a small program that a graphics card runs thousands of times simultaneously. Almost all AI computations consist of such kernels, and how well they are written determines speed and power costs.
A graphics card is a special component in a computer that performs a great many simple calculations simultaneously. For it to do this, it needs a suitable program. That is exactly what a GPU kernel is: a short piece of computational instruction that the graphics card executes not once, but thousands of times in parallel. Each execution processes a different small portion of the data. The term has nothing to do with the operating system kernel of Windows or Linux, even though the word sounds the same.
Why it matters
Modern AI systems compute almost exclusively on graphics cards. The actual work consists of enormous number tables that are multiplied and added together. Kernels do this work. Whether a model responds in two seconds or in ten therefore depends heavily on how well the kernels are written.
The difference is not a subtlety. A poorly optimized kernel can, in extreme cases, leave a graphics card idle 90 percent of the time. It then waits for data instead of computing. For companies operating tens of thousands of graphics cards, this translates directly into millions in power and hardware costs. That’s why people who can write good kernels are among the highest-paid specialists in the industry.
How it works
A normal processor works like a single, very fast employee who ticks off tasks one after another. A graphics card is more like a hall with ten thousand assistants who all receive the same simple instruction. The kernel is this instruction. It reads, for example: take number X from table A, multiply it by number X from table B, write the result to C. Each assistant gets a different X.
The bottleneck is rarely the computation itself. It is the path the data takes to reach the compute unit. Graphics cards have a large but slow main memory alongside tiny, very fast intermediate memory. Good kernels fetch data into the fast memory once and reuse it there multiple times.
A second important trick is called fusion. Instead of running three kernels one after another, each writing its intermediate results back to slow memory every time, they are combined into a single one. The most famous example of this is FlashAttention, a kernel that significantly sped up a central computational step of language models without changing the result.
Where you encounter the term
As an ordinary user, you don’t write kernels. But you notice them when a chatbot responds faster than before, even though the model is the same. Often, better kernel work is behind it.
In business news, the topic usually comes up in connection with Nvidia. Nvidia’s programming environment CUDA is the standard in which kernels are written. Over the years, a huge collection of ready-made, highly optimized kernels has emerged there. This very collection is Nvidia’s actual moat against the competition. Chips from AMD or Google may be similarly powerful on paper, but the matching software is often missing.
That’s why many projects are working to make kernels easier to write. Tools like Triton or compilers in PyTorch generate kernels automatically from simpler code. If you read in the news that a provider has halved its inference costs, kernel optimization is almost always part of the explanation.