Megakernel
A megakernel is a single large graphics-card program that processes an entire chain of computations in one pass instead of splitting it into many small individual programs. This eliminates the waiting times between steps, which saves anywhere from a few percent up to a third of the time, especially for the response speed of AI models.
Graphics cards compute AI models because they can perform very many simple calculations simultaneously. For a graphics card to do something, the regular computer sends it a small program that each handles a single task. Such a program is called a kernel, and an AI response usually consists of hundreds or thousands of such calls in succession. A megakernel merges these many small programs into a single large one. The graphics card is thus started once and then works through the entire chain on its own. The name is meant literally: a kernel that contains an unusually large amount at once.
Why the pauses between computation steps are costly
Every kernel launch costs time. The graphics card must accept the task, occupy its compute units, calculate, write out the result, and then wait until the next job arrives. Considered individually, these are just microseconds. But with thousands of calls per response, this adds up to a measurable share of the total time.
This is especially painful when a language model responds word by word. For each individual word, the entire chain runs through once. The computational work per word is small, but the number of launch operations remains the same. It is precisely in this situation that the graphics card can spend a considerable portion of the time idle, even though its compute units would be free.
For companies operating AI services, this is directly money. A graphics card costs the same per hour whether it is computing or waiting. Published measurements on megakernel conversions report, depending on model and hardware, speedups ranging from a few percent up to well over twenty percent. The same response arrives faster, and the same hardware can serve more users.
How an entire computation chain fits into one program
A megakernel is launched once on the graphics card and then keeps running. Within this program there is no rigid sequence, but rather a kind of work list. The compute cores fetch tasks from this list, complete them, and reach for the next one. One can imagine this like a workshop where orders hang on a board, instead of a boss assigning every single step individually.
The difficult part is the ordering. Some tasks need the result of an earlier task. The megakernel must therefore ensure on its own that no one proceeds with calculations too early. For this purpose there are counters and signals in memory that a compute core can check to see whether its preliminary work is finished. Previously, this control was handled by the main processor between kernel calls.
One side effect concerns memory usage. Because everything stays within a single program, intermediate results can remain in the fast memory areas directly on the chip. Otherwise they would have to be written to the slower main memory of the graphics card and immediately read back. The price for this is complexity: a megakernel is difficult to program, difficult to test, and usually tailored to a specific model and a specific type of graphics card.
Megakernels in chatbots and in tech news
A megakernel is never directly visible. One only notices it by the fact that a chatbot outputs the first sentence faster or the text appears more fluid. The technology resides in the software that sits between the AI model and the hardware. Such programs are called inference engines, because inference refers to the phase in which a finished model is used.
The term appeared more frequently in trade reports starting in 2025. Research groups and providers of acceleration software introduced megakernel variants for common language models, some as open source code. Anyone reading such announcements should pay attention to the conditions: the stated gains often apply to small numbers of requests and a single graphics card, not to every operational case.
Economically, the megakernel belongs in the same category as other optimizations of the operating phase. Such methods do not make a model smarter, but cheaper. This is relevant for investors because it affects how many expensive data-center chips a provider has to buy for the same number of users.