Pipeline Parallelism

Modern AI programs are so large that they no longer fit into the working memory of a single compute chip. So they have to be split up. In pipeline parallelism, the program is cut crosswise: it consists of many consecutive computation stages, and each chip takes over a group of these stages. An input then travels from chip to chip, similar to how a car moves from station to station on an assembly line. Chip 1 computes the beginning, sends its intermediate result to chip 2, which continues computing, and at the end the finished result comes out. The name comes from exactly this assembly line, which in English is called a “pipeline”.

Why large models can’t run without splitting

An AI model consists of billions of learned numbers, the parameters. During computation, these numbers must lie completely in the fast memory of the graphics chips. A single data center chip typically has 80 to 192 gigabytes of this. A model with several hundred billion parameters needs a multiple of that, and even more during training, because additional auxiliary values must be stored. Without splitting, such a model would not be trainable at all.

Pipeline parallelism is the most obvious answer to this, because it follows the structure of the model. The computation stages are already arranged one after another anyway. You just have to cut them at sensible points. Between two chips, only a relatively small intermediate result is then sent per step, not a huge amount of data.

This is an economic argument, not just a technical one. Anyone who can distribute a model across 64 chips instead of 8 can push into a whole different size class in the first place. Almost all well-known large language models were trained this way across thousands of chips.

How the assembly line is kept running without idle time

The naive approach has an obvious problem. While chip 1 is computing on an input, chips 2, 3, and 4 have nothing to do. Afterward, chip 2 is busy, and chip 1 is waiting. This waiting time is called a bubble. With four chips, three quarters of the expensive compute power could remain unused this way.

The solution is called micro-batching. Each data batch is broken down into many small chunks, which are sent off in quick succession. As soon as chip 1 has handed off the first chunk, it starts on the second. After a brief ramp-up phase, all chips work simultaneously, each on a different chunk. This is exactly how a real assembly line works too: it’s only efficient once there’s a car at every station.

During training, there’s an additional wrinkle. After the forward pass, the correction must run backward through all stages. Methods with names like GPipe or 1F1B interleave forward and backward steps in such a way that the bubble stays small. A second pitfall is the splitting itself: if the segments are unequal in size, the slowest chip sets the pace for all the others.

Alongside tensor and data parallelism in the data center

In practice, pipeline parallelism is rarely used alone. In tensor parallelism, a single computation stage is split widthwise and processed jointly by multiple chips. In data parallelism, the entire model exists multiple times, and each copy gets different training data. Large training runs combine all three variants and then speak of 3D parallelism.

A common split: tensor parallelism within a server, because the chips there are connected via very fast cables. Pipeline parallelism across server boundaries, because less data is sent back and forth that way. Data parallelism then across the resulting groups.

You’ll encounter the term in announcements of new models, in technical reports from providers like Meta or DeepSeek, and in software like Megatron-LM or DeepSpeed. A common misconception is that pipeline parallelism makes computation faster. For a single request, that’s not true, since it travels through all the chips anyway. The gain lies in memory capacity and in throughput when handling many parallel requests.