Dense Transformer

A language model is a program that has learned to continue texts. Its knowledge is stored in billions of adjustable numbers, called parameters. These parameters are arranged in layers that process an input step by step. A Dense Transformer is such a model in its original form: every single request runs through all of the model’s parameters. Nothing is skipped, nothing is switched off. “Dense” describes exactly that - the model is fully engaged for every computation.

The standard against which everything is measured

Almost all well-known AI models from 2018 to 2022 were dense transformers. GPT-3 belongs to this group, as do Meta's early Llama models. This design is considered well-established and well understood. When researchers test a new idea, they usually compare their result to a dense model of the same size.

The advantage lies in predictability. A dense model requires roughly the same amount of compute time for every request, regardless of which question is asked. This makes planning data centers easier and costs easier to calculate. Training also proceeds more stably, because there are no internal selection decisions that could go wrong.

The disadvantage becomes apparent with growth. If you increase the parameter count tenfold, the computational effort per response roughly increases tenfold as well. This is exactly why many large providers now rely on Mixture of Experts. There, only a fraction of the building blocks is activated per request. A dense model with 400 billion parameters can barely be operated affordably anymore, whereas a Mixture-of-Experts model of that size can.

What happens inside the model with every request

The text is first broken down into small units, usually word fragments. Each unit is converted into a long sequence of numbers. These numbers then travel through dozens of identically structured layers. Each layer contains two components: an attention component, which establishes relationships between the words, and a computation component made of simple multiplications.

The crucial point is that both components are always run through completely. Whether the model is processing “Hello” or solving a physics problem makes no computational difference. You can picture it like a factory in which every workpiece has to pass through all the machines in the hall - including the ones it doesn’t actually need.

Because the amount of computation cannot be reduced through selection, it is reduced in other ways. One widespread method is quantization: the parameters are stored more coarsely, for example with 8 instead of 16 digits of precision. Another is distillation, in which a small model imitates the behavior of a large one. Neither changes the dense underlying structure, but both make it cheaper.

Where dense models are still being built today

In product announcements, the term usually appears as a point of distinction. When a company emphasizes that its new model is a “Mixture-of-Experts model,” it is thereby distinguishing itself from the dense design. Conversely, providers of small models deliberately advertise “dense,” because such models run more easily on a single graphics card.

Especially in the field of small open models, the dense design remains dominant. Models with 7 or 8 billion parameters are almost always dense. They fit on good gaming PCs and can be fine-tuned for one’s own purposes with little effort. Anyone running a language model on a laptop or phone is therefore very likely using a dense transformer.

A common misconception is that dense models are simply outdated. At the same parameter count, a dense model is often even somewhat more capable, because it uses all of its capacity. The advantage of the alternative lies not in better quality, but in lower costs at extreme scale. For small and medium-sized models, the dense design therefore remains a deliberate, sensible choice.