Mixture of Experts

Imagine an editorial team with a hundred staff members. For every reader question, all hundred research it simultaneously. That would be thorough, but absurdly expensive. It makes more sense to ask, per question, only the two people who know the topic. A Mixture-of-Experts model is built on exactly this principle: it contains many subnetworks, called experts, and uses only some of them for each request. The rest remain idle and cost no computing time at that moment.

Large models that still answer affordably

The capabilities of an AI model depend strongly on its size. Size is measured in parameters: these are the adjustable numbers in the model that receive their values during training. More parameters usually mean better answers. But they also mean more computational work per question, and that costs electricity and money.

Mixture of Experts breaks exactly this connection. Such a model can have hundreds of billions of parameters in total, but activates only a fraction of them per request, often less than a tenth. The operator thus gets the quality of a very large model at the computational cost of a much smaller one. For services with millions of users per day, this difference decides whether an offering can sustain itself at all.

That’s why MoE models regularly appear in business news. When a provider announces it is drastically lowering its prices per request, this construction principle is often behind it. The principle became especially well known through Chinese models such as DeepSeek, whose low training and operating costs moved chipmakers' stock prices in early 2025.

The router decides who computes

The heart of an MoE model is a small auxiliary network, the router. It looks at every piece of text that passes through the model and selects the appropriate experts for it. Typically two out of perhaps 64 or 128 experts are chosen. Their results are combined and passed on.

Importantly, no one defines the areas of expertise by hand. The router is trained together with the model, using the same example data. Which responsibility an expert ends up with emerges on its own. Often it doesn’t match human categories like “mathematics” or “French,” but rather finer patterns in language.

This construction has a catch that is easily overlooked. The entire model must reside in the memory of the graphics cards, including the experts that are currently idle. What is saved is computing time, not storage space. A second problem is distribution: if the router permanently favors certain experts, these become overloaded while the others remain unused and untrained. Developers therefore build in rules that reward even utilization.

Where MoE models are in use

Almost all major chatbots and coding assistants of recent years use this architecture, wholly or in part. Users notice nothing of this. You type in a question and get an answer; which experts were involved is stated nowhere. The effect becomes visible only indirectly, for instance in short response times and in prices that are often in the cents range per million words processed.

In model names and technical spec sheets, however, you encounter the principle quite directly. There you find figures like “8x7B” or “671B total, 37B active.” The first number states the total size, the second the part that actually computes per request. Anyone reading both numbers immediately recognizes an MoE model.

A common misconception is the idea that individual experts can be downloaded or swapped out separately like apps. That doesn’t work, because they only deliver meaningful results in interplay with the router and the rest of the model. Mixture of Experts is also not the same as several separate AI systems that are addressed alternately. It remains a single model that divides up its work internally.