Routing Policy

Anyone running an AI service today usually doesn’t have just one program running in the background, but several. Some are small, fast, and cheap. Others are large, slow, and burn through a lot of money for every answer. Now a question comes in from a user: who should answer it? The routing policy is precisely the rule that makes this decision. You can think of it like the front desk in a large medical practice: someone there directs patients onward depending on what they need.

How much money a good distribution rule saves

The most important reason is money. The computing costs of large language models differ by factors, not by percentages. A small model can be ten to thirty times cheaper per request than the largest one available. But a large share of requests is mundane: translations, summaries, greetings, repetitions. If a provider sends these requests to the most expensive model, it burns money without getting a better result.

The second reason is speed. Users abandon a request if they have to wait too long for an answer. Small models often respond in under a second, while very large ones take considerably longer. A routing policy can therefore also decide based on latency, not just cost.

The third reason is often underestimated: control. A routing policy can encode requirements that have nothing to do with technology. For example, that requests containing health data are only processed on servers within Europe. Or that a particular customer always gets the most powerful model because they’re paying for it.

How the router recognizes a hard question

Ahead of the actual model sits a small program, often called a router. It reads the request and estimates how difficult it is. This estimate can be based on simple rules: length of the text, keywords contained, attached files. But it can also come from a dedicated, very small model that has been trained specifically for this assessment. It’s important that the router itself costs almost no compute time, otherwise it eats up the savings.

The policy is then the collection of decisions that follow from this assessment. A typical pattern is a chain of conditions: simple request to the small model, complex one to the large model, coding tasks to a model specialized for that. Some systems work with a second attempt. The small model answers first, and only if its answer is deemed uncertain does the question go to the large model again.

A common misconception is to confuse routing policy with the router inside a mixture-of-experts model. There, a network distributes the work across building blocks within a single model, invisible to the user and without rules that a human wrote down. A routing policy, by contrast, is deliberately configured and decides between entire models or data centers.

Routing in the chat window and in provider announcements

Routing is most noticeable in chat programs that no longer offer a model to choose from. Instead of a list of names, there’s just an input field. Behind the scenes, the routing policy decides whether your question is handled quickly or thought through in depth. If the same question is answered with different quality on two different days, this may be due to a changed policy.

In companies, the term shows up in tools that connect to multiple providers at once. Such intermediary layers are called gateways or proxies. There, developers write the policy as a configuration file, test it, and then measure cost per thousand requests. Outages are also caught this way: if one provider goes down, the policy automatically redirects to a second one.

In news about AI companies, it’s worth paying attention to phrases like “dynamic model selection” or “automatic routing.” They’re often a sign that a provider wants to reduce its operating costs. This matters for investors because computing costs make up a large share of expenses for AI services.