API Gateway

Large online services do not consist of a single program. Behind an app like an online shop there are often dozens of small programs: one for login, one for search, one for payment. So that these programs can exchange data, they have defined access points through which other software may make requests. Such an access point is called an interface, or API in English. An API gateway is the one door through which all these requests from outside must first pass. It accepts every request, checks it, and forwards it to the correct program in the background.

One door instead of fifty back entrances

Without such a central point, every single background program would have to check for itself who is making the request and whether that person is authorized to do so. The same check would thus be programmed dozens of times, in dozens of variants. As soon as a rule changes, a team would have to touch every single program. This is exactly where security updates often fail in practice: one gets forgotten.

The gateway pulls these tasks together into one place. Login, permissions, logging, and counting of requests happen there once for everyone. This makes the background programs simpler, because they only have to take care of their actual task. For operators, this is also a matter of money: anyone who sells access to their services must reliably count who uses how much.

With AI services, there is an additional reason. A single request to a large language model costs real computing time on expensive graphics cards. Operators must therefore very precisely control who may send how many requests per minute. This brake usually sits in the gateway.

What happens to a request in the gateway

A request arrives as a message from the internet. The gateway first reads the address it is directed to. Based on a rule list, it decides which background program is responsible. This forwarding is called routing.

But several checks take place beforehand. The gateway checks the credential sent along with the request, usually a long character key that a user received upon logging in. It checks whether this user has already used up their request quota. If something does not match, the gateway itself responds with an error message, and the background program never notices anything about the request.

In addition, a gateway can briefly cache responses. If a hundred users ask the same thing, the answer only needs to be calculated once. And it can distribute requests evenly across several identical copies of a program so that none of them becomes overloaded. This is why an API gateway is easily confused with a pure load balancer. The difference: a load balancer barely looks into the request, whereas a gateway understands its content and its authorization.

Where API gateways show up in everyday life

Every time a mobile app reloads data, the request very likely passes through a gateway. This is rarely visible, but it is noticeable. Messages like “Too many requests, please try again later” typically do not come from the actual service, but from its gateway.

In the tech industry, the term is also a product name. Amazon, Google, and Microsoft sell ready-made gateways as a rental service in their data centers. Alongside these, there are free programs like Kong or Nginx that companies operate themselves. This is why the word constantly appears in job postings and conference talks.

The topic is currently discussed a great deal in connection with AI. Companies that use multiple language models from different providers place a gateway in front of them. It then decides which request goes to which model, tracks the costs, and filters out sensitive inputs. Such setups go under the buzzword “AI Gateway” and are a direct descendant of classic API gateways.