Queue

A queue is a waiting line for tasks in a computer system. You know the principle from the checkout at the supermarket: whoever comes first gets served first. Instead of customers, it’s tasks that are lined up here — such as “calculate this answer” or “send this email”. One program adds tasks at the back, another takes them out at the front and processes them. The English term has become the standard usage; it simply means a waiting line. Experts also speak of the FIFO principle, short for “first in, first out”.

What happens when everyone arrives at once

Computers rarely receive requests evenly spread out. During a football match, hundreds of thousands of people click on the same livestream in the same minute. Without a queue, the system would have to serve every request immediately or reject it. Both are bad: immediate service overloads the machines, rejection annoys the users.

The queue decouples the arrival of tasks from the completion of tasks. Requests are allowed to come in faster than they are processed. The queue then simply gets longer. As soon as the rush subsides, the system works through the backlog. A total outage thus becomes a mere delay — and that is a huge difference.

This is especially noticeable with AI services. The computing chips that power a language model are expensive and scarce. No provider can keep enough chips on hand to handle the largest conceivable rush without any wait. So requests are placed in a queue. If a chatbot responds slowly to you, your request is often simply sitting in a queue.

Line up, pick up, confirm

Technically, a queue consists of three roles. The producer generates tasks and places them in the queue — for example, the website where you click “Send”. The queue itself stores the tasks in the order they arrived. The consumer picks up a task, completes it, and reports back that it’s done.

This confirmation matters more than it sounds. If a consumer crashes in the middle of the work, the confirmation never arrives. The task then counts as unfinished and moves back into the queue. Another consumer picks it up. This way, even a server failure doesn’t cause any order to get lost.

For more speed, multiple consumers are set to work on the same queue in parallel — like additional checkout counters at the supermarket. For very long queues, operators automatically add more machines. Some systems also support priorities: a paying customer may cut in line, while an unimportant background job waits longer. Strictly speaking, that’s no longer a pure FIFO queue but a priority queue.

From ticket numbers to the GPU cloud

You encounter this principle constantly in everyday life, usually without noticing. In an online shop, “Order received” appears instantly, even though payment and inventory aren’t processed until seconds later. For ticket sales, some sites even show you your exact position in the queue. Even the printer at school runs a queue: all jobs land there and get printed one after another.

In companies, entire systems run on specialized queue software with names like Kafka or RabbitMQ. It connects programs that otherwise have nothing to do with each other. One service writes events into it, others read them at their own pace. If a service goes down for ten minutes, the work piles up instead of getting lost.

In news about AI, the term usually comes up in connection with computing capacity. Providers then report long queues for training runs because graphics chips are fully booked. A common misconception: a queue doesn’t make a system faster. It only prevents work from being lost and spreads out load spikes over time. If a queue keeps growing permanently, the computing power simply isn’t enough.