Leaderboard

A leaderboard is a public ranking list. At the top is the program with the best result, below it all the others. What’s being compared are computer programs that, for example, answer questions, write texts, or generate images. For the comparison to be fair, all of them must solve the same tasks: for instance, about 500 math problems or 1000 programming tasks with known solutions. Whoever solves more tasks correctly climbs the list. The principle is the same as with a league table in soccer, except here software competes against software.

Why a spot on the list moves billions

For companies, a top spot is advertising. When a new program tops a well-known list, it’s in the news the next day. Investors read such reports as proof that a company is technically ahead. That’s why companies almost always announce their products with leaderboard placements.

Customers also orient themselves by this. A software company integrating an AI into its product has to choose among dozens of providers. Testing all of them individually is expensive. A glance at the ranking is more convenient, even if it says less.

That’s exactly the weak point. When a lot of money hinges on a single number, efforts get optimized toward that number. Experts call this overfitting: a program is tuned to the test tasks for so long that it shines there but disappoints with real user questions. So a good ranking doesn’t prove that a model is useful in everyday life.

From test task to placement

The basis of every leaderboard is a benchmark. That’s the name for a fixed collection of tasks with stored model solutions. Every program gets the same tasks, and the answers are automatically compared to the solutions. In the end there’s a percentage, for example 87 percent correct. This number determines the rank.

This doesn’t work for creative tasks, because a good essay has no clear-cut model solution. There, people are asked to vote instead. Two programs answer the same question anonymously, and users choose the better answer. From millions of such duels, a ranking is calculated, similar to the Elo rating in chess. The platform LMArena, formerly Chatbot Arena, is known for this.

A well-known problem is called data contamination. AI programs learn from huge amounts of text from the internet. If the test tasks along with their solutions are posted somewhere online, they end up in the training material as well. The program has then effectively seen the answers beforehand. That’s why some operators keep their tasks secret and only publish the results.

Well-known rankings and their limits

Anyone reading AI news constantly comes across such lists. Frequently mentioned are MMLU with knowledge questions from many school subjects, SWE-bench with real programming bugs from open-source software projects, and the user voting on LMArena. The platform Hugging Face, a kind of library for freely available AI models, runs its own public leaderboards.

You know this principle from outside AI too. Mobile games show high-score lists to motivate you to keep playing. Language-learning apps rank users by points collected. And graphics cards or processors have been compared in rankings for decades. The effect is the same everywhere: visible competition changes the behavior of participants.

When reading such reports, three questions are worth asking. What tasks were actually tested? How big is the gap to second place – a single percentage point could be chance? And did the provider measure this itself, or was it an independent body? Rankings are a useful reference point, but not a verdict on a model’s quality.