Benchmark

A benchmark is a fixed collection of test tasks with known correct solutions. It is presented to various computer programs, and the number each one solves correctly is counted. Because everyone receives exactly the same tasks, the results can be directly compared. The principle resembles a centralized final exam: it is not the teacher who decides on the tasks, but an independent body. The result is usually a single number, such as “79 percent solved.” The term originates from technology in general, but today it is used especially often for systems that process human language.

Why benchmarks matter

Without benchmarks, there would be no way to verify manufacturers' claims. Every company would simply say its model is the best. A public test makes such claims verifiable. That’s why benchmark results function as a kind of currency in the industry.

The numbers also have financial impact. When a provider jumps to the top of a well-known test, customers and investors take notice. Companies decide which system to purchase based on such values. Research groups also need them to show that a new idea actually improves something.

At the same time, benchmarks are a weak point. A single score says nothing about whether a system responds politely, consumes little power, or is dangerously wrong in rare cases. Anyone who only looks at the ranking easily overlooks such qualities.

How a benchmark works

A benchmark consists of three parts. First, the tasks, for example a thousand multiple-choice questions from biology, law, and mathematics. Second, the official model solutions. Third, a fixed rule for how scoring is done.

The process is always the same. The system being tested receives the tasks, its answers are compared with the solutions, and the score is derived from that. For tasks with clear answers, this happens automatically. For open-ended tasks, such as writing a text, humans or a second AI model assess the quality.

A well-known problem is called contamination. Language models learn from enormous amounts of text from the internet. If the test questions along with their solutions are somewhere on the web, the model may have already seen them during training. In that case, the test only measures memory instead of ability. A comparable situation would be a class test whose solutions had been passed around beforehand.

For this reason, experts regularly build new benchmarks. Some keep the tasks secret, others generate them continuously anew. Once a test is mastered by almost all systems, it is considered saturated and loses its value.

Where you encounter benchmarks

In news about new AI models, benchmark names almost always appear. MMLU tests school and university knowledge across many subjects. GSM8K contains word problems from mathematics. SWE-bench has models fix real bugs in program code. When a report says a model has “achieved state of the art,” there is usually such a test behind it.

The same principle applies when buying hardware. Phone tests, graphics card comparisons, and website loading times are measured using standardized test runs. There too, the goal is to compare different devices under equal conditions.

A critical eye is worthwhile. Useful questions include which test was used, how large the lead is, and who carried out the measurement. A one-percentage-point lead in a self-chosen test often means nothing in everyday practice.