Experiment Design

Experiment design refers to planning a test before carrying it out. You determine exactly what you want to know, what you’re changing, and what you’re measuring in the process. An example: A company wants to know whether a new recommendation service keeps users on the site longer. To find out, a randomly selected half of visitors see the new version, while the other half see the old one. In the end, both groups are compared. Without this prior planning, you do collect numbers, but you can’t say what’s causing a difference.

Why bad tests produce costly wrong decisions

Numbers seem objective, but they only are with clean planning. A common mistake is comparing two groups that already differed beforehand. If the new version happens to be shown only to users from one country, you end up measuring the country rather than the software. Such distorted results are known in technical jargon as confounding — a mixing up of causes.

In the AI industry, experiment design determines which claims can be trusted. Almost every announcement about a better model rests on a comparison test. If the comparison was made against a weak old version, the progress looks bigger than it actually is. If only a single run was done, the lead could be pure chance. Experts therefore look first at how the test was set up, and only afterward at the result.

There’s also a lot riding on this economically. Large platforms run thousands of such tests every year before rolling out a change to all users. A poorly planned test doesn’t just cost testing time. It can wave through a change that costs the company users or revenue for months on end.

Control group, chance, and the question of sample size

The core of almost every good design is comparison with a control group. This group receives the old version or no treatment at all. Only this way can you know what would have happened without the change. Assignment to groups happens randomly, usually via computer. Randomness ensures that differences between participants are distributed evenly across both sides.

Group size is equally important. With ten users per group, a difference says almost nothing, because fluctuations are normal. Statisticians therefore calculate in advance how many participants are needed to reliably detect an effect of a certain size. This calculation is called a power analysis. The smaller the expected effect, the more participants are needed.

A third building block is fixing the measurement metric before the start. Anyone who only decides afterward what to look at will almost always find some positive value. This mistake is so widespread that it has its own name: p-hacking. Pre-registration helps guard against this, where the plan is documented in writing before any data is collected.

A/B tests, drug trials, and model leaderboards

You most commonly encounter experiment design as an A/B test online. Streaming services, online shops, and social networks constantly test new buttons, colors, and recommendations this way. In all likelihood, you’ve often been part of such a test group without noticing. Some providers expand this into multivariate tests, where several changes are examined at once.

In medicine, the same principle is called a randomized controlled trial and is considered the gold standard for approvals. There, blinding is often added on top: neither patient nor doctor knows who is receiving the real treatment. This prevents expectations from coloring the result.

In AI news, this topic usually comes up when comparing models. Leaderboards like Chatbot Arena have users rate two anonymous answers and derive a ranking from that. A common misconception is that a higher score automatically means the better model. What matters is whether the test was set up fairly and whether the gap is large enough not to be due to chance.