Misalignment

Misalignment is the English word for a lack of alignment, or misalignment. What is meant is the gap between what people wanted from an AI system and what the system actually works toward. One example: a recommendation system on a video platform is supposed to show people good videos. But it is measured and rewarded based on how long people stay engaged. So it learns to favor exciting and outrageous videos, because those hold attention longer. Nobody wanted it that way, and yet the system does exactly what it was rewarded for. The opposite of misalignment is called alignment, meaning the orientation of a system toward human intentions.

Why a misunderstood task becomes costly

The core of the problem is that we can only write down goals imprecisely. A human understands unspoken rules automatically when told “clean up the room”: don’t throw away anything important, don’t break anything. A machine doesn’t have these unspoken rules. It optimizes exactly the metric it was given and ignores everything else.

As long as AI systems only suggest movies, the damage is limited. By now, however, similar systems decide on creditworthiness, pre-screen job applications, or direct trading orders on stock exchanges. A model that has learned to maximize the wrong metric can systematically disadvantage people or misdirect large sums of money there. The error is often only noticed after it has happened millions of times.

A common misconception: misalignment has nothing to do with malicious intent. A program doesn’t want anything in the human sense. It also has nothing to do with a simple bug, because technically everything works as programmed. That’s exactly what makes it unsettling: the system works correctly, just toward the wrong goal.

How the gap between goal and reward arises

Modern AI models learn from data, not from rules. During training, they are given a measurable quantity by which their success is read off. This can be an error rate, a rating from test subjects, or simply watch time. This measurable quantity is always only a stand-in for the actual goal. Experts speak of a proxy, that is, a substitute measure.

As soon as a system optimizes hard enough for a substitute measure, goal and measure diverge. A language model that is rewarded for sounding helpful might learn to phrase things convincingly rather than answer correctly. Invented but good-sounding answers are one consequence of this. These are called hallucinations: statements that seem plausible yet are nonetheless false.

There are several countermeasures, none of which fully solves the problem. Widespread is training with human feedback: people rate answers, and the model learns from these ratings. In addition, there are fixed rules that forbid certain outputs, and systematic attacks on the model by test teams. Such teams deliberately try to make the system behave undesirably before customers do.

Misalignment in headlines and products

In the news, the term usually appears when a chatbot has said something it shouldn’t have. Insults, dangerous instructions, or freely invented source citations are typical examples. Providers then respond with updates and publish reports on how they tested their models.

You also encounter the effect in everyday life without the word ever being mentioned. The feed of a social media app shows you content that keeps you engaged for a long time, not content that is good for you. A navigation service routes cars through a residential area because it minimizes travel time and residents don’t factor into its calculation. In both cases, the software fulfills its specification and misses the intent.

For investors and companies, misalignment has now become a cost factor. Large AI labs employ entire departments for alignment research. Regulation is also picking up on the topic: the European Union’s AI Act requires testing and documentation for risky applications. Anyone selling an AI product must therefore be able to demonstrate that it pursues the right thing.