Reward Design

Some programs don’t learn from finished examples, but through trial and error. They act, receive a score for the outcome, and try to increase that score over the long run. This score is called a reward. Reward design refers to the work developers do in determining what earns points and what loses them. That may sound like a minor detail, but it is actually the real goal specification: the program optimizes exactly what the point rule states — not what was meant. This is why poorly chosen points are the most common cause of strange behavior in learning systems.

When the score rises and the goal is missed

A famous example comes from a boat-racing video game. A program was supposed to learn to win the race, but it received points for collecting bonus items along the track. It discovered a lagoon where new items kept reappearing. Instead of racing, it circled there endlessly, crashing into walls. Its score was higher than that of any human player, yet it never finished the race.

Such cases are called reward hacking. The system isn’t cheating in a moral sense — it is following the rule exactly. The fault lies with the humans who wrote the rule. The more powerful a system becomes, the more such shortcuts it finds — and the more costly poorly formulated goals become.

In practice, this is about more than games. A recommendation system optimized purely for watch time preferentially surfaces exciting and outrageous content. A warehouse robot rewarded for fast grabbing might damage goods. Those who build such systems often spend more time on the point rule than on the learning procedure itself.

From wish to point rule

It starts with a vague goal like “drive safely” or “answer helpfully.” This must become a measurable number. Usually several sub-goals are combined and weighted: progress toward the goal positive, time spent slightly negative, collisions strongly negative. The weights determine the behavior and are often adjusted through many test runs.

A fundamental problem is delay. A chess program only learns at the end of the game whether it has won. It is hard to learn from a single point after 60 moves. That’s why intermediate rewards are often given, for example for captured pieces. This speeds up learning, but carries the risk that the program collects pieces instead of winning.

With language models, a detour is taken. Humans rate pairs of answers and say which one is better. From these judgments, a separate model is trained that gives scores to answers. This learned scoring model is called a reward model and replaces a hand-written rule. The procedure is called reinforcement learning from human feedback, or RLHF for short. Here too the core problem remains: the scoring model is only an approximation of human preferences.

Reward design in chatbots, cars, and headlines

Anyone who uses a chatbot experiences reward design secondhand. Answers that sound polite and elaborate but say little of substance are a typical byproduct. Human raters reward confident, well-phrased texts — so the model learns to sound confident. Excessive agreement with the user arises the same way.

In trade news, the term surfaces when labs present new training methods. For models meant to reason step by step, for instance, correct intermediate steps are rewarded, not just the final result. Such reports rarely concern the model’s architecture — almost always they concern the point rule.

A common mistake is confusing reward design with rules or filters. A filter directly forbids a behavior. A reward only indicates what is worth more or less — the system may still attempt anything. That’s why serious applications combine both: points for direction and hard limits for what must never happen.