Reinforcement Learning

Reinforcement learning is a method by which a computer program learns through trial and error. No one tells the program what the correct answer is. It acts, receives feedback in the form of a score, and adjusts its behavior accordingly. Actions that brought many points it repeats more often. Actions with few points it avoids. The comparison with a dog that gets treats for tricks captures it fairly well. In German, it is called “verstärkendes Lernen” (reinforcing learning).

Why it matters

Most AI systems learn from ready-made examples: here a photo, there the answer “cat.” But for many tasks there is no collection of examples with correct solutions. In chess, no one knows for certain which move is the best. One only knows at the end whether one has won. Reinforcement learning is made precisely for such cases.

A second reason is even more important for the current discussion. Language models like ChatGPT initially only learn to continue texts. They only become polite, helpful, and cautious in a second step. Humans rate different answers, and the model learns from these ratings. This procedure is called RLHF, that is, reinforcement learning from human feedback. The newer models, too, which carry out longer computational steps before answering, were trained this way.

How it works

There are always two sides: the learning program, the agent, and its environment. The environment can be a game board, a simulation, or a real machine. The agent sees the current state, chooses an action, and receives a reward for it. This reward is simply a number. Then the process begins anew with the new state.

The agent’s goal is not the highest reward in the next step. It is meant to maximize the sum of all rewards until the end. That is why it can make sense to forgo points in the short term. A chess program sacrifices a piece if it will win later as a result. This foresight is the difficult part of the procedure.

The agent must also weigh two things. It can use the strategy that has worked best so far. Or it tries something new that might be better. Only trying new things leads nowhere, and only repeating does too. Another problem lies in the reward itself: if it is chosen poorly, the agent finds shortcuts. A program in a boat race drove in circles and collected points instead of finishing the race. Experts call this reward hacking.

Where you encounter the term

The method became well known through games. Google DeepMind’s AlphaGo defeated the world’s best Go player in 2016. Its successor, AlphaZero, learned chess solely by playing against itself and beat every existing program. Similar successes occurred with computer games such as Dota 2 and StarCraft.

In everyday life, reinforcement learning is found in robotics, for instance when a robotic arm learns to grasp. Data centers use it to control their cooling and thereby save electricity. It also plays a role in autonomous driving. In financial and tech news, mainly the abbreviations RL and RLHF appear. When a company says its model has learned reasoning abilities, reinforcement learning is almost always behind it. Incidentally, the term also explains why training is so expensive: the agent must go through millions of attempts before it becomes good.