Preference Optimization

A chat program like ChatGPT first learns from vast amounts of text how language is structured. After that, it can continue sentences, but it doesn’t yet know what kind of answer people actually want. That is exactly the task of preference optimization. In this process, the program is repeatedly presented with two finished answers to the same question. A human has previously marked which of the two answers is better. The program is then adjusted so that it produces such preferred answers more often in the future.

From text continuer to useful assistant

A model that only continues text is barely usable as an assistant. To the question “How do I explain photosynthesis?” it might simply list ten more similar questions. That is linguistically correct, but useless. Preference optimization teaches the model that an explanation is expected, not a list of questions.

The second reason is safety. Almost everything found in instructions for weapons or insults appeared somewhere in the training text. A raw model repeats it readily. Through preference data, it learns to avoid such answers and instead refuse or ask follow-up questions. When a company says its model is “aligned,” that is, aligned with human expectations, preference optimization is almost always behind it.

An important distinction: preference optimization does not make a model smarter. Knowledge and language sense arise in the first, large training step. This later step only shapes how the existing capability is presented to the outside world. Some experts compare it to an etiquette course for a person who already knows a lot.

Two answers, one thumbs up

The process begins with collecting data. The model is asked a question and made to write two different answers. Paid reviewers decide which answer is better. From many such comparisons, a dataset of millions of pairs emerges.

The classic approach is called RLHF, short for Reinforcement Learning from Human Feedback. In this process, a second model is first trained to give scores to answers. This scoring model then replaces the humans and automatically evaluates millions of new answers. The main model is then shifted step by step toward higher scores.

For a few years now, a simpler method has become widespread: Direct Preference Optimization, abbreviated DPO. Here, the scoring model is skipped entirely, and calculations are made directly with the answer pairs. The result is similarly good, but the effort involved is significantly lower. A well-known failure mode in all variants is so-called reward hacking: the model finds tricks that earn points without actually being helpful. Typical examples are excessively long answers or constantly praising the user.

The thumbs-up beneath the chatbot’s answer

The most direct encounter with this method happens in chat programs themselves. Below many answers there is a thumbs-up and a thumbs-down. Some services also show two suggested answers and ask which one is better. These clicks are exactly the preference data from which later versions are trained.

In business news, the term comes up in disputes about costs and working conditions. Large providers employ thousands of reviewers, often through service providers in countries with low wages. Who the reviewers are shapes which answers a model considers appropriate. Critics call this a hidden political decision.

Preference optimization also plays a role in product differences. Two models can know a similar amount and yet feel completely different. One answers briefly and factually, the other in a friendly and detailed manner. This difference usually does not arise in the basic training, but in this final fine-tuning.