Post-Training

Large language programs like ChatGPT learn in two stages. In the first stage, they read enormous amounts of text from the internet and from books. Afterwards, they can astonishingly well predict which word comes next in a sentence. What they still cannot do, however: follow an instruction, answer a question cleanly, or decline sensitive requests. That is exactly what gets made up for in the second stage, and this stage is called post-training. Here one no longer works with arbitrary text, but with carefully selected examples and with feedback from humans.

What makes the difference between a text machine and an assistant

A model after the first stage is like an extremely well-read person who has never learned to hold a conversation. If you ask it something, it might complete the sentence with further similar questions. That is statistically correct, but worthless to the user. Post-training teaches the model that a question demands an answer.

Almost everything users notice about a chatbot is tied to this. The tone, the length of the answers, the willingness to ask follow-up questions, the refusal of instructions for weapons or fraud: all of this arises here, not in the first learning step. Two companies can start from the same base model and end up with very different products.

For companies, this is also a matter of cost. The first stage consumes millions and takes weeks on thousands of specialized chips. Post-training is comparatively cheap and can be repeated within days. That is why new model versions often appear at intervals of months, even though the expensive base training behind them has remained the same.

The building blocks: sample answers, rewards, and practice

The first building block is called supervised fine-tuning. Humans write thousands of example conversations: a request and the desired good response. The model is trained to replicate such responses. In doing so, it learns the format of a dialogue, not new world knowledge.

The second building block is learning from human evaluation. The model generates several answers to a question, and test subjects rank them by quality. From these judgments, a second, smaller program emerges that assigns points to answers. The main model then practices until it collects as many points as possible. Experts call this procedure RLHF, short for Reinforcement Learning from Human Feedback.

In the meantime, a third building block has been added: practice on tasks with verifiable solutions. In mathematics or programming code, a computer can check for itself whether the result is correct. The model is allowed to think for a long time, and correct solution paths are reinforced. This is how the so-called reasoning models came about, which think through tasks step by step. It is important to draw the distinction: post-training changes behavior and draws out capabilities, but it hardly adds any new factual knowledge.

Post-training in products and headlines

Anyone reading a model card or a technical report from OpenAI, Google, or Meta will almost always find a dedicated section on post-training there. News about open models plays on this too: A freely available base model is transformed by a university or a start-up with its own post-training into a specialist for medicine or law. This is a major reason why there are so many model variants.

In everyday use, post-training is noticed precisely when it fails. A model that considers every second request dangerous is regarded as overfiltered. A model that agrees with everything and tells the user what they want to hear is regarded as too submissive. Both are criticisms of this training phase, not of the model’s knowledge base.

A common misconception: post-training is the same as prompting, i.e., skillfully phrasing a request. Prompting only works for this one conversation. Post-training changes the model permanently, before it even reaches users.