Uptraining

A program that writes text or recognizes images must first learn from huge amounts of data. This learning phase is called training and often costs millions of euros in electricity and computing time. Uptraining is the attempt to avoid repeating this work. You take a model that has already finished learning and train it further. In doing so, it receives new data, new tasks, or even a modified internal structure. The basic idea: knowledge already stored in the model should not be thrown away.

What companies save by doing this

Training a large language model from scratch takes weeks on thousands of specialized chips. Uptraining often needs only a small fraction of that. For companies, this is the difference between a project for large corporations and a project for a mid-sized team. That is exactly why new models keep appearing constantly, even though there are only a handful of truly expensive base trainings worldwide.

A second reason is timeliness. A model only knows the world up to the day its data was collected. Anyone who uptrains it with more recent texts brings it up to a more current state. This is cheaper than building a completely new model every year.

There is a price to pay, however. During further training, a model can lose old abilities because the new data overwrites the old patterns. Experts call this catastrophic forgetting. That is why old and new data are usually mixed together, so the model retains its basic knowledge.

From finished model to restructured model

Technically, training changes what are known as the parameters of a model, i.e., millions to billions of numbers that store its knowledge. In uptraining, you don’t start with random numbers but with the ones already learned. So you continue calculating from a good intermediate state. It’s like a mountain hike that begins at the cabin halfway up, instead of down in the valley.

Particularly interesting is the variant in which the construction itself is changed. A common case: a normal model is turned into a mixture-of-experts model. In this process, building blocks of the model are copied and restructured into several specialized departments, the experts. A small additional network then learns which expert is responsible for which request. Because the copied building blocks bring their knowledge with them, the new model is already usable after a short period of further training.

The distinction from fine-tuning is important. In fine-tuning, a model is adapted to a style or a task, such as customer dialogues, using relatively few examples. Uptraining works with significantly more data and aims to expand the model’s general capabilities or change its architecture. The boundary is fluid, but the effort involved differs by orders of magnitude.

Uptraining in model announcements and company news

When a provider presents a new model version, uptraining is very often behind it. Phrases like “building on,” “further trained on additional data,” or “derived from an existing model” are indicators of this. The many freely available models that are based on a single open base model also come about this way.

For investors and observers of the industry, this is a cost issue. Anyone who does a lot of uptraining needs fewer of their own data centers, but becomes dependent on the base models of others. That’s why the license of an open model is often more important than its leaderboard ranking: it determines whether you are allowed to build on it and sell the result.

A common misconception is to consider uptraining a cheap trick. It does not replace a good base model. If a capability is missing from the starting model, such as a language that barely appeared in the data, a little further training won’t fully instill it. Uptraining reinforces and shifts existing capabilities; it does not reinvent them.