Model Weights

An AI system such as a chat program consists, at its core, of a very long list of numbers. These numbers are called model weights. At the outset they are set randomly, and the program outputs only nonsense. During learning, they are slightly readjusted millions of times until the outputs match the examples that have been presented to the system. Afterward, everything the system can do is contained in exactly these numbers. Large systems have hundreds of billions of such numbers; one then also speaks of billions of parameters.

The weights are the actual asset

The architecture of an AI system is usually no secret. The construction of modern language models is described in research papers and can be written down in a few hundred lines of code. What is valuable is not the blueprint but the result of the training: the fully tuned numbers. Computing them often costs many millions of euros in electricity and compute time.

That is why the question of who gets access to the weights is highly contested in the industry. Some companies release them for download. Such models are called open-weight, often somewhat imprecisely also called open source. Anyone can then run them on their own machines, modify them, and continue training them. Other providers keep the weights strictly secret and let their model be used only over the internet.

For companies, this has practical consequences. Whoever owns the weights themselves is not dependent on any provider and can process sensitive data in-house. Whoever only has access via an interface over the network pays per request and must accept that the model can be changed or shut down at any time.

How random numbers become knowledge

You can picture a weight as a dial that determines how strongly a particular signal is passed on within the system. A high value amplifies, a negative value dampens. Millions of such dials together produce the model’s behavior. No single dial has a clearly nameable meaning in this.

The tuning always follows the same scheme. The model receives an example and makes a prediction, for instance the next word in a sentence. Then the error is measured, i.e. the distance to the correct answer. A procedure then calculates in which direction each individual weight must be shifted so that the error becomes smaller. Each change is tiny, but the process repeats billions of times.

After training, the weights are fixed. When you talk to a chat program afterward, it does not change through your conversation. This is a common misconception. The model only computes with the existing numbers and then forgets the conversation again. Subsequent adaptations to a specialized field are indeed possible, but they require a separate training run, known as fine-tuning.

Weights in the news and in products

In news reports one frequently reads phrases such as “Meta has released the weights of Llama” or “a French model with seven billion parameters.” These always refer to the same thing: the set of numbers that constitutes the model. The number of parameters serves as a rough measure of size, similar to the displacement figure of an engine.

The size also determines where a model can run. Each weight requires storage space, typically one to two bytes. A model with seven billion parameters thus takes up several gigabytes and fits on a good home computer or a newer smartphone. A model with several hundred billion parameters requires a data center with many specialized chips.

The theft of weights is also a real issue. Whoever copies the file possesses a complete copy of the model without paying a single cent for the training. Security agencies and companies therefore treat such files like trade secrets. Conversely, there are platforms like Hugging Face where freely available weights are downloaded hundreds of thousands of times.