Checkpoint

An AI model learns by working through millions of examples over the course of weeks. Its entire knowledge is contained in a huge collection of numbers, known as parameters. These numbers change a tiny bit constantly during learning. A checkpoint is a copy of all these numbers, captured at a specific point in time and written to disk. You can think of it like a save point in a video game: you save your current progress so you can pick up exactly where you left off later. Without such save points, any mishap during learning would mean total loss.

Why weeks-long training would be impossible without save points

Large models are trained on thousands of graphics cards simultaneously. With this much hardware, something is practically always failing. A card overheats, a network cable comes loose, the power supply fluctuates. Without intermediate states, you would have to start from zero after every such incident. For a run lasting two months, that would simply be unworkable.

Compute time is also very expensive. A large training run can consume several million euros in electricity and data center rental costs. A lost day therefore costs real money. Checkpoints limit the damage to the time since the last save, usually just a few hours.

There is a second reason as well. A model does not always just keep improving while learning. Sometimes the learning process derails, and results suddenly get worse. In that case, developers fall back on an older checkpoint and restart from there with adjusted settings. The save point is therefore not just a safety net, but also a tool for experimentation.

What such a file actually contains

A checkpoint first contains the parameters, meaning all the learned numerical values of the model. For a large language model, this quickly adds up to hundreds of billions of values. Every value requires storage space, which is why such files can grow to several hundred gigabytes in size. For comparison: that’s more than fits on many laptop hard drives.

For training to be resumable, however, that is not enough. Also saved is the state of the optimizer, meaning the procedure that decides how strongly each number is adjusted at every step. In addition, there is the number of the current training step and the position within the dataset. Only with all this information does training continue seamlessly, as if nothing had happened.

The writing process itself takes time and blocks the computational work. That’s why saving doesn’t happen constantly, but at fixed intervals, for instance every few hours. A distinction is often made between full checkpoints for continuing computation and leaner versions that contain only the parameters. The latter are sufficient if you only want to use the model rather than continue training it.

Checkpoints in downloads, releases, and technical reports

Anyone who downloads a freely available model from the internet is, in truth, downloading a checkpoint. On platforms like Hugging Face, multiple states of a model are often available. They differ in how long training lasted or which additional task the model was further tailored to. With image generators, users regularly exchange different checkpoints with each other to achieve different image styles.

The term also appears in news reports. When a company writes that it is releasing an intermediate state of its model, that is exactly what it means: a snapshot, not the final product. Some research groups deliberately release many intermediate states so that others can trace when the model acquired which capability.

A common misconception is that a checkpoint is the same thing as the finished model. More precisely, a model is the architecture, meaning the arrangement of computational building blocks. The checkpoint supplies the numbers that make this architecture usable in the first place. Without a matching architecture, the file is just a meaningless collection of numbers. Only both together produce a functioning AI.