BF16

A computer stores every number as a sequence of zeros and ones. The more digits it uses for this, the more precise the number is — and the more space it takes up. For a long time, 32 digits were the norm for decimal numbers. BF16 uses only 16 digits and is thus half the size. The name stands for “Brain Float 16”; the format was developed at Google for computing chips that train AI systems.

Why this matters

Large AI models consist of billions of individual numbers, so-called parameters. These numbers are what a model learns during training. If each of them needs half as much space, a model twice as large fits into the same graphics card memory.

The computation also becomes faster. Modern AI chips process 16-digit numbers several times faster than 32-digit ones. In training runs that take weeks and cost millions, this is a tangible economic difference. Practically every large language model of recent years has been trained in BF16 or a similar format.

How it works

A decimal number in a computer consists of two parts. One part indicates the order of magnitude, i.e. whether it concerns thousands or millionths. The other part indicates the exact digits. You can think of it like the notation 3.14 times 10 to the power of 8: the 10 to the power of 8 is the order of magnitude, the 3.14 are the digits.

When shortening from 32 to 16 digits, one has to cut back on one of the two parts. BF16 cuts back entirely on the digits and keeps the full order of magnitude. It can therefore represent just as tiny and just as huge numbers as the 32-digit format, only more coarsely. Instead of 3.14159, it stores something like 3.14.

This exact division is the trick. During the training of a model, very small numbers occur, for example when a parameter is only minimally adjusted. A format with a narrow order of magnitude would simply round such values down to zero, and the information would be lost. BF16 can handle them. That the digits are imprecise is barely noticeable during training, since huge amounts of data are averaged over anyway. Small rounding errors largely cancel each other out.

The older competitor is called FP16 and divides the 16 digits differently: more digits, less order of magnitude. This allows for more precise calculation, but values can more easily “overflow” or drop to zero. Training runs then crash. This is why BF16 has prevailed in AI, even though FP16 is more precise on paper.

Where you encounter the term

Model cards and technical announcements often state in which format a model is available. Anyone who downloads an open model themselves will find files labeled bf16 and can use this to estimate how much graphics memory they need: roughly two gigabytes per billion parameters.

BF16 also appears in product announcements from chip manufacturers. Nvidia, AMD, and Google state the performance of their accelerators separately for different number formats. Investors and analysts compare these figures because they show how quickly a chip handles AI tasks. It’s important to note: such figures are only comparable if the same format is meant.

When simply using finished models, many providers go even further down, to 8 or even 4 digits. This subsequent coarsening is called quantization. BF16 is usually considered the starting point and the benchmark for how much quality is lost through this savings.