FP8

Computers store numbers as a sequence of zeros and ones. Each of these positions is called a bit. The more bits a number is given, the more precisely it can be represented – and the more memory and computing time it costs. FP8 means: only eight bits are available for a number. The “FP” stands for floating point – a method that can represent both very small and very large numbers. For a long time, AI systems computed with 32 or 16 bits per number; FP8 cuts that in half again.

Eight bits instead of sixteen: what that saves in costs

A large AI model consists of many billions of stored numbers. These numbers are what the model absorbed during training. At 16 bits per number, a model with 70 billion numbers needs around 140 gigabytes of memory. In FP8, it’s only about 70 gigabytes. Suddenly the model fits on half as many graphics cards.

The second benefit is speed. Modern AI chips have compute units that support FP8 directly. Depending on the chip, they manage roughly twice as many computing steps per second in FP8 as in 16 bit. For a company, that means: the same answer costs less electricity and less waiting time. With millions of requests per day, that determines whether a product is economically viable.

But there is a price. With eight bits, only about 256 different values can be distinguished. Fine gradations are lost, so the model is constantly rounding. If one proceeds carelessly, the quality of the answers drops noticeably. The whole art lies in halving memory usage without a user noticing the difference.

Sign, exponent, mantissa – and the scaling factor

A floating-point number consists of three parts. One bit indicates the sign, i.e. plus or minus. A group of bits, the exponent, determines the order of magnitude – whether it’s about thousandths or millions. The rest, the mantissa, contains the precise digits. In FP8, the exponent and mantissa must share seven bits between them.

That’s why there are two common variants. E4M3 takes four bits for the exponent and three for the mantissa: more precise, but with a smaller value range. E5M2 takes five and two: coarser, but it covers very large and very small numbers. In practice, both are used in different places within the same model.

To keep anything from falling outside the range, scaling factors are used. All numbers in a group are multiplied by a common factor before computation, so they slide into the representable range. Afterwards, the result is scaled back. Sensitive parts of the model are also often deliberately kept in 16 bit. This mixing is called mixed precision and is standard practice today.

FP8 in chip announcements and model spec sheets

Anyone reading news about AI chips will almost inevitably come across FP8. Manufacturers like Nvidia or AMD state the performance of their accelerators in computing steps per second – separately for each number format. The largest, most attention-grabbing figures are usually listed next to FP8. That’s why, when comparing two chips, it’s important to always look at the same format.

The format is also mentioned for models themselves. When the Chinese lab DeepSeek unveiled its V3 model in 2024, one key claim was that training itself already ran in FP8. This was considered risky, but it lowered training costs considerably. Previously, most labs used FP8 only when running the finished model.

A common misconception: FP8 is the same as quantization to 8 bit, as known from INT8. INT8 stores only whole numbers at fixed intervals. FP8, on the other hand, retains the floating-point logic and copes better with strongly varying orders of magnitude. For AI models in which individual values deviate widely from the norm, this is a real advantage.