Learn · GPU fundamentals

GPU number formats: FP16 to FP4 and MX

Updated 10 Nov 2026 · 9 min read

A model is billions of numbers (its weights). How many bits each number takes decides three things at once: how much GPU memory the model needs, how many bytes each generated token must read from memory, and how fast the matrix engines run. Datacenter GPUs now compute on numbers as small as 4 bits. This page shows what those formats are, encodes real values in them, and shows what they cost in accuracy.

The idea first

A float is scientific notation with a fixed number of digits. Write 0.3 as 1.2×2−21.2 \times 2^{-2}. A floating-point number stores exactly those pieces, in binary, in a fixed number of bits:

Field The question it answers In 0.3=1.2×2−20.3 = 1.2 \times 2^{-2} What more bits buy
sign positive or negative? + (always 1 bit)
exponent roughly how big: which power of two? 2−22^{-2} a wider range: bigger and smaller numbers
mantissa where exactly between that power of two and the next? 1.2 finer precision: more digits

A format is named by how it splits its bits: E4M3 is 4 exponent bits and 3 mantissa bits, plus the sign, 8 bits in total.

A float is a ruler whose ticks spread out as numbers grow. With 3 mantissa bits, every power-of-two interval (1 to 2, 2 to 4, 4 to 8 …) gets the same 8 ticks. So between 1 and 2 the step is 0.125; between 8 and 16 it is 1. Every value you store snaps to the nearest tick. These are all the values FP8 E4M3 can hold between 0 and 16:

FP8 E4M3: every value it can hold from 0 to 16
0124816

Step size: 0.0625 near 0.5 · 0.125 near 1 · 0.25 near 2 · 0.5 near 4 · 1 near 8. The relative error stays about the same everywhere; the absolute error grows with the value.

Scaling is choosing the unit. A 4-bit float holds only 0, 0.5, 1, 1.5, 2, 3, 4 and 6 (and their negatives). A real weight such as 0.0137 is far below all of them. The fix is the one you use when you write a length in millimetres instead of metres: store the small numbers in a unit (a scale factor) that fits the data, and write the unit down once next to them. Real value=stored number×scale\text{Real value} = \text{stored number} \times \text{scale}.

Where the analogy breaks. The unit is not chosen per number: one scale is shared by a block of neighbouring weights (32 in the OCP MX formats), and in MX it must be a power of two. If one weight in the block is much bigger than the rest, the unit has to fit that one, and the small ones lose their digits. The second worked example below shows exactly that.

The formats

Format Bits Layout Range and precision, roughly Typical use
FP32 32 E8M23 huge range, about 7 decimal digits accumulators, optimizer state
BF16 16 E8M7 FP32’s range, about 2–3 digits training, most served weights today
FP16 16 E5M10 small range (max 65,504), more digits older training, some inference
FP8 E4M3 8 E4M3 max ±448 inference weights and activations
FP8 E5M2 8 E5M2 max ±57,344, fewer digits gradients in training
FP6 6 E2M3 or E3M2 max 7.5 or 28 weights, with block scales
FP4 4 E2M1 only 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6 weights, with block scales

BF16 keeps FP32’s 8 exponent bits and drops mantissa bits: same range, fewer digits. FP16 makes the opposite trade. FP4 has exactly 16 bit patterns.

Worked example: one weight in FP8 E4M3

Encode w = 0.3 in FP8 E4M3, bit by bit

1 · Sign

0.3 is positive, so the sign bit is 0. A negative weight would flip only this bit.

2 · Exponent: which power of two?

The largest power of two not above 0.3 is 2-2 = 0.25, so 0.3 = 1.2 × 2-2. The exponent field stores the power plus a fixed bias of 7, so that negative powers fit in an unsigned field: -2 + 7 = 5 = binary 0101.

3 · Mantissa: where between 2⁻² and 2⁻¹?

The leading 1 of 1.2 is implied and not stored. The rest, 1.2 − 1, must be written in eighths (3 bits): (1.2 − 1) × 8 = 1.6. That is not a whole number, so it rounds to 2 = binary 010. This rounding is the only place information is lost.

4 · Decode and measure the error

Stored bits: 0 0101 010. Reading them back: (1 + 2/8) × 2^-2 = 1.25 × 0.25 = 0.3125. The weight changed from 0.3 to 0.3125, an error of 4.2 %. Between 0.25 and 0.5 E4M3’s ticks are 1/32 apart, and 0.3 falls between 0.28125 and 0.3125.

5 · The same weight in every format

FormatBitsSignExponentMantissaStoredError
FP3232001111101001100110011001100110100.3000000123.97e−6 %
BF161600111110100110100.300781250.26 %
FP161600110100110011010.3000488280.0163 %
FP8 E4M38001010100.31254.17 %
FP8 E5M28001101010.31254.17 %
FP6 E2M360000100.2516.7 %
FP4 E2M1400010.566.7 %

Each halving of the bits costs digits. FP4 can only say “0.5”: a 67 % error on its own. That is why 6- and 4-bit formats are never used without a scale.

E4M3: 1 sign bit, 4 exponent bits with bias 7, 3 mantissa bits. Computed by the same encoder as the tool below.

Try your own values. Watch where each format runs out of range (500 in E4M3, 70000 in FP16) or of digits (0.00001 becomes 0 in E4M3):

Number-format encoder
FormatBits: sign exponent mantissaStored valueErrorRange (largest, smallest step near 0)
FP32E8M23±3.4028e+38, then ∞ · 1.4e−45
BF16E8M7±3.3895e+38, then ∞ · 9.18e−41
FP16E5M10±65504, then ∞ · 5.96e−8
FP8 E4M3E4M3±448 (saturates) · 0.00195
FP8 E5M2E5M2±57344, then ∞ · 1.53e−5
FP6 E2M3E2M3±7.5 (saturates) · 0.125
FP6 E3M2E3M2±28 (saturates) · 0.0625
FP4 E2M1E2M1±6 (saturates) · 0.5

Round to nearest, ties to even. FP32, BF16, FP16 and E5M2 overflow to ∞; E4M3, FP6 and FP4 have no ∞ and saturate at their largest value, as quantizers do. Values below half the smallest step become 0 (“flushed”).

Scaling: one shared multiplier per block

  • Per-tensor scaling (classic FP8): one scale for the whole tensor. One outlier anywhere forces a large scale, and the small values lose their digits.
  • Block scaling (microscaling): one scale per small block of neighbouring values. An outlier only hurts its own block.

The industry standard for block scaling is the OCP Microscaling (MX) specification, written by AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm and others. In every concrete MX format a block is 32 elements and the shared scale is E8M0: an 8-bit power of two, with no mantissa.

MX format Element Bits per value including the scale
MXFP8 E4M3 or E5M2 8 + 8/32 = 8.25
MXFP6 E2M3 or E3M2 6.25
MXFP4 E2M1 4.25
MXINT8 8-bit integer 8.25

NVIDIA’s own 4-bit format, NVFP4, uses two levels of scale: an FP8 (E4M3) scale for every 16 values, plus one FP32 scale for the whole tensor. The smaller block and the finer scale (E4M3 has mantissa bits; E8M0 does not) cost a little more space, about 4.5 bits per value, and NVIDIA states the accuracy is close to FP8. AMD CDNA 4 (MI350 series) supports the OCP MX formats in hardware. NVIDIA documents NVFP4 as the Blackwell 4-bit format; this page did not verify which OCP MX formats Blackwell runs natively.

Worked example: one MXFP4 block of 32 weights

32 weights → one E8M0 scale + 32 four-bit numbers

1 · The block: 32 neighbouring weights

w: 32 weights (4 rows of 8)
−0.00190.0066−0.00530.00270.01370.00370.00070.0082
−0.00630.00090.0043−0.00590.00830.02340.0116−0.0003
0.0013−0.0002−0.0082−0.0063−0.0025−0.0016−0.0029−0.0105
−0.00540.0050−0.0037−0.0042−0.00730.0087−0.0061−0.0021

In BF16 these take 32 × 2 = 64 bytes. Most are a few thousandths. One (outlined, 0.0234) is several times bigger than the typical value. Follow it and a small one, −0.0053 (also outlined).

2 · Pick the scale from the largest value

The largest magnitude is 0.0234, and its power of two is 2-6. FP4’s largest exponent is 2 (6 = 1.5 × 2²), so the shared exponent is -6 − 2 = -8: scale = 2^-8 = 0.00390625. Stored as E8M0 that is the exponent plus 127, 01110111: one byte for the whole block.

3 · Divide every weight by the scale

w ÷ scale
−0.491.69−1.360.693.510.950.182.10
−1.610.231.10−1.512.125.992.97−0.08
0.33−0.05−2.10−1.61−0.64−0.41−0.74−2.69
−1.381.28−0.95−1.08−1.872.23−1.56−0.54
The 32 scaled weights (dots) and the FP4 value each snaps to (ticks)
−6−4−3−2−1.5−1−0.500.511.52346

In units of the scale, the weights now span about −3 to +6: the range FP4 covers. The outlier lands at 5.99, right at FP4’s top; the small one at −1.357.

4 · Round each one to the nearest FP4 value

FP4 elements (4 bits each)
−0.51.5−1.50.54.01.00.02.0
−1.50.01.0−1.52.06.03.00.0
0.50.0−2.0−1.5−0.5−0.5−0.5−3.0
−1.51.5−1.0−1.0−2.02.0−1.5−0.5

−1.357 rounds to -1.5; 5.99 rounds to 6. 4 of the 32 weights round to 0: they were smaller than a quarter of the scale. None had to saturate.

5 · Decode: element × scale, and the error

Decoded = element × scale
−0.00200.0059−0.00590.00200.01560.00390.00000.0078
−0.00590.00000.0039−0.00590.00780.02340.01170.0000
0.00200.0000−0.0078−0.0059−0.0020−0.0020−0.0020−0.0117
−0.00590.0059−0.0039−0.0039−0.00780.0078−0.0059−0.0020
Error = decoded − w
−0.0001−0.0007−0.0006−0.00070.00190.0002−0.0007−0.0004
0.0004−0.0009−0.00040.0000−0.00050.00000.00010.0003
0.00070.00020.00040.00040.0005−0.00040.0009−0.0012
−0.00050.0009−0.00020.0003−0.0005−0.00090.00020.0001

The outlier comes back almost exactly (0.0234375). The small weight comes back as −0.00585938, 11 % off. Over the block the error is 8.7 % (root-mean-square, relative to the weights’ size). Models trained or calibrated for this keep their quality close to the original, but the error is real: low-bit serving is a model-quality decision, not only a hardware setting.

6 · What it saved

BF16 block
64 B
32 × 16 bits
MXFP4 block
17 B
32 × 4 bits + 8-bit scale = 136 bits
Smaller by
3.8×
memory and bytes read per token
Error
8.7 %
RMS, this block

3.8× less memory and bandwidth, and on MI355X the matrix cores run MXFP4 at 4× the BF16 rate (next section).

7 · Why per block: the same weights with one scale for the tensor

Decoded with the tensor's scale
0.00000.00000.00000.00000.00000.00000.00000.0000
0.00000.00000.00000.00000.00000.03130.00000.0000
0.00000.00000.00000.00000.00000.00000.00000.0000
0.00000.00000.00000.00000.00000.00000.00000.0000

Suppose another block of the same tensor holds a weight of 0.375, 16× our outlier. One scale for the whole tensor must fit that one: 2^-4 = 0.0625. In those units 31 of our 32 weights round to 0 and the error is 111 %: the block is gone. A scale per 32 weights keeps an outlier’s damage inside its own block.

8 · NVFP4 on the same weights

NVFP4 splits the 32 weights into two blocks of 16, each with an E4M3 scale, times one FP32 scale for the tensor. With the tensor scale chosen so that the largest block scale lands at E4M3’s maximum, 448 (one common recipe, not the only one), the two scales are 448 × 8.705e−6 = 0.0039 and 208 × 8.705e−6 = 0.001811. The second half has no outlier, so it gets a finer unit. Error: 8.68 % against MXFP4’s 8.7 %, for 144 bits instead of 136. On this block the gain is small; it grows when one half has a much smaller range than the other, and because an E4M3 scale need not be a power of two.

The 32 weights are made up (seeded random, a typical size for an LLM weight matrix) with one outlier, 0.0234. The rule is the OCP MX v1.0 one: scale = 2^(floor(log2(largest |w|)) − 2), elements rounded to FP4 with saturation. All numbers computed at build time.

Why fewer bits are faster

Two separate effects:

  1. Memory and bandwidth. Half the bits means half the bytes to store and half the bytes to read per token. Decode (generating one token at a time) reads every weight once per step, so it gets faster almost in proportion. See memory and roofline.
  2. Math throughput. A matrix engine has fixed wires. Narrower numbers let it do more multiply-adds per clock. AMD’s CDNA 4 white paper lists the rate per CU per clock; times 256 CUs and 2.4 GHz that gives the spec-sheet peaks:
Format on MI355XFLOPs per clock per CUPer matrix core (4 per CU)× 256 CUs × 2.4 GHzSpec sheet, approximate
FP16 / BF164,0961,0242.52 PFLOPS2.5 PFLOPS
FP8 (OCP) / MXFP88,1922,0485.03 PFLOPS5 PFLOPS
MXFP6 / MXFP416,3844,09610.1 PFLOPS10.1 PFLOPS

FP32 matrix math runs at 157.3 TFLOPS and FP64 at 78.6 TFLOPS: a GPU built for AI gives up wide-number speed. MI355X runs MXFP6 at the MXFP4 rate. NVIDIA states 15 PFLOPS dense NVFP4 and 5 PFLOPS dense FP8 for one Blackwell Ultra GPU (the GPU in GB300); it lists FP6 support for Blackwell Ultra Tensor Cores, but this page did not verify its rate. All of these are vendor peaks: real kernels reach a fraction.

“Sparsity” numbers on spec sheets are double the dense ones. They apply only when a tensor follows a 2:4 pattern (two of every four values are zero). LLM weights normally do not, so use the dense figures.

How big is the model?

A 70-billion-parameter dense model is 140 GB in BF16, 70 GB in FP8 and about 37 GB in MXFP4. Reading it once at 8 TB/s takes at least 17.5, 8.8 and 4.6 ms. Halving the bits halves the floor on time per token, and the memory freed goes to the KV cache, which decides how many requests fit at once. Try other sizes:

Model weights by number format
Weights
MI355X needed for weights alone
Floor per decode step (read all weights once at 8 TB/s)
Ceiling, tokens/s for one sequence

Dense model, weights only, peak bandwidth: real steps are slower, and the KV cache and activations need memory too. A mixture-of-experts model reads only its active experts per token, so its floor is lower than this. Dashed line: one MI355X (288 GB).

Where the format lives in the stack

  • The checkpoint (model files) is stored in some format, for example BF16 or a pre-quantized FP8 or MXFP4 release.
  • The serving engine (vLLM) chooses kernels for that format. A flag or the model’s config picks the quantization method.
  • The kernels use matrix-core instructions for that format: MFMA (matrix fused multiply-add) on AMD, Tensor Core MMA on NVIDIA. See PTX and AMDGCN ISA.
  • Accumulation happens in wider precision (FP32), so only the inputs are narrow.

If a GPU lacks native support for a format, software can still unpack it to a wider type before the math. That saves memory but not math time. MI300X (CDNA 3), for example, has FP8 but no FP4 or FP6 matrix support; MI355X (CDNA 4) adds them.

Common misconceptions

  • “FP4 is 4× faster than FP16 everywhere.” Only the matrix math is. Attention, normalization and communication do not speed up the same way.
  • “MXFP4 and NVFP4 are the same.” Both are 4-bit E2M1 elements, but the block size (32 vs 16) and scale type (E8M0 vs E4M3) differ. A checkpoint is quantized for one of them.
  • “FP8 is one format.” There are two (E4M3, E5M2), and older non-OCP variants also exist. The OCP names are the standard ones.
  • “Peak PFLOPS tell you the speed.” Decode is limited by memory bandwidth, so HBM TB/s usually matters more than PFLOPS for it.

Further reading