Quantization and the quantization parameter (QP)
Quantization replaces each transform coefficient with an integer level, and the decoder uses that level to rebuild an approximation of the coefficient. The quantization parameter (QP) controls the scale: a larger QP usually means coarser quantization and fewer bits.
Why round at all
After the discrete cosine transform (DCT), a block's prediction residual is a set of coefficients, most of them small. A reversible transform changes the representation without discarding information. Quantization reduces precision: small coefficients become exact zeros and large ones become small integer levels, which entropy coding then compresses.
level = round(coefficient / step) reconstructed coefficient = level × step
The encoder codes the level. The reconstructed coefficient is a multiple of the step, not the original value.
How it works
Move the QP slider. The ruler shows one coefficient snapping to the nearest multiple of the step size. The patch below it is 16 blocks of 8×8 pixels put through a DCT, the same rounding, and the inverse transform. The third square is the error, reconstructed minus source, amplified four times: mid-gray means no change, lighter means brighter than the source and darker means darker. QP 0 here is only the finest step; it does not simulate an encoder's lossless mode.
The steps in a real encoder:
- Transform. The prediction residual becomes a block of coefficients.
- Divide and round. Each coefficient is divided by the step size and rounded to an integer, the level. Fixed-offset quantizers can favor zero, as the next page shows; trellis and similar searches choose levels by rate-distortion cost.
- Send levels and QP. The bitstream carries the levels and what the decoder needs to determine QP. In H.264, QP starts from a slice value and can be updated by macroblock deltas; a macroblock without one keeps the current value.
- Rebuild. The decoder reconstructs each coefficient from its level and the step. The level does not identify the original value exactly. With rounding to nearest, as in this demo, the error is at most half a step per coefficient; biased rounding and rate-distortion searches can leave more.
The scale
QP is a logarithmic index into step sizes. In H.264 and High Efficiency Video Coding (HEVC) the step doubles every 6 QP, so one QP is about 12% on average; the six values inside each doubling are not evenly spaced. In the model used here, QP 4 is a step of 1. x264 and x265 do not divide: each holds a six-row table of integer multipliers and adds a bit shift of QP ÷ 6. The two encoders' table values differ.
| Codec | Scale at 8 bits | Step size |
|---|---|---|
| H.264, HEVC | QP 0 to 51 | Doubles every 6 |
| AV1 | Index 0 to 255 | Read from lookup tables, one for the constant-component coefficient (DC) and one for the rest (AC) |
Each extra bit of sample depth extends the H.264 and HEVC range by six finer steps. The standards number those with negative QPs; x264 numbers a 10-bit encode 0 to 63.
Trade-offs
- QP is not quality. The same step can be invisible in busy texture and obvious on a flat wall, which is why adaptive quantization varies QP block by block.
- Coarse steps tend to remove detail first. High-frequency coefficients are usually the small ones, so they tend to reach zero first: texture blurs, and block edges can start to show.
- QP also affects mode decision. The weight on bits in mode decision is derived from it; see How lambda follows the quantization parameter.
- QP numbers don't transfer. AV1's index is a different scale, and two encoders at the same QP still differ in rounding and mode decision. Compare by bitrate and quality.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
dequant4_scale,quant4_scale, theq%6andq/6indexing,QP_BD_OFFSETandQP_MAX_SPEC, the slice QP delta, the macroblock QP delta, and when it is skipped, constant QP 0 selecting the lossless path. - x265 at commit 4191822 (GitHub mirror):
remandper, its six scale values. - libaom at commit 1b7f9ef (a GitHub mirror):
dc_qlookup_QTX,ac_qlookup_QTX. - Not checked against the standards themselves: the step sizes in the model, which are x264's
dequant4_scalefirst column divided by 16 (the usual statement of H.264's step-size table); the negative QP numbering at higher bit depths; the HEVC QP syntax.