encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 2 of 5 in Transform and quantization

Quantization and the quantization parameter (QP)

Transform and quantization · concept · added 2026-11-13

In Brief:
Quantization replaces each transform coefficient with an integer level, so the decoder rebuilds an approximation. The quantization parameter sets the step: a larger value usually means coarser steps and fewer bits.

Quantization replaces each transform coefficient with an integer level, and the decoder uses that level to rebuild an approximation of the coefficient. The quantization parameter (QP) controls the scale: a larger QP usually means coarser quantization and fewer bits.

Why round at all

After the discrete cosine transform (DCT), a block's prediction residual is a set of coefficients, most of them small. A reversible transform changes the representation without discarding information. Quantization reduces precision: small coefficients become exact zeros and large ones become small integer levels, which entropy coding then compresses.

level = round(coefficient / step)
reconstructed coefficient = level × step

The encoder codes the level. The reconstructed coefficient is a multiple of the step, not the original value.

rounding error one step everything in this band becomes 0 Coefficient in → ↑ Value the decoder gets
Illustrative. The dashed diagonal is exact reconstruction; the staircase is a quantizer that rounds to nearest. A larger step makes every tread wider, the zero band included.

How it works

Move the QP slider. The ruler shows one coefficient snapping to the nearest multiple of the step size. The patch below it is 16 blocks of 8×8 pixels put through a DCT, the same rounding, and the inverse transform. The third square is the error, reconstructed minus source, amplified four times: mid-gray means no change, lighter means brighter than the source and darker means darker. QP 0 here is only the finest step; it does not simulate an encoder's lossless mode.

26
← fine steps, more bitscoarse steps, fewer bits →
0 100 source coefficient: 37.3
Source
Reconstructed
Error, ×4
Step size
Level for 37.3
Nonzero coefficients
Mean squared error
Simplified demo: the patch is invented and its pixels are transformed directly, not a prediction residual. It uses a simplified H.264-style step scale with a floating-point 8×8 DCT and rounding to nearest. It does not reproduce H.264's integer transform, its coefficient-dependent scaling or an encoder's search. The nonzero count is how many coefficients survive rounding, not a bitrate. The error view is amplified four times and clipped to the display range; mean squared error uses the unamplified error.

The steps in a real encoder:

  1. Transform. The prediction residual becomes a block of coefficients.
  2. Divide and round. Each coefficient is divided by the step size and rounded to an integer, the level. Fixed-offset quantizers can favor zero, as the next page shows; trellis and similar searches choose levels by rate-distortion cost.
  3. Send levels and QP. The bitstream carries the levels and what the decoder needs to determine QP. In H.264, QP starts from a slice value and can be updated by macroblock deltas; a macroblock without one keeps the current value.
  4. Rebuild. The decoder reconstructs each coefficient from its level and the step. The level does not identify the original value exactly. With rounding to nearest, as in this demo, the error is at most half a step per coefficient; biased rounding and rate-distortion searches can leave more.

The scale

QP is a logarithmic index into step sizes. In H.264 and High Efficiency Video Coding (HEVC) the step doubles every 6 QP, so one QP is about 12% on average; the six values inside each doubling are not evenly spaced. In the model used here, QP 4 is a step of 1. x264 and x265 do not divide: each holds a six-row table of integer multipliers and adds a bit shift of QP ÷ 6. The two encoders' table values differ.

CodecScale at 8 bitsStep size
H.264, HEVCQP 0 to 51Doubles every 6
AV1Index 0 to 255Read from lookup tables, one for the constant-component coefficient (DC) and one for the rest (AC)

Each extra bit of sample depth extends the H.264 and HEVC range by six finer steps. The standards number those with negative QPs; x264 numbers a 10-bit encode 0 to 63.

Trade-offs

Sources

Read 2026-11-13, at the commits linked.