encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 3 of 5 in Transform and quantization

Choosing a transform size

Transform and quantization · concept · added 2026-11-13

In Brief:
The residual can be transformed in larger or smaller blocks. Larger transforms often represent a smooth residual with fewer nonzero coefficients; smaller ones can limit how far the error around an edge extends.

After prediction, what is left to code is the residual: the difference between the source block and its prediction. The encoder can transform that residual in larger or smaller blocks. Larger transforms often represent a smooth residual with fewer nonzero coefficients; smaller ones can limit how far the error around an edge extends.

Why one size does not fit

Each coefficient of a discrete cosine transform (DCT) describes a pattern that extends across its whole block. Quantizing a coefficient changes the strength of that pattern, so the resulting error can extend across the block too. Near a sharp edge that error can show as ripples, called ringing, and a smaller block limits how far they reach. Where the residual changes smoothly, a larger block can often describe it with fewer nonzero coefficients.

What counts is the residual, not the picture. An edge that the prediction reproduces well leaves little to transform, whatever the picture looks like there.

Gray lines: large transforms on smooth areas Coral: small transforms along the edge
Conceptual illustration, not the output of an encoder. A real encoder decides from the residual and from measured cost, not from how the picture looks.

Try different transform sizes

Pick a patch and a transform size. The patch is 16×16 pixels, coded with one 16×16 transform, four 8×8 or sixteen 4×4, all at the same quantization step. Teal lines mark the transform blocks. In the error panel, mid-gray means no change, lighter means brighter than the source and darker means darker.

16×16
← small: 4×4large: 16×16 →
Source
Reconstructed
Error, ×4
Nonzero coefficients
Mean squared error
Pixels off by 3 or more levels
Simplified demo: invented patches, a floating-point DCT, plain rounding and a fixed quantization step of 20. It transforms the patches directly, with no prediction. The nonzero count is how many coefficients survive quantization; it is not a bitrate. All three sizes share a quantization step, not a bitrate.

How an encoder chooses

x264 is the example. For inter-predicted macroblocks where the 8×8 transform is allowed, it uses one of two methods:

For intra macroblocks the transform size comes with the prediction mode. Either way, the encoder then tells the decoder which size it used: H.264 carries a flag at macroblock level, written only when the syntax calls for it. x264 omits it, for example, when the macroblock has no luma coefficients.

What codecs offer

CodecSizesEncoder settings
H.2644×4; also 8×8 in profiles that support it, such as Highx264: --no-8x8dct removes the 8×8 choice, and so do the Baseline and Main profile settings
High Efficiency Video Coding (HEVC)4×4, 8×8, 16×16 and 32×32, chosen by a quadtree: a block is split into four, and each quarter can be split againx265: --max-tu-size (default 32), --tu-intra-depth and --tu-inter-depth (default 1). Presets change the depths.
AV1Square and rectangular sizes from 4×4 up to 64×64, such as 16×64. It also has an identity option, which is not a DCT-style transform.Not yet covered

Trade-offs

Sources

Read 2026-11-13, at the commits linked.