Choosing a transform size
After prediction, what is left to code is the residual: the difference between the source block and its prediction. The encoder can transform that residual in larger or smaller blocks. Larger transforms often represent a smooth residual with fewer nonzero coefficients; smaller ones can limit how far the error around an edge extends.
Why one size does not fit
Each coefficient of a discrete cosine transform (DCT) describes a pattern that extends across its whole block. Quantizing a coefficient changes the strength of that pattern, so the resulting error can extend across the block too. Near a sharp edge that error can show as ripples, called ringing, and a smaller block limits how far they reach. Where the residual changes smoothly, a larger block can often describe it with fewer nonzero coefficients.
What counts is the residual, not the picture. An edge that the prediction reproduces well leaves little to transform, whatever the picture looks like there.
Try different transform sizes
Pick a patch and a transform size. The patch is 16×16 pixels, coded with one 16×16 transform, four 8×8 or sixteen 4×4, all at the same quantization step. Teal lines mark the transform blocks. In the error panel, mid-gray means no change, lighter means brighter than the source and darker means darker.
- For this gradient, the larger transform does better. One 16×16 transform leaves 6 nonzero coefficients where sixteen 4×4 transforms leave 48, and its mean squared error is 1.5 against 6.8.
- For this edge, smaller transforms reduce the large errors. Pixels off by at least 3 brightness levels fall from 170 at 16×16 to 37 at 4×4, and mean squared error from 27 to 7.8. Smaller changes still occur elsewhere: nearly every pixel moves a little at every size.
- For this noise sample, size makes little difference. Nonzero coefficients run from 153 to 162 and mean squared error from 33 to 37. That is random noise; a repeating or directional texture has structure a transform can use.
How an encoder chooses
x264 is the example. For inter-predicted macroblocks where the 8×8 transform is allowed, it uses one of two methods:
- Quick estimate. When mode decision is not using reconstruction-based rate-distortion cost, x264 compares an 8×8-based Hadamard cost of the residual with a 4×4-based one and takes the smaller; see Distortion measures.
- Rate-distortion comparison. Otherwise it encodes the macroblock again with the other transform size and keeps whichever costs less in distortion plus lambda times bits. For a macroblock split into four 8×8 partitions it will also try changing the sub-partitions so that the 8×8 transform becomes allowed. Prediction and transform choices are therefore not made strictly one after the other.
For intra macroblocks the transform size comes with the prediction mode. Either way, the encoder then tells the decoder which size it used: H.264 carries a flag at macroblock level, written only when the syntax calls for it. x264 omits it, for example, when the macroblock has no luma coefficients.
What codecs offer
| Codec | Sizes | Encoder settings |
|---|---|---|
| H.264 | 4×4; also 8×8 in profiles that support it, such as High | x264: --no-8x8dct removes the 8×8 choice, and so do the Baseline and Main profile settings |
| High Efficiency Video Coding (HEVC) | 4×4, 8×8, 16×16 and 32×32, chosen by a quadtree: a block is split into four, and each quarter can be split again | x265: --max-tu-size (default 32), --tu-intra-depth and --tu-inter-depth (default 1). Presets change the depths. |
| AV1 | Square and rectangular sizes from 4×4 up to 64×64, such as 16×64. It also has an identity option, which is not a DCT-style transform. | Not yet covered |
Trade-offs
- Large transforms let ringing travel. Quantization error near an edge can show as ripples anywhere in the block, most visibly on a flat area beside the edge.
- Small transforms often cost more on smooth residuals. Each block carries its own average and its own low frequencies, and there are more block boundaries that can show.
- Prediction limits the choice. x264 does not allow the 8×8 transform when a macroblock is split into partitions smaller than 8×8.
- More sizes means more search. Every extra size or quadtree depth is another encode to try, which is why x265's depth settings change with its speed presets.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
mb_analyse_transformandmb_analyse_transform_rd, the two methods; where the quick estimate is selected;x264_transform_allowedandx264_mb_transform_8x8_allowed; the conditional flag; 8×8 on by default and off for Baseline and Main. - x265 at commit 4191822 (GitHub mirror): defaults, a preset raising the depths, the allowed sizes.
- libaom at commit 1b7f9ef (a GitHub mirror): the list of transform sizes and types.
- Not checked: the H.264 and HEVC size lists against the specification text, and the explanation of ringing, which is standard transform theory.