The hybrid codec loop
H.264, High Efficiency Video Coding (HEVC) and AV1 share a basic design: predict a block, then transform, quantize and code the difference between the source and the prediction. "Hybrid" refers to combining prediction with transform coding. This page follows that common path; some coding modes skip or replace stages.
One block around the loop
The source is the block the encoder was given. The prediction is an estimate of it built from available reconstructed pixels: ones the encoder has already coded and rebuilt the way a decoder will. They can be neighbors in the same frame (intra prediction) or come from another frame that has already been decoded (motion compensation). The residual is the source minus the prediction.
A transform turns the residual into coefficients, and quantization turns each coefficient into a level, an integer that is often zero. The entropy coder writes the levels, the prediction information and the parameters needed to interpret them, such as the quantization parameter, as bits.
The lower path undoes the upper one as far as it can: the levels are scaled back into approximate coefficients, inverse transformed and added to the prediction. The result is the reconstruction, the block a decoder will produce. The next page, Why the encoder contains a decoder, covers why predictions use it and not the source.
Send a cat around the loop
Choose a prediction and a quantization step. The picture is 32×32 pixels in 4×4 blocks. In the residual and error panels mid-gray is zero, lighter is positive and darker is negative; the error is reconstructed minus source, amplified four times. The grid holds the 16 levels of the outlined block, the cat's winking eye. Its rows and columns are frequencies, not pixel positions. The top-left level represents the residual's average offset after quantization; the others describe patterns that vary across the block.
Select Rough and move Step. The prediction changes too, because later blocks use neighbors that have already been reconstructed.
At step 16, for this picture:
- The closer predictions reduced the residual and the total level magnitude. Mean squared residual is 4,225 with Flat, 1,176 with Rough and 376 with Close; the sum of level magnitudes is 1,459, 907 and 473.
- The error followed the step more than the prediction. Mean squared error is 13.0, 12.9 and 13.4.
- Close misses the wink. The earlier frame had the outlined eye open, so in that block Close leaves 14 nonzero levels of 16 and Rough leaves 7.
A standard is not an encoder
H.264 is a coding standard. It defines what the bits mean and how a decoder turns them into pictures: in the diagram, the teal steps and the format of the bits. It does not say how to choose a prediction or which levels to code. x264 is an encoder implementation, one program that makes those choices and writes H.264. Two encoders can produce different conforming bitstreams from the same source, and a decoder that supports the profile and level they use can decode either.
| Standard | Encoders named in this course |
|---|---|
| H.264 | x264 |
| HEVC | x265 |
| AV1 | libaom, SVT-AV1 |
So a setting belongs to an encoder. x264's --me and --trellis change how it searches for a prediction and how it picks levels. Neither changes the format, and a decoder needs neither to decode the result.
The loop in x264
In its lossy 4×4 intra path, x264 codes a block in x264_mb_encode_i4x4, and the same stages appear in that function. The source sits in a buffer named fenc; the prediction is in a second one, fdec. sub4x4_dct subtracts one from the other and transforms the difference, and x264_quant_4x4 produces the levels. If any level is nonzero, dequant_4x4 and add4x4_idct rebuild the residual and add it to the prediction in fdec, which then holds the reconstruction. Around this, the main macroblock loop separates analysis, encoding and writing the coded data.
What to carry forward
- The transform and the quantizer work on the residual, so what they are given depends on the prediction.
- The decoder gets levels, prediction information and the parameters needed to interpret them, not the source.
- A standard fixes the decoder's side of the loop. Every setting in this course is an encoder's.
Sources
Read 2026-11-13, at the commit linked.
- x264 at commit c24e06c (GitHub mirror):
x264_mb_encode_i4x4, with the two buffers, the prediction, the subtraction and transform, the quantization and the rebuilding; what the buffers hold;sub4x4_dctandadd4x4_idct;x264_quant_4x4; analysis, encoding and writing in the macroblock loop. Two paths that skip stages: the lossless branch, which neither transforms nor quantizes, and the macroblock type that copies the source pixels. A parameter sent for the decoder: the slice's quantization parameter, as a difference. - ITU-T Recommendation H.264, the 05/2003 edition with Corrigendum 1, clause 3 (definitions) only, read as extracted text: the decoding process is the one the Recommendation specifies (3.38), the encoding process is not specified (3.43), and a profile and a level are a subset of the syntax and a set of constraints on its values (3.100, 3.65). Later editions number these clauses differently.
- ITU-T H.264.1 (10/2014), summary page only: a decoder claims conformance to a profile and level.
- Not checked: the rest of H.264, including the clauses for the modes that skip stages; the HEVC and AV1 specifications; the origin of the term "hybrid"; x264's other block types, which were not read line by line for this page; the loop in x265, libaom and SVT-AV1.