encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 2 of 3 in Why video compresses

The hybrid codec loop

Why video compresses · concept · added 2026-11-13

In Brief:
H.264, High Efficiency Video Coding and AV1 share one basic design: predict a block, then transform, quantize and code what the prediction missed. A standard fixes the decoder's side of that loop; the choices are the encoder's.

H.264, High Efficiency Video Coding (HEVC) and AV1 share a basic design: predict a block, then transform, quantize and code the difference between the source and the prediction. "Hybrid" refers to combining prediction with transform coding. This page follows that common path; some coding modes skip or replace stages.

One block around the loop

The source is the block the encoder was given. The prediction is an estimate of it built from available reconstructed pixels: ones the encoder has already coded and rebuilt the way a decoder will. They can be neighbors in the same frame (intra prediction) or come from another frame that has already been decoded (motion compensation). The residual is the source minus the prediction.

A transform turns the residual into coefficients, and quantization turns each coefficient into a level, an integer that is often zero. The entropy coder writes the levels, the prediction information and the parameters needed to interpret them, such as the quantization parameter, as bits.

Source − Transform Quantize Entropy coder + Rescale, inverse transform Reconstruction Predict Choose prediction residual coefficients levels prediction bits rebuilt residual prediction information
teal boxes: steps a decoder repeats gray boxes: steps only the encoder runs
Conceptual illustration of the block loop. The arrow from Reconstruction to Predict stands for reconstructed pixels available to later predictions, not to the same block. Picture storage and in-loop filtering are omitted. A decoder first reads what it needs from the bitstream, then runs the teal steps.

The lower path undoes the upper one as far as it can: the levels are scaled back into approximate coefficients, inverse transformed and added to the prediction. The result is the reconstruction, the block a decoder will produce. The next page, Why the encoder contains a decoder, covers why predictions use it and not the source.

Send a cat around the loop

Choose a prediction and a quantization step. The picture is 32×32 pixels in 4×4 blocks. In the residual and error panels mid-gray is zero, lighter is positive and darker is negative; the error is reconstructed minus source, amplified four times. The grid holds the 16 levels of the outlined block, the cat's winking eye. Its rows and columns are frequencies, not pixel positions. The top-left level represents the residual's average offset after quantization; the others describe patterns that vary across the block.

16
← fine stepscoarse steps →
Source
Prediction
Residual
Block levels
Reconstructed
Error, ×4
Mean squared residual
Nonzero levels
Sum of level magnitudes
Mean squared error
Simplified demo: the cat and all three predictions are invented. Rough fills each block with the average of the reconstructed pixels just above and to its left. Flat predicts mid-gray, 128, everywhere. Close is an invented earlier frame, taken as already decoded and used directly, without motion search: the cat half a pixel to the left, both eyes open. The transform is a floating-point 4×4 cosine transform with rounding to nearest, not H.264's integer transform. In the residual and error panels, values beyond the display range appear black or white; the statistics cover the whole picture and use the full differences. Neither the count of nonzero levels nor the sum of their magnitudes is a bitrate, and the demo does not count the bits needed to describe each prediction.

Select Rough and move Step. The prediction changes too, because later blocks use neighbors that have already been reconstructed.

At step 16, for this picture:

A standard is not an encoder

H.264 is a coding standard. It defines what the bits mean and how a decoder turns them into pictures: in the diagram, the teal steps and the format of the bits. It does not say how to choose a prediction or which levels to code. x264 is an encoder implementation, one program that makes those choices and writes H.264. Two encoders can produce different conforming bitstreams from the same source, and a decoder that supports the profile and level they use can decode either.

StandardEncoders named in this course
H.264x264
HEVCx265
AV1libaom, SVT-AV1

So a setting belongs to an encoder. x264's --me and --trellis change how it searches for a prediction and how it picks levels. Neither changes the format, and a decoder needs neither to decode the result.

The loop in x264

In its lossy 4×4 intra path, x264 codes a block in x264_mb_encode_i4x4, and the same stages appear in that function. The source sits in a buffer named fenc; the prediction is in a second one, fdec. sub4x4_dct subtracts one from the other and transforms the difference, and x264_quant_4x4 produces the levels. If any level is nonzero, dequant_4x4 and add4x4_idct rebuild the residual and add it to the prediction in fdec, which then holds the reconstruction. Around this, the main macroblock loop separates analysis, encoding and writing the coded data.

What to carry forward

Sources

Read 2026-11-13, at the commit linked.