encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 3 of 3 in Why video compresses

Why the encoder contains a decoder

Why video compresses · concept · added 2026-11-13

You'll want firstThe hybrid codec loop
In Brief:
A decoder predicts from the pixels it has rebuilt, not from the source. So the encoder rebuilds the same pixels and predicts from those. If the two sides predicted from different pictures, the mismatch could carry from frame to frame.

The encoder and decoder need the same reference pixels for prediction. The decoder has reconstructed pixels, not the original source pictures, so the encoder rebuilds the pixels it will use as references. If the two sides use different references, the mismatch can carry into later predictions. This is called drift.

Two machines, matching pictures

The hybrid codec loop followed one block through an encoder. A decoder is given levels and prediction information, rebuilds the block and keeps the result for later predictions. The encoder performs the same reconstruction steps on the same levels; it does not need to read back and decode the bitstream it just wrote. Given the same coded data and reference state, its reconstruction matches a conforming decoder's.

Source Choose prediction and code residual Rebuild the block Reconstruction Rebuild the block Reconstruction Encoder Decoder bits same pixels levels + prediction reference pixels reference pixels
teal boxes: in both machines gray box: encoder only
Conceptual illustration. "Rebuild the block" stands for the previous page's teal steps: rescale the levels, inverse transform and add the prediction, which the decoder forms from its reference pixels. Reference pixels serve later blocks and frames, not the block being rebuilt. The dashed line marks a match, not a transfer: each machine computes its own reconstruction. Entropy coding and parsing, picture storage and in-loop filtering are omitted.

Let the encoder predict from the source

The cat from the previous page walks a third of a pixel per frame for 25 frames. Frame 1 is predicted from flat mid-gray. Every later frame is predicted from the frame before it, with no motion search, and quantized at step 16. Choose what the encoder subtracts: its reconstruction of the previous frame, or the previous source frame. Either way the decoder can add the residual only to its own previous picture, which is what the Decoded panel shows. In the error panel mid-gray is zero, lighter is brighter than the source and darker is darker. Compare the two choices at frame 1, then move to frame 25 and switch between them.

25
← first framelast frame →
Source
Decoded
Error, ×4
Mean squared error in the decoder's picture Frame
Previous reconstructionPrevious source frame
Frame
Error, previous reconstruction
Error, previous source frame
Simplified demo: the cat and its motion are invented, and the transform is the previous page's floating-point 4×4 cosine transform with rounding to nearest. Both readouts are the mean squared error of the decoder's picture against the source frame. In the error panel, values beyond the display range appear black or white; the plot and readouts use the full differences. Choosing the previous source frame deliberately forms the coded residual from a prediction the decoder cannot match. It shows why the final encoding loop uses reconstructed references.

For this cat at step 16:

Both paths have quantization error. In the correct loop, encoder and decoder agree on the reconstructed references. Using different references introduces an additional mismatch that can carry into later predictions: drift. It need not grow: with a cat that stands still, the error from the previous source frame stays at its frame-1 value of 13.0 on all 25 frames, because every later residual is zero.

Where x264 keeps its copy

x264 rebuilds each frame in a buffer named fdec. For a frame that later frames will use, fdec_filter_row deblocks the finished rows unless deblocking is off, and reference_update then adds fdec to the list of reference frames. So the reference is the filtered reconstruction, the step the diagram leaves out.

x264 also uses source pictures for analysis and quality measurements. Its lookahead estimates prediction costs from reduced-resolution source pictures; the final coding loop uses reconstructed references. With --psnr, the error it reports is measured between fdec and the source. --dump-yuv writes fdec to a file, which is the picture a decoder will produce.

What to carry forward

Sources

Read 2026-11-13, at the commit linked.