Why the encoder contains a decoder
The encoder and decoder need the same reference pixels for prediction. The decoder has reconstructed pixels, not the original source pictures, so the encoder rebuilds the pixels it will use as references. If the two sides use different references, the mismatch can carry into later predictions. This is called drift.
Two machines, matching pictures
The hybrid codec loop followed one block through an encoder. A decoder is given levels and prediction information, rebuilds the block and keeps the result for later predictions. The encoder performs the same reconstruction steps on the same levels; it does not need to read back and decode the bitstream it just wrote. Given the same coded data and reference state, its reconstruction matches a conforming decoder's.
Let the encoder predict from the source
The cat from the previous page walks a third of a pixel per frame for 25 frames. Frame 1 is predicted from flat mid-gray. Every later frame is predicted from the frame before it, with no motion search, and quantized at step 16. Choose what the encoder subtracts: its reconstruction of the previous frame, or the previous source frame. Either way the decoder can add the residual only to its own previous picture, which is what the Decoded panel shows. In the error panel mid-gray is zero, lighter is brighter than the source and darker is darker. Compare the two choices at frame 1, then move to frame 25 and switch between them.
For this cat at step 16:
- Predicting from the previous reconstruction, the error stayed within a narrow range. The decoder's mean squared error is between 13.0 and 14.3 on all 25 frames.
- Predicting from the previous source frame, it grew on every frame: 13.0 on frame 1, 23.5 on frame 2 and 318 on frame 25.
- The quantizer is the same in both; the residual is not. Suppose the previous source pixel was 100 and its reconstruction 96, and the next source pixel is 103. Ignoring new quantization error, subtracting the source gives a residual of 3, and the decoder gets 96 + 3 = 99. Subtracting the reconstruction gives 7, and the decoder gets 96 + 7 = 103. The correction for the earlier error is in the second residual and missing from the first.
Both paths have quantization error. In the correct loop, encoder and decoder agree on the reconstructed references. Using different references introduces an additional mismatch that can carry into later predictions: drift. It need not grow: with a cat that stands still, the error from the previous source frame stays at its frame-1 value of 13.0 on all 25 frames, because every later residual is zero.
Where x264 keeps its copy
x264 rebuilds each frame in a buffer named fdec. For a frame that later frames will use, fdec_filter_row deblocks the finished rows unless deblocking is off, and reference_update then adds fdec to the list of reference frames. So the reference is the filtered reconstruction, the step the diagram leaves out.
x264 also uses source pictures for analysis and quality measurements. Its lookahead estimates prediction costs from reduced-resolution source pictures; the final coding loop uses reconstructed references. With --psnr, the error it reports is measured between fdec and the source. --dump-yuv writes fdec to a file, which is the picture a decoder will produce.
What to carry forward
- The encoder reconstructs the pixels needed for later prediction.
- To measure reconstruction error, compare the decoded picture with the source.
- The encoder may inspect the source when making decisions. The prediction used to form the coded residual must match the decoder's prediction.
Sources
Read 2026-11-13, at the commit linked.
- x264 at commit c24e06c (GitHub mirror): the buffer being reconstructed;
fdec_filter_row, with the deblocking switch, the test for a frame kept as a reference and the deblocking call; the frame joining the reference list and the list being built from it; motion compensation reading a reference frame's planes; the input picture copied into the source frame, its reduced-resolution copy made byx264_frame_init_lowresand the frame queued for the lookahead; in the lookahead'sslicetype_mb_cost, the block to match and the reference both taken from reduced-resolution source pictures, the motion search and intra neighbors copied from the same source picture; the squared error for--psnr; the frame dump; the options--psnrand--dump-yuv. - ITU-T H.264.1 (02/2016), the conformance specification, clauses 6.5.3 and 6.5.5 only, read as extracted text: a conforming decoder's output sample values must equal those the decoding process specifies, they are compared with the reference decoder's, and the display process is outside its scope.
- Not checked: the matching requirements in HEVC and AV1; the rest of H.264.1 and H.264's own decoding clauses; how a decoder recovers from drift after lost data; x265, libaom and SVT-AV1; the rest of x264's lookahead.