encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 2 of 7 in Decisions

Rate-distortion optimization

Decisions · concept · added 2026-11-13

In Brief:
The same block can be coded in several ways, some with fewer bits and some closer to the source. Rate-distortion optimization scores each candidate as error plus lambda times bits and keeps the lowest of those it tested. Lambda sets what a bit is worth.

An encoder can code the same block in several ways. Some use fewer bits; others reproduce the source more closely. Rate-distortion optimization (RDO) compares them with one score:

cost J = distortion + λ × bits

Lambda sets how strongly the score penalizes bits. A larger lambda favors smaller encodings; a smaller one gives the encoder more reason to spend bits reducing error. The encoder takes the lowest-cost option among those it tests.

Try changing lambda

Below are six candidate encodings of one macroblock, each plotted by its bits and its distortion. Distortion here is the sum of squared differences (SSD) between the source pixels and the reconstructed ones. The slider is the quantization parameter (QP), and lambda follows it.

One simplification matters: the six candidates stay fixed while you drag. In a real encode, a different QP would also change every candidate's bits and distortion.

26
← low QP: bits are cheaphigh QP: bits are expensive →
Bits → ↑ SSD 0 2,000 4,000 0 100 200 300
λ at this QP
Winning candidate
Its cost J
ModeBitsSSDCost J
Simplified demo: the six bit and SSD figures are invented. Lambda is based on x264's relationship for SSD costs, 0.9 × 2^((QP − 12)/3), before x264 rounds it into a table; see How lambda follows the quantization parameter. Lambda is shown to two decimals, and the costs use the unrounded value.

The dashed line joins every point that would cost the same as the winner. All other candidates lie on or above it. Across the slider the winner moves from Intra 4×4 (QP 12 to 17) through Inter 8×8, Inter 16×8 and Inter 16×16 to Skip (QP 32 and up). Intra 16×16 never wins: Inter 8×8 beats it on both counts, 104 bits against 150 and an SSD of 1,250 against 1,500.

The geometry

Points of equal cost lie on a straight line whose slope is −λ. The winner is the first candidate that line reaches as it rises from the origin. As lambda changes, the winner moves between the points on the lower convex hull, the lower boundary of the set. A candidate above that boundary is never chosen, whatever lambda is.

Bits → ↑ Distortion
Lower convex hullEqual-cost line, high λ: steepEqual-cost line, low λ: shallow
Conceptual illustration using the demo's six points. The hollow point is Intra 16×16, above the hull.

Why one lambda

In a simplified model, where blocks are independent and their costs simply add up, the best allocation gives every block the same price for a bit: if one block gained more per bit than another, moving a bit to it would help. Real encoders adapt the idea, because blocks predict from one another, choices are discrete, and different error measures need differently scaled weights. Rate control manages the overall bitrate or quality target. RDO chooses how to code individual blocks within it.

In a real encoder

  1. Generate candidates. Partition sizes, intra prediction modes, reference frames, motion vectors, transform sizes, skip.
  2. Use quick estimates to narrow the choices. Early stages skip the coding transform and quantization. They measure the prediction residual with the sum of absolute differences (SAD) or the sum of absolute transformed differences (SATD), and add the header bits weighted by a lambda scaled for those measures; see Distortion measures.
  3. Evaluate the remaining candidates. Transform and quantize the residual, reconstruct the block, measure its SSD against the source, and count or estimate the bits. x264 counts them exactly for its variable-length coder and estimates fractional bits for its arithmetic coder.
  4. Keep the minimum. The same score is reused at finer grain, for example per coefficient in trellis quantization.

How perceptual adjustments change the cost

Most perceptual tools change one side of the score. Adaptive quantization changes QP per block, and lambda moves with it. x264's psychovisual rate-distortion optimization (psy-RD) adds a term to SSD for how far the reconstruction's texture energy is from the source's, so losing texture costs more.

Trade-offs

Sources

Read 2026-11-13, at the commit linked.