encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 3 of 7 in Decisions

How lambda follows the quantization parameter

Decisions · implementation · added 2026-11-13

x264 and x265 normally derive lambda, the weight on bits in rate-distortion optimization, from the quantization parameter (QP), so that candidate selection and quantization work at related scales. This page compares their baseline formulas, as documented beside each encoder's lambda tables.

Why it grows as 2^(QP/3)

A common approximation makes squared-error distortion proportional to the square of the quantization step, while rate falls roughly with the logarithm of the step. Together these suggest a lambda proportional to the step squared. The step doubles every 6 QP, so the baseline lambda for squared-error costs grows about fourfold over 6 QP. These are modeling assumptions that explain the trend; the encoders tune the constants and store finite-precision tables.

The curves

The page compares two cost scales. The rate-distortion (RD) lambda weights bits against the sum of squared differences (SSD) in reconstruction-based decisions. The fast-cost lambda weights bits against the sum of absolute differences (SAD) or the sum of absolute transformed differences (SATD) in the fast early stages. Move the QP slider to read them off. The two scales have different distortion units, so the vertical gap between them is not a like-for-like comparison.

26
← finer quantizationcoarser quantization →
0.01 1 100 10,000 ↑ lambda, log scale QP → 0 12 24 36 48
RD lambda formula, x264RD lambda formula, x265Fast-cost formula, before x264's rounding
RD lambda, x264
RD lambda, x265
x265 ÷ x264
Fast-cost formula
Smooth baseline formulas for 8-bit operation. The readouts evaluate these formulas, not the integer and fixed-point values the encoders store. x264 rounds its fast-cost lambda to an integer with a minimum of 1, and both encoders store their RD weights at finite precision.

On a log scale all three are straight lines. Absolute-error costs grow in proportion to the error and squared-error costs with its square, so the fast-cost line rises at roughly half the rate of the other two. The exact constants and exponents are each encoder's choice; the fast-cost lambda is not exactly the square root of an RD lambda.

The formulas

EncoderFast-cost lambdaRD lambda
x2642^(QP/6 − 2)0.9 × 2^((QP − 12)/3)
x2652^(QP/6 − 2)0.038 × e^(0.234 × QP)

Sources

Read 2026-11-13, at the commits linked.