How lambda follows the quantization parameter
x264 and x265 normally derive lambda, the weight on bits in rate-distortion optimization, from the quantization parameter (QP), so that candidate selection and quantization work at related scales. This page compares their baseline formulas, as documented beside each encoder's lambda tables.
Why it grows as 2^(QP/3)
A common approximation makes squared-error distortion proportional to the square of the quantization step, while rate falls roughly with the logarithm of the step. Together these suggest a lambda proportional to the step squared. The step doubles every 6 QP, so the baseline lambda for squared-error costs grows about fourfold over 6 QP. These are modeling assumptions that explain the trend; the encoders tune the constants and store finite-precision tables.
The curves
The page compares two cost scales. The rate-distortion (RD) lambda weights bits against the sum of squared differences (SSD) in reconstruction-based decisions. The fast-cost lambda weights bits against the sum of absolute differences (SAD) or the sum of absolute transformed differences (SATD) in the fast early stages. Move the QP slider to read them off. The two scales have different distortion units, so the vertical gap between them is not a like-for-like comparison.
On a log scale all three are straight lines. Absolute-error costs grow in proportion to the error and squared-error costs with its square, so the fast-cost line rises at roughly half the rate of the other two. The exact constants and exponents are each encoder's choice; the fast-cost lambda is not exactly the square root of an RD lambda.
The formulas
| Encoder | Fast-cost lambda | RD lambda |
|---|---|---|
| x264 | 2^(QP/6 − 2) | 0.9 × 2^((QP − 12)/3) |
| x265 | 2^(QP/6 − 2) | 0.038 × e^(0.234 × QP) |
- x265's baseline RD formula gives smaller values than x264's. About 68% of x264's at QP 0 and 79% at QP 51. With all other costs equal, a smaller lambda makes rate count for less against distortion. The ratio alone does not predict which encoder produces the larger file or the better picture at the same QP.
- Six QP is about a factor of four. Exactly 4 in x264's smooth formula and about 4.07 in x265's. Ratios between stored table entries vary a little.
- x264's stored fast-cost lambda is an integer. It is 1 for every QP below 16, where the formula gives 0.25 at QP 0. It is 5 at QP 26 and 91 at QP 51, close to the formula.
- x264 picks its mode-decision weights from the analysis QP. That QP includes per-block offsets such as those from adaptive quantization, so the weights move with them.
- When reconstruction-based decisions begin. In x264, reconstruction-based macroblock mode decision starts at
--subme 6in I and P slices and 7 in B slices. Below that, mode decision uses the fast cost. Trellis quantization has its own lambda table. - The mapping can be replaced. x265's
--lambda-fileloads different tables.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
x264_lambda_tab,x264_lambda2_tab, stored times 256, the trellis table,mb_analyse_init_qp, where reconstruction-based mode decision begins,rd_cost_mb. - x265 at commit 4191822 (GitHub mirror):
x265_lambda_tab,x265_lambda2_tab, the conversion to fixed point,--lambda-file. - Not taken from either encoder: the step-squared argument in the first section, which is a standard approximation and was not taken from a source read for this page.