encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 5 of 6 in Perceptual optimization

Psychovisual rate-distortion optimization

Perceptual optimization · concept · added 2026-11-15

In Brief:
A reconstruction can lose visible texture and still score well on squared error and bits. x264's psychovisual rate-distortion optimization adds a penalty when source and reconstruction differ in their amount of variation. It can favor texture; matching that measure does not guarantee matching detail or better-looking video.

Rate-distortion optimization picks the candidate with the lowest squared error plus priced bits. Squared error counts the change that losing texture makes, but gives it no extra weight for being a loss of texture. Removing fine detail lowers the bit count and can cost little squared error, so a search on those two alone can favor a smoother reconstruction. x264's psychovisual rate-distortion optimization adds a third term that charges for the difference in texture energy between the source and the reconstruction.

What the extra term measures

For a block, x264 measures how much it varies: it applies a Hadamard transform, drops the coefficient that holds the average, and adds up the magnitudes of the rest. It does this for the source block and for the reconstruction, and the cost of a candidate becomes

cost = squared error
     + psy x lambda_E x |energy gap|
     + lambda_R x bits

The two weights are different numbers: lambda_E scales the energy term and lambda_R prices the bits, and x264 takes them from two tables. In this experiment the variation is measured at two scales, 4×4 and 8×8, and the penalty combines the absolute difference between source and reconstruction at each. "Energy" here is a sum of transform magnitudes, not the squared deviation of Block energy. A reconstruction with less variation than the source pays, and so does one with more. The term does not ask where the detail is, only that there is about as much of it; the squared error beside it still does.

Turn the term up

The cat has invented grain of up to 8 levels added to every pixel. This is an invented search over four rounding offsets with an energy penalty of x264's kind; it is not x264's mode search or its trellis term. Each of the cat's 16 units of 8×8 pixels is coded with an offset of 0 (truncate toward zero), 1/6, 1/3 or 1/2 (round to nearest). A larger offset quantizes fewer coefficients to zero, as Dead zone and rounding explains. The lowest cost wins. Psy is the weight of the energy term, as in x264's --psy-rd. Error is the mean squared error against the source. In the error panel mid-gray is no change.

0.0
← 0: squared error and bits only2.0 →
34
← finer quantizercoarser quantizer →
Source
Reconstructed
Error, ×4
Rounding
lightest: offset 0, truncated toward zero darkest: offset 1/2, rounded to nearest; the two shades between are offsets 1/6 and 1/3
Units by rounding
Energy ratio
Modeled bits
Error
Simplified demo: the cat and its grain are invented, and grain is random noise, not texture with structure. The energy measure and the way it joins the cost are x264's, from ssd_plane and pixel_hadamard_ac. Invented: the four candidates, the prediction (each 4×4 block's own average, which is not priced, nor is any other syntax), the price of a level, and the two weights, lambda_E = 2^(QP/6 − 2) and lambda_R = 0.9 × 2^((QP − 12)/3): nominal forms, not x264's integer tables, and 12.7 and 145.1 at QP 34. "Units by rounding" counts units from offset 0 to offset 1/2. "Energy ratio" is the reconstruction's total energy divided by the source's over all units. It is not the cost's sum of gaps: a rise in one unit offsets a fall in another, and 100% would not mean every gap is zero. Values beyond the display range in the error panel appear black or white.

For this cat at a quantization parameter (QP) of 34:

In x264

--psy-rd takes two numbers, 1.0:0.0 by default. The first is this term, which x264 turns on from --subme 6. The second is a related term inside trellis quantization, off by default. --no-psy turns both off, as do the psnr and ssim tunings.

Slice types, chroma and the tunings
  • The term is part of the reconstruction-based cost, which mode decision uses from 6 in I and P slices and from 7 in B slices. The energy term is luma only; the option does more, among it a lower chroma QP offset to compensate.
  • The two metric tunings switch the terms off to suit what each of those metrics measures; Why quality metrics disagree is about the difference. The animation tuning lowers the first to 0.4; stillimage raises them to 2.0 and 0.7; film, grain and touhou turn on the second (0.15, 0.25, 0.2).

Trade-offs

Sources

Read 2026-11-15, at the commit linked.