Psychovisual rate-distortion optimization
Rate-distortion optimization picks the candidate with the lowest squared error plus priced bits. Squared error counts the change that losing texture makes, but gives it no extra weight for being a loss of texture. Removing fine detail lowers the bit count and can cost little squared error, so a search on those two alone can favor a smoother reconstruction. x264's psychovisual rate-distortion optimization adds a third term that charges for the difference in texture energy between the source and the reconstruction.
What the extra term measures
For a block, x264 measures how much it varies: it applies a Hadamard transform, drops the coefficient that holds the average, and adds up the magnitudes of the rest. It does this for the source block and for the reconstruction, and the cost of a candidate becomes
cost = squared error
+ psy x lambda_E x |energy gap|
+ lambda_R x bits
The two weights are different numbers: lambda_E scales the energy term and lambda_R prices the bits, and x264 takes them from two tables. In this experiment the variation is measured at two scales, 4×4 and 8×8, and the penalty combines the absolute difference between source and reconstruction at each. "Energy" here is a sum of transform magnitudes, not the squared deviation of Block energy. A reconstruction with less variation than the source pays, and so does one with more. The term does not ask where the detail is, only that there is about as much of it; the squared error beside it still does.
Turn the term up
The cat has invented grain of up to 8 levels added to every pixel. This is an invented search over four rounding offsets with an energy penalty of x264's kind; it is not x264's mode search or its trellis term. Each of the cat's 16 units of 8×8 pixels is coded with an offset of 0 (truncate toward zero), 1/6, 1/3 or 1/2 (round to nearest). A larger offset quantizes fewer coefficients to zero, as Dead zone and rounding explains. The lowest cost wins. Psy is the weight of the energy term, as in x264's --psy-rd. Error is the mean squared error against the source. In the error panel mid-gray is no change.
ssd_plane and pixel_hadamard_ac. Invented: the four candidates, the prediction (each 4×4 block's own average, which is not priced, nor is any other syntax), the price of a level, and the two weights, lambda_E = 2^(QP/6 − 2) and lambda_R = 0.9 × 2^((QP − 12)/3): nominal forms, not x264's integer tables, and 12.7 and 145.1 at QP 34. "Units by rounding" counts units from offset 0 to offset 1/2. "Energy ratio" is the reconstruction's total energy divided by the source's over all units. It is not the cost's sum of gaps: a rise in one unit offsets a fall in another, and 100% would not mean every gap is zero. Values beyond the display range in the error panel appear black or white.For this cat at a quantization parameter (QP) of 34:
- With the term off, 14 of 16 units chose the offset of 1/3, and the energy ratio was 82.2%. The price of bits made the more aggressive zeroing look cheaper.
- Turning it up moved units to finer rounding. At 0.5, 11 units round to nearest; the energy ratio rises to 93.3% and modeled bits from 1,003 to 1,133. At 1.0, 12 units, 93.4% and 1,138.
- In this demo the error fell too, from 59.86 to 49.81 at 1.0, because rounding to nearest is also the lowest-error choice here. The term's effect was to outbid the price of bits. The case usually described, where it accepts more squared error to keep texture, did not arise with these four candidates and was not measured.
- In this experiment the candidates the term chose cost more bits. The comparison is at a fixed QP, not at an equal bitrate. Under a bitrate target a real encoder may pay with more distortion elsewhere.
In x264
--psy-rd takes two numbers, 1.0:0.0 by default. The first is this term, which x264 turns on from --subme 6. The second is a related term inside trellis quantization, off by default. --no-psy turns both off, as do the psnr and ssim tunings.
Slice types, chroma and the tunings
- The term is part of the reconstruction-based cost, which mode decision uses from 6 in I and P slices and from 7 in B slices. The energy term is luma only; the option does more, among it a lower chroma QP offset to compensate.
- The two metric tunings switch the terms off to suit what each of those metrics measures; Why quality metrics disagree is about the difference. The animation tuning lowers the first to 0.4; stillimage raises them to 2.0 and 0.7; film, grain and touhou turn on the second (0.15, 0.25, 0.2).
Trade-offs
- It optimizes a proxy. Matching the amount of variation is not the same as matching the detail. Energy of the wrong kind, such as ringing, also counts as energy.
- It can work against the metrics. Measured by squared error, an encode with the term can score worse than one without, though it did not in this demo. That is why comparisons of encoders have to say whether such tools were on.
- Its place is textured content. The flat areas of a cartoon have little energy to protect, though its outlines are high in contrast. The animation tuning sets a lower value; that is a setting, not a measurement.
Sources
Read 2026-11-15, at the commit linked.
- x264 at commit c24e06c (GitHub mirror): in
encoder/rdo.c,ssd_planewith luma only, the energy difference, its weight and lambda and the sum with squared error, and the second term inside trellis;pixel_hadamard_acand the larger block shapes summed from 8×8 pieces; the lambda the term uses, the two lambda tables, the chroma weighting that depends on the option, the level taken one lower in B slices and the levels of--submethat use the rate-distortion cost; inencoder/encoder.c, the term enabled from level 6 and the chroma QP offset lowered when it is on; incommon/base.c, the defaults,--psy-rd,--no-psyand the tunings film, animation, grain, stillimage, psnr, ssim, touhou; the options in the command-line list. - Not checked: the trellis term beyond its existence; the other measure x264 uses for blocks smaller than 8×8; what the chroma changes do to a picture; x265's
--psy-rdand--psy-rdoq; any study of whether viewers prefer the result, which this page does not cite because none was read.