Why quality metrics disagree
A full-reference quality metric compares two pictures, the source and the decoded encode, and gives one number. Different metrics look at different things, so they can rank the same encodes differently, and none of them is a viewer. Knowing what each one measures tells you which disagreements to expect.
Four errors of the same size
The cat is damaged four ways, each tuned to nearly the same mean squared error (MSE): the average of the squared pixel differences. Peak signal-to-noise ratio (PSNR) is that error on a decibel scale, so the four score nearly the same. Structural similarity (SSIM) compares the two pictures window by window and gives 1 when they are identical. The version here is x264's: one scale, luma only, 49 overlapping 8×8 windows on this picture. The last panel has one cell per window, 7 by 7, not one per pixel: white is 1, darker is lower.
| Damage | PSNR, dB | SSIM | Lowest window |
|---|---|---|---|
| Brightness shift | |||
| Noise | |||
| Blur | |||
| Coarse quantization |
At an error of 25:
- All four score 34.2 dB, and SSIM runs from 0.9790 to 0.9994. Noise and coarse quantization are lowest, blur is at 0.9827, the brightness shift is highest.
- SSIM changes little for this brightness shift. At an error of 144 every pixel is 12 levels brighter, none clips, and SSIM is 0.9969: each window keeps its contrast and pattern, and its mean moves little relative to its brightness. PSNR gives the shift 26.5 dB, the same as a blur that SSIM puts at 0.8901.
- The order changes with the size of the error. At 25 blur scores above noise, 0.9827 against 0.9790. At 144 it scores below, 0.8901 against 0.9136.
- The mean hides the worst window. Noise at 144 has a mean of 0.9136 and one window at 0.26: top middle, between the ears, where the source is plain sky and the noise is most of what the window contains.
Which of the four looks worst is for you to judge from the Damaged panel; neither number settles it.
What each one measures
PSNR = 10 × log10(255² / MSE) SSIM of one window = (2·a·b + C1) × (2·cov + C2) / ((a² + b² + C1) × (va + vb + C2))
Here 255 is the peak of an 8-bit sample. In the second formula a and b are the two windows' means, va and vb their variances and cov their covariance; C1 and C2 are small constants that keep flat windows from dividing by zero. The first factor compares brightness and the second compares contrast and pattern. A picture's SSIM is the mean over its windows. Implementations differ in window, weighting, scale and edge handling, so another tool's SSIM need not match these digits.
Video Multi-Method Assessment Fusion (VMAF) is a different kind of thing: a model trained on viewers' scores turns several features of each frame into one number, for one viewing setup.
The model read for this page and its viewing setup
The model read for this page, vmaf_v0.6.1, takes six features in three families: visual information fidelity at four scales, a detail loss measure and a measure of the difference between adjacent frames. Its output is clipped to 0 to 100. It was trained for a 1080-line television watched from three screen heights, and its documentation warns against comparing a score computed at 1080 lines with one computed at 480. Other models target other viewing setups, and a score is not a percentage of detail kept.
In x264
x264 can print both measures of its own output, and it warns when one is taken with psychovisual optimization or adaptive quantization set the wrong way for that metric. The warning is about benchmark setup, not arithmetic: to learn how your normal settings behave, measure those settings.
The options, the warning and the two tunings
The options are --psnr and --ssim, and SSIM is also printed in decibels as −10 × log10(1 − SSIM). If psychovisual optimization is on, x264 logs that the results "will be invalid", and the same for PSNR with adaptive quantization on or SSIM with it off; it tells you to use --tune psnr or --tune ssim when benchmarking. The psnr tuning turns both tools off; the ssim tuning turns psychovisual optimization off and sets adaptive quantization to its auto-variance mode. With those tools on, each score still measures the decoded picture. When you report a measurement, say which settings it was made with.
Trade-offs
- PSNR ranks by squared error and is blind to arrangement. For pictures of one size and one peak it orders them exactly as their squared error does, and it cannot tell a shift from a blur of the same error.
- SSIM responds to pattern more than to a small brightness shift. The same shift costs more where the picture is darker, and a mean over windows can hide a few bad ones.
- VMAF is fitted to viewers, for one viewing setup. Its scores depend on the model and the resolution it is computed at.
- A metric can be tuned for. An encoder set up to score well on one measure is not thereby better on another, or to a viewer.
Sources
Read 2026-11-15 and 2026-11-16, at the commits linked.
- x264 at commit c24e06c (GitHub mirror): in
encoder/encoder.c,calc_psnr,calc_ssim_db, the warnings about measuring with psychovisual optimization or adaptive quantization set the wrong way, the line that names a tuning for benchmarking and the two-pixel offset of the windows; incommon/pixel.c, the sums over 4×4 blocks,ssim_end1with its constants, andx264_pixel_ssim_wxh, which steps the windows;--psnrand--ssimin the option list; the psnr and ssim tunings incommon/base.c. - Netflix's VMAF repository at commit a6ed0f1: the name; the three core features in
resource/doc/features.md; the feature list, model type and score range inmodel/vmaf_v0.6.1.json; the viewing setup the default model was trained for and the warning about comparing resolutions, the note that its SSIM tool downsamples first and others do not and the separate model for a 4K television inresource/doc/faq.md. - The SSIM page at the University of Waterloo that VMAF's documentation cites: its suggested usage, which averages and downsamples a picture by a factor set from its size before measuring. Read as extracted text through a fetch tool. x264 measures the frame at its own size, and its
calc_psnrreports 100 dB for identical pictures, whose PSNR is not finite. - Not checked: the papers that define SSIM and the VMAF features, which the page does not cite; VMAF's feature code and any VMAF score (none is computed here); the code of any SSIM implementation other than x264's; chroma; any viewing test.