Test design pitfalls
An encoder comparison is a small experiment: encode, measure, summarize, compare. Each step has a choice in it that can change the conclusion with no change to the encoder's code. This page collects the ones the last three pages ran into, and shows one more with numbers: how per-frame scores are pooled into one.
Four places a test can go wrong
Pool the same frames two ways
Twenty frames: ten of the cat out of focus, then ten with grain. At the left end of the slider every frame is coded at quantization parameter (QP) 28. Moving right codes the soft frames finer and the grainy frames coarser by the same number of steps. Peak signal-to-noise ratio (PSNR) is computed per frame, then summarized two ways. The mean of the frame values averages decibels. The pooled-error value averages the frames' squared error first and converts once. The demo treats every frame as equally long and measures luma only. x264 prints both kinds, weighting each frame by its duration: "Mean" averages the frame values of one plane, and "Global" converts the squared error of all planes and frames once.
At QPs 22 and 34:
- By the mean of frame values the change looks small. 36.68 dB against 36.74 at QP 28 for every frame, with 11.4% fewer modeled bits.
- By pooled error the same frames are 3.7 dB lower. 32.65 dB against 36.36. Squared error grows faster in the frames that got worse than it shrinks in the frames that got better.
- The worst frame fell from 34.68 to 29.62 dB. Frame 11 went from 35.0 to 29.9 dB while frame 1 went from 38.5 to 43.2.
- At the right end the two summaries order the settings oppositely. The mean is the highest of any setting, 37.04 dB against 36.74 at QP 28 for every frame, with 6.4% fewer modeled bits; the pooled value is the lowest, 28.84 against 36.36.
Neither summary is wrong, and neither says which allocation a viewer would prefer. They answer different questions, so a result has to say which one it is.
For frames of equal size and weight the pooled value is never above the mean of frame values.
The two formulas, and the gap in this demo
Mean frame PSNR = 10 × log10(255² / G) Pooled-error PSNR = 10 × log10(255² / A)
G is the geometric mean of the frames' mean squared errors and A is their arithmetic mean. An arithmetic mean is never below a geometric mean, which fixes the order of the two summaries, and they agree only when every frame has the same error. Here they are 0.38 dB apart at QP 28 for every frame and 8.20 dB apart at the right end.
A checklist
- Decide the question before the settings.
- Line the pictures up, and keep a record.
- Measure at one size, chosen for the metric's model and the viewing setup.
- Say how frames were pooled.
- Compare curves, not points.
- Use more than one kind of content.
- Look at the pictures.
Each item, with its reason
- Decide the question before the settings. To ask how efficiently an encoder can code for a metric, use settings meant for that metric and say so: x264 has
--tune psnrand--tune ssimfor it, and warns when PSNR or structural similarity is measured with its psychovisual optimization on. To ask how a configuration you will deploy behaves, keep its settings, measure that output with each metric, and look at it. Encoding again with another tuning for each metric answers the first question, not the second. Where encoding effort matters to the conclusion, record the preset, encoding time, hardware and threads; a preset's name does not make two encoders' effort equal. - Line the pictures up, and keep a record. Match frames, crop, size, bit depth and color range before scoring: an offset of one frame changes a full-reference score by itself. Keep the source, encoder version and command line, achieved bitrate, metric version or model, scaler and pooling rule with the result.
- Measure at one size, chosen for the metric's model and the viewing setup. For a smaller decode with a full-size reference, the documentation of Video Multi-Method Assessment Fusion (VMAF) says to scale the decode up to the reference's size before measuring, and not to compare scores computed at different sizes.
- Say how frames were pooled. VMAF reports the plain mean of its frame scores by default, and its documentation lists a pooling option in several of its command-line tools: harmonic mean, median, minimum and low percentiles. Say whether frames count equally or by duration, and for several clips give each clip's result beside the aggregate.
- Compare curves, not points. Equal QP is not equal bitrate. Use several points a side and report the range they cover.
- Use more than one kind of content. On the previous page the best picture size at 1000 bits differed between two versions of one cat.
- Look at the pictures. Metrics disagree, and none of these pages measured a viewer.
Sources
Read 2026-11-15 and 2026-11-16, at the commits linked.
- x264 at commit c24e06c (GitHub mirror): in
encoder/encoder.c, the PSNR of one frame, the running total of squared error and the running sum of frame PSNRs, each weighted by the frame's duration, the closing line that prints Mean, Avg and Global, and the warnings about measuring with psychovisual optimization or adaptive quantization set the wrong way. Its PSNR routine reports 100 dB for a frame with no error, which has no finite PSNR; no frame of the demo is one. - Netflix's VMAF repository at commit a6ed0f1: in
resource/doc/faq.md, not comparing scores across sizes, scaling up before measuring, the default mean over frames and the other pooling choices. - Not checked: any standard test procedure or common test conditions; how many clips or runs a sound test needs; statistics of viewer tests; any real encoder run.