Mode decision and early termination
For every block an encoder has many ways to code it: skip it, predict it from another frame whole or in pieces, or code it as intra. Rate-distortion optimization says how to compare two candidates. Mode decision is the search that produces the candidates and picks one, and it saves work by dropping candidates before they are measured in full.
The order of the tests
--subme 3 x264 makes it during the 16×16 search. The drawing omits reference frame choice, sub-pixel refinement and B frames.The pattern is general: rank candidates with a cheap estimate, then spend the expensive measurement on the few that are close. x264's cheap estimate is a sum of absolute differences at --subme 0 and 1 and, in lossy coding, a sum of absolute transformed differences above that, plus an estimate of the side bits. Its expensive one codes the block and uses the reconstruction-based rate-distortion cost.
Early termination stops a search before every candidate is measured; pruning drops single candidates. A bound can prove a candidate cannot win, as in x264's exhaustive motion search. A judgment that it is unlikely to win is heuristic pruning, which is what the three rules below are.
Prune the search and see what it costs
One frame of the walking cat is predicted from the frame before, in 16 units of 8×8 pixels. Each unit has four candidates: skip, inter 8×8 with one motion vector, inter 4×4 with four, and intra. With Pruning off, all four are evaluated in full and the lowest cost J wins, where J is the squared error plus lambda times modeled bits: the all-four baseline. With Pruning on, sums of absolute differences decide what is worth a full evaluation: a unit whose skip difference is at or under Skip threshold is skipped with no other test, the 4×4 split is evaluated only if its vectors beat the single one, and intra only if its prediction beats inter. Full evaluations counts these; the motion searches and the intra estimate run for every unit either way and are not counted.
For this frame at QP 28:
- At threshold 0, pruning cut the full evaluations by about a third and chose the same modes. 43 against 64, and every unit got the baseline's mode. The count is not a timing.
- Skipping more readily saved more evaluations and raised the cost. At 300, 35 evaluations and a cost J 6.4% over the baseline, with 3 of 16 units decided differently. At 400, 24 evaluations and 21.9% over. A rise in J is not a rise in bits: at 300 the modeled bits fell from 1,143.5 to 943.5 while the squared error rose from 18,037 to 28,578.
- In this experiment the threshold that reproduced the baseline depended on QP. At QP 34 the baseline itself skips 13 units, and a threshold of 400 gives the baseline's result with 35 evaluations. At QP 22 a threshold of 300 costs 136% more. QP changes the reconstructed reference here as well as lambda and the step.
What x264 does
x264 ranks candidates by the cheap estimate and makes the final choice by the rate-distortion cost from --subme 6 in I and P slices and 7 in B slices. With fast skip on, an early skip check can end a P macroblock's search, and later tests are opened only when earlier estimates were close. Only --subme 11, which the placebo preset asks for, turns these mode-analysis shortcuts off.
The levels, the skip check and the gates
- The other levels of
--subme. Below 6 the comparison is by the cheap estimate only. From 8, for I and P slices, the choice among macroblock types is also refined within the type; B slices reach each stage one level later, at 7 and 9. From 10, in every slice type, x264 tries neighboring QPs too. That needs--trellis 2and adaptive quantization: without them x264 lowers 10 and 11 to 9. The default is 7. - How the skip check works. It needs
--subme3 or more: the 16×16 search on the first reference can trigger a skip probe when its prediction cost is small and its vector is close to the predicted skip vector, and x264 takes skip only if the probe succeeds.--no-fast-pskipturns the shortcut off, not skip: a 16×16 macroblock that ends with the skip vector and no coded residual is still written as skip. - Which estimates open which tests. The searches below 8×8 are opened only if 8×8 beat 16×16, and 16×8 and 8×16 are tried only if 8×8 came close. Intra is held to a tighter limit when the evidence x264 has makes intra look unlikely. Several P partition candidates reach the rate-distortion pass only when their estimate is within about 25% of the best; other candidates have other gates.
- What
--subme 11leaves in place. The limits other options set on references, partitions and motion search stay.
Trade-offs
- Heuristic pruning trades a risk of a worse choice for less work. A rule is good when the candidates it drops would rarely have won. The demo's first two rules never dropped a winner on this frame; that is one frame.
- A fixed threshold is wrong somewhere. x264 scales its limits by lambda or by another candidate's cost, so they follow QP and content.
- The cheap estimate can disagree with the real cost. When it does, pruning by it throws away the better mode before it is measured. Higher
--submelevels measure more candidates for that reason.
Sources
Read 2026-11-15, at the commit linked.
- x264 at commit c24e06c (GitHub mirror): in
encoder/analyse.c, the comparison level from--submeand the level taken one lower in B slices, except for the QP test, early termination off only at 11, the fast skip test with its form from level 3 and its conditions in the 16×16 search (the first reference, a cost under 300 times lambda, a vector within one quarter-pixel step of the skip vector, and a successful probe), the searches below 8×8 opened only if 8×8 was cheaper, the test for trying 16×8 and 8×16, the fast intra decision (intra neighbors, the same macroblock of the reference frame, and how many intra macroblocks the slice has had) and the limit it sets, the rate-distortion pass limited to close candidates and its separate limit of three halves for 16×16; inencoder/encoder.c, levels 10 and 11 lowered to 9 without trellis 2 and adaptive quantization and the comparison function chosen by level; inencoder/macroblock.c, the later change of a 16×16 macroblock to skip; incommon/mvpred.c, the predicted skip vector; inencoder/me.c, successive elimination in the exhaustive search; incommon/base.c, the default level of 7, fast skip on by default, placebo at level 11 and with fast skip off,--subme;--no-fast-pskipin the command-line list. - Not checked: the tests inside the skip probe (
x264_macroblock_probe_pskip); the rest of x264's analysis (B frame modes, reference choice, the further limits inside the 16×8 and 8×16 searches, chroma); what each--submelevel does to sub-pixel motion refinement; mode decision in x265, libaom and SVT-AV1; how much time or quality any rule saves on real video.