encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 4 of 7 in Decisions

Mode decision and early termination

Decisions · concept · added 2026-11-15

In Brief:
Mode decision is the search that proposes ways to code a block and picks one. Encoders save work by comparing the promising candidates carefully and dropping others early. A shortcut that rests on an estimate can miss a better choice; a valid bound can rule a candidate out without changing the winner.

For every block an encoder has many ways to code it: skip it, predict it from another frame whole or in pieces, or code it as intra. Rate-distortion optimization says how to compare two candidates. Mode decision is the search that produces the candidates and picks one, and it saves work by dropping candidates before they are measured in full.

The order of the tests

Early skip check16×168×8, smaller16×8, 8×16IntraMeasure in full only if 8×8 was closetighter if unlikelyclose candidates only
teal: can end the search gray: cheap estimates coral: the expensive comparison
Conceptual illustration of x264's order for a macroblock in a P frame, with the conditions under the lower row shortened. The early skip check is drawn as its own step; from --subme 3 x264 makes it during the 16×16 search. The drawing omits reference frame choice, sub-pixel refinement and B frames.

The pattern is general: rank candidates with a cheap estimate, then spend the expensive measurement on the few that are close. x264's cheap estimate is a sum of absolute differences at --subme 0 and 1 and, in lossy coding, a sum of absolute transformed differences above that, plus an estimate of the side bits. Its expensive one codes the block and uses the reconstruction-based rate-distortion cost.

Early termination stops a search before every candidate is measured; pruning drops single candidates. A bound can prove a candidate cannot win, as in x264's exhaustive motion search. A judgment that it is unlikely to win is heuristic pruning, which is what the three rules below are.

Prune the search and see what it costs

One frame of the walking cat is predicted from the frame before, in 16 units of 8×8 pixels. Each unit has four candidates: skip, inter 8×8 with one motion vector, inter 4×4 with four, and intra. With Pruning off, all four are evaluated in full and the lowest cost J wins, where J is the squared error plus lambda times modeled bits: the all-four baseline. With Pruning on, sums of absolute differences decide what is worth a full evaluation: a unit whose skip difference is at or under Skip threshold is skipped with no other test, the 4×4 split is evaluated only if its vectors beat the single one, and intra only if its prediction beats inter. Full evaluations counts these; the motion searches and the intra estimate run for every unit either way and are not counted.

300
← 0: skip only exact copies800: skip readily →
28
← finer quantizercoarser quantizer →
Source
Prediction
Modes
Reconstructed
light: skip teal: inter 8×8 dark teal with a cross: inter 4×4 coral: intra
Full evaluations
Total cost J
Over baseline
Decided differently
Simplified demo: the cat is invented; the transform is a floating-point 4×4 cosine transform at the nominal H.264 step for the quantization parameter (QP). The reference is the previous frame coded as intra. Motion search tries nine whole-pixel positions. The bits are modeled (a fixed price per mode, vector and level), lambda is an invented 0.85 × 2^((QP − 12) / 3), and the three pruning rules are simplified versions of the kind x264 uses, not its thresholds. Skip here copies the pixels at the same position in the reference; x264's skip uses a predicted vector, which need not be zero. The baseline compares four built candidates per unit, not every vector and partition. "Decided differently" counts units whose mode differs from the baseline at the same QP. Intra wins at no setting offered.

For this frame at QP 28:

What x264 does

x264 ranks candidates by the cheap estimate and makes the final choice by the rate-distortion cost from --subme 6 in I and P slices and 7 in B slices. With fast skip on, an early skip check can end a P macroblock's search, and later tests are opened only when earlier estimates were close. Only --subme 11, which the placebo preset asks for, turns these mode-analysis shortcuts off.

The levels, the skip check and the gates
  • The other levels of --subme. Below 6 the comparison is by the cheap estimate only. From 8, for I and P slices, the choice among macroblock types is also refined within the type; B slices reach each stage one level later, at 7 and 9. From 10, in every slice type, x264 tries neighboring QPs too. That needs --trellis 2 and adaptive quantization: without them x264 lowers 10 and 11 to 9. The default is 7.
  • How the skip check works. It needs --subme 3 or more: the 16×16 search on the first reference can trigger a skip probe when its prediction cost is small and its vector is close to the predicted skip vector, and x264 takes skip only if the probe succeeds. --no-fast-pskip turns the shortcut off, not skip: a 16×16 macroblock that ends with the skip vector and no coded residual is still written as skip.
  • Which estimates open which tests. The searches below 8×8 are opened only if 8×8 beat 16×16, and 16×8 and 8×16 are tried only if 8×8 came close. Intra is held to a tighter limit when the evidence x264 has makes intra look unlikely. Several P partition candidates reach the rate-distortion pass only when their estimate is within about 25% of the best; other candidates have other gates.
  • What --subme 11 leaves in place. The limits other options set on references, partitions and motion search stay.

Trade-offs

Sources

Read 2026-11-15, at the commit linked.