Estimating bit cost before coding
x264 compares candidates by reconstruction-based rate-distortion cost, which needs a bit count for each one, and it does not write the candidates to find out. For ordinary syntax of its variable-length coder it adds up exact codeword lengths. For decisions of its arithmetic coder it usually adds up fractional prices from a table, in units of 1/256 bit, using a copy of the coder's contexts. Its decision routines also use shortcuts and penalties.
Where the counting happens
rd_cost_mb reconstructs a macroblock, measures its error, and then counts bits one of three ways. A skipped macroblock is given a fixed price of one bit. With the arithmetic coder, COPY_CABAC copies the context bytes and macroblock_size_cabac runs on the copy. With the variable-length coder, macroblock_size_cavlc runs.
Neither is written separately: rdo.c includes the writers themselves, cavlc.c and cabac.c, a second time, with each write call redefined to add to a counter. Most of the count therefore walks the same decisions as the writer. Some branches are taken only when counting: one prices a partition without all of its syntax, and another adds a 2,000-bit penalty where a level would overflow.
Compare the estimate with interval narrowing
The demo carries x264's own tables. It takes the cat's flags from the first page of this unit, row by row, all in one context, and keeps two tallies. A flag is a symbol with two possible values: there is one per pixel, zero where the pixel equals the pixel to its left and nonzero where it differs.
Table estimate is what x264_cabac_size_decision does: add the table's price for the flag in the context's current state, then advance the state. Interval cost measures how much the chosen intervals shrink under x264's rules for splitting its range, in bits:
interval cost = shifts + log2(510 / range)
The range starts at 510, and the shifts are the doublings that bring it back up after a split. Interval cost is not a count of output bits: the demo writes nothing and ends no stream. The plot shows the estimate minus the interval cost; below zero, the estimate is the smaller. Starting state is what the context expects before the first flag.
Measured on these 1,024 flags:
- For these flags the estimate stayed within 4.81 bits of the interval cost. From even odds it ends at 839.62 bits against 843.06, a difference of 3.44 bits, or 0.41%. That is not a bound on its error against a finished H.264 stream.
- It ran low. Over the three starting states the difference ranges from −4.81 to +0.52 bits.
- The first flag shows the two rules. From even odds the table says 1.00 bit, and the interval narrows by a factor worth 1.09 bits. From Expects zero the first flag, a nonzero, is 5.66 bits by the table and 5.82 by the interval.
- The starting state is paid for once. By the table, starting from Expects nonzero costs 866.18 bits and from Expects zero 831.76, a difference of 34.4. For this sequence the three context states first coincide after 89 flags, and every later table price is then the same.
The table prices a decision from the context's state alone. The split that sets the interval cost also depends on the range, through a table with four columns. x264's own comment calls the estimate not exactly the same.
The prices
Every entry adds to one counter. For the arithmetic coder the unit is 1/256 bit.
| What is priced | Added |
|---|---|
| A decision in a context | x264_cabac_entropy[state ^ value] |
| An even-odds decision | 256 |
| A terminal decision | 7 |
A skipped macroblock, in rd_cost_mb | 1 bit |
| A variable-length codeword | Its length in bits |
A decision in a context is priced from the table and advances the context. An even-odds decision is one bit and touches no context. The terminal decision, x264_cabac_encode_terminal_c, gives the other outcome 2 of a range of at least 256 when written and is a fixed 7 when counted. In rd_cost_mb the count becomes the rate term with lambda, which x264 stores multiplied by 256:
arithmetic coder: (count × lambda + 32768) >> 16 variable-length coder: (bits × lambda + 128) >> 8
- Alternatives for the same decision start from the same contexts. The copy is made for each one, so pricing one does not change what the next is priced with.
- Runs of decisions are priced in one lookup.
x264_rdo_initprecomputes the price and the final state of a run of ones in one context, which trellis quantization uses for level magnitudes. - Earlier stages use cheaper stand-ins. During motion search a motion vector difference is priced from a table built by formula, for either coder: the fast-cost lambda × (2 × log2(|d| + 1) + 1.718) for each nonzero component d of the difference, in quarter pixels, and that lambda × 0.718 for a zero component.
- x265 and libaom keep price tables too. x265's
Entropy::encodeBinaddsg_entropyBits[state ^ bin], in units of 1/32768 bit, when no bitstream is attached. libaom'sav1_cost_symbol_clampedprices a symbol from its current probability in units of 1/512 bit.
Sources
Read 2026-11-14 and 2026-11-15, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
rd_cost_mb, the skip price, the arithmetic coder's rate term, the variable-length coder's, the stored lambda, 256 times the formula,COPY_CABAC, the variable-length writer redefined as a counter, with its comment that the count is exact, the arithmetic writer redefined, the terminal decision counted as 7,x264_cabac_encode_terminal_c, the even-odds decision,x264_cabac_size_decision,x264_cabac_size_decision_noup, the calls it replaces when counting, the same calls as ordinary decisions when writing,x264_cabac_pos, the partition path used only for counting, the overflow penalty,x264_cabac_encode_decision_c,cabac_encode_renorm, the starting range, the range table, the transition table, the shift table, the price table,x264_rdo_init, the motion vector formula, its use per component. - x265 at commit 4191822 (GitHub mirror): the estimate branch of
Entropy::encodeBin, an even-odds decision priced at 32768,g_entropyBits, its lookup. - libaom at commit 1b7f9ef (GitHub mirror): the 1/512 unit,
av1_cost_symbol_clamped, its table. - Not checked: where x264 uses the price lookup that does not advance the state; every branch that only the counting takes; which of x264's analysis levels call which of these paths; how far the estimate is from the bits of a finished stream, on these flags or on real video; how x265 and libaom use their tables in mode decision; SVT-AV1.