Trellis quantization
Trellis quantization chooses coefficient levels by weighing reconstruction error against estimated bit cost. A level is the integer that quantization assigns to a coefficient. The choices are linked, because changing one level can change what the others cost to code, so the search looks for a low-cost combination for the whole block:
cost J = squared error + λ × bits
Why decide jointly
Plain rounding judges each coefficient alone, by distance. Bit cost does not work that way. Keeping a nonzero coefficient late in the scan can also require signaling the zeros before it; removing it may let the coded block end earlier, saving more bits than the coefficient alone would cost. And in adaptive arithmetic coding, coding one coefficient can change the estimated cost of later ones.
Try changing the price of a bit
Ten coefficients of one block, low frequency on the left. Gray bars show the original coefficients in units of the quantization step. Rings are plain rounding; teal dots are what the search picks. For each nonzero rounded level the search may keep it or move it one step closer to zero: 4 can become 3, and −4 can become −3.
This example uses invented bit costs and checks every permitted combination. It shows why coefficient choices interact; it does not reproduce x264's trellis algorithm.
| Method | Modeled bits | Squared error | Cost J |
|---|---|---|---|
| Rounding | |||
| Search |
At the starting setting the search saves 15 bits and adds 0.44 of squared error. With λ at 0.10 the bit savings outweigh the extra error, so the cost falls from 5.46 to 4.40. A lower cost does not mean less error or a better-looking picture.
Move λ from 0.03 to 0.04 to watch a joint decision: coefficients 6 and 8 go to zero together, and the block now ends after coefficient 5.
How x264's trellis works for its arithmetic coder
- Round to nearest. This gives each coefficient's starting level. The dead zone offsets are not used.
- Two candidates per nonzero level. Keep it, or move it one step closer to zero. x264 does not try a magnitude above the rounded one. That is x264's restriction, not part of what trellis quantization means.
- Walk the block in reverse scan order. x264 keeps the lowest-cost path into each of up to eight states of the level coder, pricing every candidate with estimated bits for its level and for the flags that mark nonzero positions and the last one.
- Follow the cheapest surviving path. It gives the levels to send.
The result is the best path through this restricted trellis under estimated costs, not a guarantee of the fewest real bits. In 8×8 blocks some positions share coding contexts, and x264's source says the search does not model that exactly. For its variable-length coder x264 uses a different search, which its source says is not a true trellis.
What encoders offer
| Encoder | Setting | Medium preset |
|---|---|---|
| x264 | --trellis 0 off; 1 on the final encode of each macroblock; 2 also inside mode decision | 1 |
| x265 | --rdoq-level 0 fixed rounding offsets, no cost search; 1 levels chosen by rate-distortion cost; 2 also considers zeroing whole groups of coefficients and dropping trailing levels of 1 | 0 |
Other presets and tunings change these: in both encoders the slow preset and those slower than it use 2. x265 calls the tool rate-distortion optimized quantization.
Trade-offs
- It costs encoding time. Every candidate is priced with the entropy coder's bit-cost estimates. With
--trellis 2that happens for every mode tried, not just the winner. - It can remove fine texture. When the bit savings outweigh the measured error, the search removes small coefficients, which can reduce fine texture or grain. x264's second
--psy-rdvalue biases the search toward keeping them. - It is only as good as its prices. The costs are estimates and the candidates are restricted, so the chosen levels are the best the search saw, not the best possible.
- No decoder changes are needed. Like the dead zone, this is purely an encoder choice; the bitstream just carries the levels.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
quant_trellis_cabac, the two candidates, reverse order and the shared contexts in 8×8 blocks, the eight states, the comment on the variable-length version, the round-to-nearest start, the psy-trellis bias, trellis inside mode decision, the default, the slow preset, the option help. - x265 at commit 4191822 (GitHub mirror): the default, the slow preset, level 0 against the cost search, zeroing a coefficient group at level 2, the last-position search at levels 1 and 2.
- Scope: the walkthrough describes x264. The x265 row summarizes what each level switches on in its source; the rest of x265's search was not read. How H.264's arithmetic coder signals nonzero positions is taken from x264's code, not from the standard.