Adaptive quantization
Adaptive quantization (AQ) adjusts the quantization parameter (QP) block by block across a picture. The common variance-based kind gives smooth regions finer quantization, because errors can be easier to see there, and allows coarser quantization in textured regions, where some errors are less noticeable. The aim is better perceived quality for the bits available.
Why one QP per frame falls short
Using the same QP throughout a frame treats smooth and textured regions alike. But the same-sized error can be more noticeable in a smooth region than in texture.
In a smooth region the error can show as banding or blocking; in grass or foliage it can be much harder to see. This is spatial masking, and variance-based AQ uses it when choosing local QP offsets.
Try changing the strength
Sixty blocks of a frame: sky on top, a tree and grass below. Each block gets a QP offset from its activity score. Move the slider: at 0 every block has the base QP; as strength rises, QP falls in the sky and rises in the grass.
This illustration holds the base QP at 28. Both the QP offsets and the bit estimates use simplified formulas. It shows how stronger AQ can shift the share of bits toward smooth regions; it does not simulate rate control or predict an encoder's bitrate.
Equal QP does not mean equal bits: at strength 0 the 31 textured blocks already take 78% of the modeled bits. At strength 1 the split is 42% smooth to 58% textured, with QP running from 23 to 34.
In a real encoder
- Measure each block's activity before encoding, usually its block energy: how far the pixels stray from the block's mean.
- Convert it to a QP offset. In these variance-based modes, lower-activity blocks generally get lower offsets than higher-activity ones. The mode sets the reference point and the size of the difference; offsets are not necessarily centered on zero.
- Rate control responds. AQ changes how bits are spread across the picture. When a bitrate target is set, rate control adjusts quantization to work toward it. With quality-based encoding such as constant rate factor, changing AQ can also change the total bitrate.
- The bitstream carries the result. The decoder gets what it needs to work out each region's quantizer and does not repeat the activity analysis. In H.264 that is a QP delta per macroblock.
Variants you'll see
| Setting, x264 and x265 | How energy becomes an offset |
|---|---|
aq-mode 1 | A straight line in the logarithm of energy, with a fixed zero point |
aq-mode 2 | A curve whose reference point comes from the frame's own statistics |
aq-mode 3 | Mode 2 plus a bias that lowers QP on low-energy blocks |
aq-mode 4, x265 only | Mode 2, with edge density standing in for energy on blocks that contain edges |
The rows describe broad behavior; the two encoders' constants differ, and so do their defaults (mode 1 in x264, mode 2 in x265). The exact x264 formulas are on Adaptive quantization in x264. The temporal counterpart is macroblock-tree (MB-tree), called coding-unit tree (CU-tree) in x265, which lowers QP on blocks that later frames reference. In both encoders its offset is applied on top of AQ's.
Trade-offs
- Quality metrics can disagree. AQ can improve perceived quality while lowering peak signal-to-noise ratio (PSNR). Its effect on structural similarity (SSIM) depends on content and settings: x264's
--tune psnrturns AQ off, while its--tune ssimselects mode 2. Compare how the picture looks as well as the scores, and account for any difference in bitrate. - Too much strength takes bits from texture. Grass and film grain can blur while the sky stays clean. The useful strength depends on the encoder, mode and content, so compare representative scenes, checking both smooth gradients and fine texture.
- Variance is a crude proxy. A sharp edge on a flat background has high variance but masks poorly, so it can get ringing. x265's mode 4 brings in edge information for this.
- It interacts with psychovisual rate-distortion optimization (psy-RD). AQ changes local QP, while psy-RD changes how coding choices are scored. Their effects interact, so tune them together.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
x264_adaptive_quant_frame, the note that its constants aim for about the same bitrate, the default mode,--tune psnr,--tune ssim, the macroblock QP delta, macroblock-tree applied on top of the AQ offset. - x265 at commit 4191822 (GitHub mirror): the four modes, mode 1, modes 2, 3 and 4, edge density in mode 4, the default mode, coding-unit tree applied on top of the AQ offset.
- Not checked: how codecs other than H.264 carry per-region quantizers; the claims about masking and about what viewers prefer, which are standard vision science; how any particular rate-control mode responds to AQ in practice.