encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 5 of 5 in Transform and quantization

Quantization matrices

Transform and quantization · concept · added 2026-11-13

In Brief:
A quantization matrix gives each frequency its own step size. Many use finer steps for low frequencies and coarser ones for high, which can keep broad structure while allowing more error in fine detail.

A quantization matrix changes the step size for each transform coefficient according to its frequency. Many matrices use finer steps for low frequencies and coarser steps for high ones. This can preserve broad image structure while allowing more error in fine detail.

step(row, column) = step(QP) × weight(row, column) / 16

Here step(QP) is the step size set by the quantization parameter (QP). In this simplified model a weight of 16 leaves it unchanged; smaller weights give finer quantization and larger ones coarser. Real integer transforms add their own scaling and rounding on top.

Why weight by frequency

With a flat matrix every weight is 16 and the whole block shares one step. A shaped matrix uses finer steps for some frequencies and coarser steps for others. The rows and columns below are transform frequencies, not pixel positions. This is the 8×8 intra matrix from the set x264 calls jvt: the top-left corner is quantized about 2.7 times finer than flat, the bottom-right about 2.6 times coarser.

horizontal frequency → vertical frequency ↓ 6 10 13 16 18 23 25 27 10 11 16 18 23 25 27 29 13 16 18 23 25 27 29 31 16 18 23 25 27 29 31 33 18 23 25 27 29 31 33 36 23 25 27 29 31 33 36 38 25 27 29 31 33 36 38 40 27 29 31 33 36 38 40 42
The values are x264's x264_cqm_jvt8i table. Teal is finer than flat (below 16), coral coarser.

Try bending the matrix

The slider bends the matrix from flat (0%) to that table (100%) and past it; anything above 100% exaggerates the weighting. QP is fixed at 26. Compare the reconstruction with the source, and watch the noisy band at the bottom of the patch, which is mostly high-frequency. In the error panel, mid-gray means no change, lighter means brighter than the source and darker means darker.

100%
← 0%: flat150%: exaggerated →
Weights
Source
Reconstructed
Error, ×4
Lowest-frequency step
Highest-frequency step
Nonzero coefficients
Mean squared error
Simplified demo: the patch is the invented one from the quantization page (a gradient, a soft edge and a band of noise), transformed directly with a floating-point 8×8 transform and plain rounding. 100% is x264's table exactly; other settings scale each weight's distance from 16, with a floor of 2. Weights are shown rounded. The nonzero count is not a bitrate.

At 100%, this patch has 138 nonzero coefficients, compared with 245 for the flat matrix, while mean squared error rises from 7.8 to 17. The matrix changes which errors the quantizer allows. This demo does not measure bitrate, so it cannot say whether compression efficiency improves. To test that, compare encoded results at the same bitrate, using both an error measure and your eyes.

In a real encoder

  1. Pick the matrices once. They are chosen for a whole sequence or picture, not per block. H.264 and High Efficiency Video Coding (HEVC) keep separate ones for intra and inter, for luma and chroma, and for each transform size.
  2. Make the weights known to the decoder. Depending on the codec, the bitstream supplies the values or selects predefined matrices, and the decoder uses the same weights when it reconstructs coefficients. That is the difference from the dead zone, which the decoder never sees.
  3. Quantize. Each coefficient is quantized using its weighted step size.

What encoders offer

EncoderDefaultControls
x264Flat--cqm jvt selects the JVT matrix set; --cqmfile supplies custom matrices
x265Off, which gives flat weighting--scaling-list off, default or a file. The default set is switched on by a flag without sending its values; a file's values are sent.
aomenc (libaom, AV1)Off--enable-qm=1, with --qm-min and --qm-max (defaults 5 and 9) bounding the level used

AV1 does not send matrix values. Its frame header carries a level from 0 to 15 for each plane, choosing among predefined matrices. Level 15 is flat and level 0 the steepest, the opposite direction to this page's slider.

Trade-offs

Sources

Read 2026-11-13, at the commits linked.