Quantization matrices
A quantization matrix changes the step size for each transform coefficient according to its frequency. Many matrices use finer steps for low frequencies and coarser steps for high ones. This can preserve broad image structure while allowing more error in fine detail.
step(row, column) = step(QP) × weight(row, column) / 16
Here step(QP) is the step size set by the quantization parameter (QP). In this simplified model a weight of 16 leaves it unchanged; smaller weights give finer quantization and larger ones coarser. Real integer transforms add their own scaling and rounding on top.
Why weight by frequency
With a flat matrix every weight is 16 and the whole block shares one step. A shaped matrix uses finer steps for some frequencies and coarser steps for others. The rows and columns below are transform frequencies, not pixel positions. This is the 8×8 intra matrix from the set x264 calls jvt: the top-left corner is quantized about 2.7 times finer than flat, the bottom-right about 2.6 times coarser.
x264_cqm_jvt8i table. Teal is finer than flat (below 16), coral coarser.Try bending the matrix
The slider bends the matrix from flat (0%) to that table (100%) and past it; anything above 100% exaggerates the weighting. QP is fixed at 26. Compare the reconstruction with the source, and watch the noisy band at the bottom of the patch, which is mostly high-frequency. In the error panel, mid-gray means no change, lighter means brighter than the source and darker means darker.
At 100%, this patch has 138 nonzero coefficients, compared with 245 for the flat matrix, while mean squared error rises from 7.8 to 17. The matrix changes which errors the quantizer allows. This demo does not measure bitrate, so it cannot say whether compression efficiency improves. To test that, compare encoded results at the same bitrate, using both an error measure and your eyes.
In a real encoder
- Pick the matrices once. They are chosen for a whole sequence or picture, not per block. H.264 and High Efficiency Video Coding (HEVC) keep separate ones for intra and inter, for luma and chroma, and for each transform size.
- Make the weights known to the decoder. Depending on the codec, the bitstream supplies the values or selects predefined matrices, and the decoder uses the same weights when it reconstructs coefficients. That is the difference from the dead zone, which the decoder never sees.
- Quantize. Each coefficient is quantized using its weighted step size.
What encoders offer
| Encoder | Default | Controls |
|---|---|---|
| x264 | Flat | --cqm jvt selects the JVT matrix set; --cqmfile supplies custom matrices |
| x265 | Off, which gives flat weighting | --scaling-list off, default or a file. The default set is switched on by a flag without sending its values; a file's values are sent. |
| aomenc (libaom, AV1) | Off | --enable-qm=1, with --qm-min and --qm-max (defaults 5 and 9) bounding the level used |
AV1 does not send matrix values. Its frame header carries a level from 0 to 15 for each plane, choosing among predefined matrices. Level 15 is flat and level 0 the steepest, the opposite direction to this page's slider.
Trade-offs
- Stronger high-frequency weighting can remove grain and fine texture. Those are mostly high-frequency, so they are what a steep matrix tends to quantize away first.
- The weights assume a viewer. How visible a quantization error is depends on the image and on viewing conditions, including display brightness, pixel size and viewing distance. A matrix designed for one set of conditions may be less suitable for another.
- It is one of several tools acting on quantization. A matrix sets the relative step sizes across frequencies. The dead zone controls which small coefficients become zero. Trellis quantization searches among coefficient values using estimated bit cost and distortion.
- A fixed matrix cannot adjust to each image region. A matrix changes quantization across frequencies; adaptive quantization changes it across regions, using measures such as local activity. The two can be used together.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror):
x264_cqm_jvt8i, the weight divided by 16, the flat default, how the lists are written, the option help. - x265 at commit 4191822 (GitHub mirror): the three values of
--scaling-list, the enabled and data-present flags. - libaom at commit 1b7f9ef (a GitHub mirror): the 16 levels and their defaults, which end is flat, matrices off by default, the options, the frame header fields.
- Not checked: the statement about viewing conditions, which is standard vision science and was not taken from a paper read for this page; the H.264 and HEVC standards themselves.