encoding.info
You are hereSourcePredictionTransformEntropy codingFiltersFramesDecisionsRate controlPerceptionQualityBitstreamSettingsEncoder map
Page 1 of 5 in Transform and quantization

The discrete cosine transform as a change of basis

Transform and quantization · concept · added 2026-11-13

You'll want firstThe hybrid codec loop
In Brief:
The discrete cosine transform rewrites a block as a weighted sum of cosine patterns. It compresses nothing by itself, but for many smooth residuals most of the energy lands in a few coefficients, which helps the stages that follow.

The discrete cosine transform (DCT) represents a block as a weighted sum of cosine patterns. In video coding the input is usually a prediction residual: the source block minus its prediction. For many smooth residuals, most of the energy ends up in a few coefficients, which helps the quantization and entropy-coding stages that follow.

Why change basis

Neighboring pixels tend to be alike. Written as pixels, a smooth row is eight numbers of about the same size, each as important as the next. Written as amounts of eight cosine patterns, the row below is one large number, one medium one and six that are nearly zero.

8 pixels in a row The same row as 8 coefficients 96 100 107 116 125 132 137 140 337 −45 −3 0 0 0 0 0 left to right across the block low frequency to high
A real calculation on one invented row of pixels, with an orthonormal 8-point DCT. Coefficient labels are rounded. Bar height shows magnitude, on a different vertical scale in each chart; color and the signed labels show sign (teal positive, coral negative).

How it works

In two dimensions the patterns are products of a horizontal and a vertical cosine, 64 of them for an 8×8 block, running from flat (top left) to a fine checkerboard (bottom right). The top-left coefficient (DC) represents the block's average level; the other 63 describe variation around it. Pick a block and move the slider: it rebuilds the block from only the first coefficients in zigzag order, which runs broadly from low frequencies to high.

6
← the block average onlyall 64: full reconstruction →
Block
Coefficients
Rebuilt
positive negative, stronger color for larger values. Gray squares are coefficients not kept.
Coefficients kept
Detail energy captured
Mean squared error
Simplified demo: the three blocks are invented and the transform is a floating-point orthonormal 8×8 DCT, applied to the block directly with no prediction. "Detail energy" is the sum of the squares of the 63 coefficients other than DC; it is a mathematical measure, not a score of visual quality. This experiment keeps the first coefficients in zigzag order at full precision and sets the rest to zero, to show how the block's energy is spread. Real quantization rounds coefficient values; it does not keep a fixed prefix of the scan. Mean squared error is computed before the displayed gray values are rounded.

In this model the gradient reaches 99% of its detail energy with three coefficients, and the soft edge reaches 98% with ten. In the noise example, keeping 32 coefficients retains about half the detail energy: unlike the smooth examples, its energy is spread across many coefficients. A repeating or directional texture has structure the transform can use.

The steps in a real encoder:

  1. Transform the residual, not the picture. The input is the difference between the source block and its prediction.
  2. Rows, then columns. The two-dimensional transform is a one-dimensional one applied in each direction.
  3. Integer arithmetic. Codecs specify integer approximations of the cosines so that every decoder reconstructs the same values. The core of H.264's 4×4 forward transform, as x264 implements it, uses only additions, subtractions and doublings.
  4. Quantize the coefficients. See Quantization and the quantization parameter (QP).

Trade-offs

Sources

Read 2026-11-13, at the commits linked.