encoding.info

How video encoders actually work

A 4 part course presented in short, interactive pages. It starts with a single pixel and ends with every setting in x264, x265 and SVT-AV1 making sense: what each one does, and how they pull against each other.

The encoder map

A map of the course's topics, not an exact diagram of the data flow. Choose a topic to find its lessons.

Units 1 and 2Source frames Unit 8Decisions Unit 9Rate control Unit 10Perceptualoptimization Unit 3Prediction Unit 4Transform andquantization Unit 5Entropy coding Why the encoder contains a decoderReconstruct Unit 6In-loop filters Unit 7Reference frames Unit 12Bitstream Unit 11Measuring quality Unit 13Settings decoded: every knob on the machine Control panel Frames in After Muybridge, The Horse in Motion, 1878 Bits out Compares what went inwith what comes out

The lesson plan

Part 1: Foundations

1Pictures as numbers

What the encoder is given, and what the eye does with it.

  • Pixels, luma and chromaWhat an encoder is usually handed: three planes of integers, one of luma and two of chroma, and labels that say how to turn them into colors. A pixel picker on a color cartoon cat shows any pixel's values both ways, and a second demo takes the picture apart into planes.
  • Chroma subsamplingStoring the chroma planes at lower resolution than luma (4:4:4, 4:2:2, 4:2:0) and what it costs. The cartoon cat with a selector for the amount of reduction and one that applies it to luma instead, as an artificial comparison, with error readouts.
  • Bit depth and bandingHow many levels each sample has, why a smooth gradient turns into bands, and what dither trades for them. A strip of sky with a bits slider and a dither switch, a plot of one row and error readouts.
  • What the eye misses: contrast sensitivity and maskingTwo properties of vision the lossy tools lean on: sensitivity that depends on detail size, and masking. A grating picture to look at and a noise slider that adds the same error to a flat patch and a fur patch.

2Why video compresses

The four kinds of waste a codec removes.

  • Four kinds of redundancyThe four kinds of waste a codec removes (spatial, temporal, statistical, perceptual) and which tools go after each. One frame of a cartoon cat coded pixel by pixel, with a prediction selector, a step slider and a histogram of what is left to code.
  • The hybrid codec loopOne block's trip from source through prediction, residual, coefficients and levels to the reconstruction, and why H.264 the standard is not x264 the encoder. A cartoon cat sent around the loop with three predictions and a step slider.
  • Why the encoder contains a decoderWhy an encoder rebuilds every block as a decoder will and predicts from that copy, and what drift is. A walking cat coded over 25 frames, with a toggle that lets the encoder predict from the source and a plot of the decoder's error.

Part 2: The toolbox

3Prediction

Guessing a block from what's already coded.

  • Intra predictionPredicting a block from the pixels above and to its left, and how many modes H.264 and HEVC offer. Mode selector over five blocks, scored by SAD.
  • Motion estimation and compensationFinding where a block came from in an earlier frame, and what x264 charges for the vector. Move a motion vector by hand over a map of every candidate's SAD.
  • Sub-pixel interpolationMotion vectors that point between pixels, and the filters that make the in-between samples. Shift slider with whole, half and quarter-pixel vectors and two filters.
  • Block partitioningHow finely a frame is cut into prediction blocks, and what H.264 and HEVC allow. Lambda slider that trades block count against approximation error on a quadtree.
  • Multiple reference framesPredicting from frames older than the previous one, and what the reference index costs. Slider for one to four reference frames over three scenes.

4Transform and quantization

How residuals become coefficients and quantized levels.

  • The discrete cosine transform as a change of basisRepresenting a block as a weighted sum of cosine patterns, which concentrates a smooth block's energy in a few coefficients. Slider that rebuilds a block from its first few coefficients.
  • Quantization and the quantization parameter (QP)Replacing each coefficient with an integer level, and how QP sets the step size. QP slider that quantizes a coefficient and a picture patch.
  • Choosing a transform sizeWhy codecs offer several transform sizes and how an encoder picks one per block. Size slider over a smooth patch, an edged patch and a noise patch.
  • Dead zone and roundingThe band of coefficient values that quantize to zero, and how a rounding offset below one half widens it. Offset slider over a model coefficient distribution.
  • Quantization matricesA per-frequency weight on the step size, so some frequencies are quantized more coarsely than others. Steepness slider that bends a flat matrix toward x264's jvt table, with source, reconstruction and error panels.

5Entropy coding

Turning symbols into bits.

  • Why some symbols cost less than a bitThe ideal price of a symbol is minus log2 of its probability, so a likely symbol costs less than one bit. Slider for the share of zeros that moves two points along the cost curve.
  • Variable-length codesCodewords of whole bits, and how letting symbols share a codeword gets below one bit each. Row slider and group-size selector that code one row of the cat's zero-or-nonzero flags and show the code table and the bits.
  • Arithmetic codingOne interval narrowed by every symbol, and bits that point into what is left. Slider that stops a six-flag message after any flag and shows the interval, its ideal cost under the assumed probability, and the shortest bit string that would name it if the message ended there.
  • Variable-length versus arithmetic codingThe two kinds of coder side by side on the same flags. Slider and group-size selector that compare a grouped Huffman code's expected length, a working arithmetic coder's count and the entropy.
  • Context modelingSorting values into contexts that each keep an adapting probability estimate. Model selector and adaptation slider that show where the bits go on a map of the picture.
  • Estimating bit cost before codingHow x264 prices a candidate without writing it: exact codeword lengths for one coder, a table of fractional prices for the other. Demo that runs x264's own tables and plots the table estimate against the narrowing of its coder's interval.

6In-loop filters

Cleaning up the reconstruction.

  • Deblocking and boundary strengthHow a deblocking filter tells a blocking artifact from a real edge, and how x264 sets the boundary strength of each edge. Demo that runs x264's filter and tables on the cat, with sliders for QP and for both --deblock offsets together.
  • Sample adaptive offsetThe second in-loop filter of High Efficiency Video Coding: sort pixels into classes by their neighbors or their value, and add one correction per class. Demo that classifies the cat as x265 does and shows each class's mean error and offset.
  • Directional enhancement and loop restoration in AV1AV1's two filters after deblocking, as libaom implements them: a directional filter that limits how far unlike neighbors pull a pixel, and loop restoration. Demo that runs libaom's direction search and constrained filter on the cat, with sliders for the two strengths.
  • Why filtering inside the loop helps predictionWhat changes when a filter runs before the frame is stored as a reference and not after decoding. Demo that codes a moving cat with the deblocking filter off, after the loop and inside it, and counts the levels and the error.

7Frame structure

Organizing prediction over time.

  • Intra, predicted and bidirectional framesThe three frame types, what each may be predicted from, and why frames are coded out of display order. Demo that codes one frame of the walking cat as intra, P and B, with a slider for how far away the references are.
  • Groups of pictures, open and closedKeyframes, the groups of pictures between them, and what an IDR frame promises that an I frame does not. Demo that lays out sixteen frames in display and coded order and shows every frame a chosen frame depends on.
  • Hierarchical B-framesB frames that serve as references for other B frames, and x264's one-level version. Demo that codes nine frames of the walking cat with ordinary B frames, one reference B frame or a full hierarchy, and counts the levels.
  • Scene cuts and lookaheadHow an encoder finds a scene cut before coding it, with x264's test and its bias. Demo that runs the test on 36 frames with two cuts and a fade, with sliders for --scenecut and --min-keyint.

Part 3: How the encoder decides

8Decisions

One criterion applied everywhere.

  • Distortion measuresSAD, SATD and SSD: what each one responds to and the stages where x264 uses them. Error-shape selector that holds SAD fixed while the other two move.
  • Rate-distortion optimizationChoosing modes by minimizing distortion plus lambda times bits. QP slider that changes the winning mode.
  • How lambda follows the quantization parameterThe baseline x264 and x265 lambda formulas, and how they relate to the values the encoders store. Plot of lambda against QP.
  • Mode decision and early terminationHow an encoder narrows many ways of coding a block to one, and the rules that stop the search early. Demo that codes a frame of the cat with and without pruning and counts the candidates coded and the cost given up.
  • Motion search patternsDiamond, hexagon and exhaustive motion search, and where a pattern can stop short. Demo that runs x264's diamond and hexagon steps over the cost map of a moved block of the cat.
  • Trellis quantizationChoosing coefficient levels by a joint search over squared error plus lambda times bits, instead of a rounding rule. Lambda slider that shows which levels the search moves toward zero.
  • Presets as search pruningx264's ten presets read as limits on one search: what each step widens, narrows or turns off. Interactive table of the twenty settings the presets change, taken from x264's preset code.

9Rate control

Choosing QP over time.

  • Constant quantizer, constant rate factor, average bitrate and constant bitrateThe four common rate control modes, by what each aims at and what it lets move. Demo that runs simplified rules adapted from x264's for each mode over a ten-second sequence of quiet, busy and medium frames and plots QP and frame size.
  • The video buffering verifierThe buffer model that bounds how far a stream may exceed its rate, and how x264 raises QP ahead of trouble. Demo with sliders for the maximum rate and buffer size, plotting QP and the buffer level.
  • Two-pass encoding and complexity estimationWhat a first pass measures and how a second pass spends a fixed budget with it. Demo that aims one-pass average bitrate and a two-pass plan at the same target and compares QP, size and how close each lands.
  • Quantizer compression and frame-type offsetsHow far QP follows complexity from frame to frame, and why I, P and B frames get different QPs. Demo with sliders for x264's qcomp, ipratio and pbratio, plotting QP by frame type and the share of bits each type takes.

10Perceptual optimization

Bending the math toward what viewers see.

  • Block energyThe sum of squared deviations from a block's mean, and why its logarithm maps onto QP offsets. Texture contrast slider.
  • Adaptive quantizationPer-block QP offsets that give smooth regions finer quantization than textured ones. Strength slider over 60 illustrative blocks.
  • Adaptive quantization in x264The exact formulas behind aq-mode 1, 2 and 3, following x264's source. Plot of QP offset against energy over an invented frame.
  • Macroblock-tree and coding-unit treeHow x264 lowers the QP of blocks that many later frames are predicted from. Demo that passes value backward through the lookahead for three kinds of block, with sliders for lookahead length, predictability and qcomp.
  • Psychovisual rate-distortion optimizationThe extra cost term that charges for lost texture energy, as x264 computes it. Demo that codes a grainy cat with four rounding choices per block and shows which one wins as the term's weight rises.
  • Grain: retention versus synthesisThree ways to treat film grain: code it, remove it, or remove it and have the decoder add new grain. Demo that compares the three on a grainy cat by levels coded, error and how uneven the sky is.

11Measuring quality

Knowing whether a change helped.

  • Why quality metrics disagreeWhat peak signal-to-noise ratio, structural similarity and Netflix's fused metric each measure, and why they rank the same encodes differently. Demo that damages the cat four ways to nearly the same squared error and shows the structural similarity of each.
  • Rate-distortion curves and Bjøntegaard delta rateWhy two encoders are compared as curves of quality against bits, and how the gap between two curves becomes one percentage. Demo that codes the cat with two encoders at four quantizers and computes the delta rate over the quality they share.
  • Convex hull and per-title encodingHow the best picture size for a bitrate depends on the content, and how a ladder is built from each title's own curves. Demo that codes three versions of the cat at three sizes and picks the best encode for a bit budget.
  • Test design pitfallsThe choices in an encoder comparison that change its conclusion: setup, metric, picture size, pooling and content. Demo that pools the same twenty frame scores two ways and gets two different summaries.

Part 4: Real encoders

12Codec by codec

The same toolbox with more options each generation.

  • From H.264 to High Efficiency Video Coding, AV1 and Versatile Video CodingFour formats that share a basic design, compared by selected block sizes, prediction choices and filters. Demo that draws each format's largest unit, largest transform and a count of intra directions, and counts the units in a frame.
  • Profiles, levels and tiersThe limits a stream declares so a device can check whether its decoder supports them: which tools, how large and fast, how many bits. Demo that finds the lowest level passing a partial check of picture size, frame rate and bitrate against the tables in x264, x265 and libaom.
  • Bitstream structureHow coded data is packed into units with headers, and how a decoder finds where each begins. Demo that wraps a payload as an H.264, HEVC or AV1 unit and shows the bytes, with the escape bytes that keep start codes unique.

13Settings decoded

The end goal: what every setting does and how settings interact.

  • x264 settings mapSixty x264 entries this course explains, arranged by the stage of the encoder each acts on, with the value x264 starts from for each and which ones presets change. Demo that lists a stage's entries with links to their pages.
  • x265 settings mapSixty x265 options by the stage of the encoder each acts on, with x265's defaults, which ones presets change, and which were read only by name. Demo that lists a stage's options, and a table of defaults that differ from x264's.
  • SVT-AV1 and libaom settings mapEleven controls as they are written in x264, x265, aomenc and SVT-AV1: option names, starting values and how each scale reads. Demo that translates one control across the four encoders.
  • Interaction map: which settings fight and which need each otherRules x264 applies on its own when one setting is given: the tools it switches off, the options it ignores and the ones left without effect, each with its condition and its line of source. Demo that lists the rules nineteen settings can trigger, and a table of where each tool enters the rate-distortion cost.
  • Anatomy of a presetWhat the ten preset names stand for in x264 and in x265: how many settings each assigns differently from medium, and which. Demo that lists every assignment that differs between any two presets of either encoder.
  • Recipes: on-demand ladder, low-latency live, high-quality, grainFour jobs and x264 settings to start each from, with the reason for every option. Demo that builds the command line for a scenario from an input frame rate, a maximum rate and a duration, and lists what to check before trusting it.