Sub-pixel interpolation
Objects do not check the pixel grid before they move. A pan of two and a third pixels leaves every whole-pixel motion vector slightly wrong. Sub-pixel interpolation lets a vector point between pixels: the encoder and the decoder compute the in-between samples from the reference frame with the same filter, and the prediction is read from those.
Try a move that misses the grid
Below is one row of pixels from the current frame, after the content has shifted, and the prediction built for it from the reference row. Set the true shift, then choose how finely the vector may point. A positive shift moves the content to the right, and the vector points back to where it was, so its sign is negative. The search keeps the allowed vector with the lowest sum of absolute differences.
- In this example, each step in precision lowers the largest error. Among the shifts the lower plot samples, with the six-tap filter, the largest mean absolute error is 18.12 for whole-pixel vectors, 9.17 for half and 5.19 for quarter. The finer grids contain the coarser ones and the demo charges nothing for a vector, so a finer grid cannot score worse here.
- For this signal and these filters, only a whole-pixel shift is predicted exactly. There the prediction is a copy and the residual is zero. At a half or quarter position the vector is right and the samples are still an estimate: 2.18 is left at a shift of one half and 2.75 at one quarter.
- The filter matters as much as the precision here. At a shift of half a pixel with a half-pixel vector, the six-tap filter leaves 2.18 and the two-tap filter 7.87. This row has fine detail, which is what a plain average of two neighbors flattens.
Two things are separate here: how precisely the vector states the displacement, and how accurately the filter rebuilds samples at that position.
Where the in-between samples come from
No camera recorded them. H.264 and HEVC each define the filters, because the decoder must produce the identical prediction; in these codecs they are not an encoder choice. The luma filters, read from x264 and x265 (High Efficiency Video Coding, HEVC):
H.264 half: 1 -5 20 20 -5 1 / 32 HEVC half: -1 4 -11 40 40 -11 4 -1 / 64 HEVC quarter: -1 4 -10 58 17 -5 1 0 / 64
The quarter row is one of the two quarter positions; the other uses the same weights in reverse order.
Sub-pixel search in x264
x264 filters each reference frame once into three extra planes: half a pixel across, half a pixel down, and both. A quarter-pixel prediction is then the average of two samples from those planes and the original, so trying a quarter-pixel vector costs an average, not a six-tap filter. Chroma is interpolated with straight-line weights; for 4:2:0 video that is at eighths of a chroma pixel.
--subme sets how much refinement follows the whole-pixel search, as a number of rounds that can end early. At 0 there is none and vectors stay on whole pixels. At 1 only the partition finally chosen is refined, with one half-pixel and one quarter-pixel round. At 2 and 3 every search gets one half-pixel round, and the quarter-pixel rounds are left for the partition chosen. From 4 every search gets quarter-pixel rounds as well; the medium preset, 7, sets two rounds of each. Higher values also change how candidates are scored, which is on Distortion measures.
Trade-offs
- The decoder does the interpolation too. A block with a fractional vector needs filtered samples at playback, and a longer filter is more work per sample.
- A finer grid has more candidates to test and can change what a vector costs. A vector's bits depend on its difference from the predicted vector, not on its precision alone. The gain per step shrinks in this demo; whether an eighth would pay for itself depends on content and codec, and is not measured here.
- An interpolated prediction is a filtered one. The two-tap option shows the cost of a filter that smooths too much. A filter cannot restore detail the reference never had.
- Refinement can stop early. x264 refines from the best whole-pixel vector for a limited number of rounds, so the vector it keeps is the best it tested, not the best that exists.
Sources
Read 2026-11-13, at the commits linked.
- x264 at commit c24e06c (GitHub mirror): the six-tap filter and
hpel_filter,mc_luma, which averages two planes for quarter positions,mc_chroma,x264_frame_filter, the refinement rounds per--submelevel, for every search and for the chosen partition, refinement after each search from level 2, refinement of the chosen partition. - x265 at commit 4191822:
g_lumaFilter. - Not checked: the H.264 and HEVC specifications, x265's sub-pixel search, and AV1's filters. The filter weights are as the two encoders implement them. The trade-offs are general reasoning, not measurements.