Hierarchical B-frames
An ordinary B frame is predicted from the I or P frames around it and is used by nothing. With several B frames in a row, the ones in the middle are far from both of their references. A hierarchy fixes that by letting some B frames serve as references for the others: code the middle frame first, then predict the frames on each side from it.
Compare the structures
Nine frames of the cat's walk are coded at quantization parameter (QP) 28. Frame 1 is intra, and a P frame closes each run of B frames, predicted from the I or P frame before it. Choose how many B frames sit in a run and which of them are references. In the drawing, left to right is display order. I and P frames are on the top row, and each row below holds B frames predicted from frames on rows above; each B frame hangs from a line that runs between its two references. The Coded order readout gives the order they are coded in. A filled teal box is a B frame that other frames use. The table counts nonzero levels over all nine frames and averages the mean squared error against the source. Its first row is the same nine frames with no B frames at all.
| Structure | Nonzero levels | Mean error |
|---|---|---|
| No B frames | ||
| No reference B | ||
| The middle one | ||
| Every level |
For this cat at QP 28:
- Seven ordinary B frames left more levels than no B frames at all. 3,038 against 2,618. With no motion search, a reference up to seven frames away matched badly.
- Making the middle B frame a reference brought that to 2,489, and a full hierarchy to 2,062. In the full hierarchy the four B frames on the lowest level are each one frame from both references and left 78 to 103 levels apiece.
- All three seven-B-frame structures wait for frame 9. It is coded second in each. What changes is the order after it: 1 9 2 3 4 5 6 7 8 with no reference B frame, 1 9 5 2 3 4 6 7 8 with the middle one, 1 9 5 3 7 2 4 6 8 with every level.
- The cost moves to the frames the others depend on. The P frame eight frames from frame 1 left 488 levels, more than the intra frame's 395.
- Mean error ran from 12.21 to 13.55 across the four rows. This is a comparison at a fixed QP, not at matched quality.
What x264 does
In its normal automatic placement, x264's --b-pyramid is one level deep. When a run has more than one B frame and none of them is a reference already, x264 makes the middle one a reference and leaves the rest ordinary. The option takes none, strict or normal, and normal is the default; strict marks the reference B frame at a lower priority than I and P frames. With the pyramid on, x264 sets its B frame reordering delay to two frames where plain B frames set one; its total input delay is a separate figure that also counts the lookahead and threads. x265 has --b-pyramid on by default too; its structure was not read. The demo's "Every level" is not x264's automatic structure, and no encoder with a deeper hierarchy was read for this page.
Trade-offs
- A longer run waits further ahead; a deeper hierarchy reorders more. A run that ends on a later frame makes the encoder wait for it. At the same run length, reference B frames change the decoding order, and can raise how long references are kept and how many decoded frames wait to be shown: at most 1, 2 and 4 in the three seven-B-frame orders above, if each frame is shown as soon as its turn comes. Those counts are this demo's, not a buffer size from a standard or a delay in time.
- A reference B frame's quality affects the frames predicted from it. It changes their predictions and the residual they need, which can make it worth coding more accurately. Its quantization error need not pass unchanged into them, since they carry residuals of their own. A B frame that is not a reference has none to protect, though its own quality still shows. Encoders usually give each level its own quantizer; the demo does not, and the rate control unit takes that up.
- These level counts do not predict a saving on real video. With no motion search, the demo's predictions are sensitive to distance in time. A real encoder's motion search, reference choices and quantizers can change the comparison; which way was not measured.
Sources
Read 2026-11-15, at the commits linked.
- x264 at commit c24e06c (GitHub mirror): the test for a run of more than one B frame and the middle one made a reference; reference B frames named by the caller, turned back into ordinary ones under none and strict and under normal when the reference frames are too few; its reference priority against an ordinary B frame; the reordering delay and, apart from it, the input delay; the three modes; the default and
--b-pyramid; the option in the command-line list. - x265 at commit 4191822, default only:
--b-pyramid. - Not checked: how x264 orders and stores references with the pyramid on (the differences between strict and normal beyond the priority); x265's, libaom's and SVT-AV1's structures; quantizer offsets by level; decoder buffer limits set by a level, and any standard's rule for how many decoded frames a decoder holds (the counts of waiting frames are this page's own, worked out from the demo's coded orders); the gain on real video.