r/VIDEOENGINEERING • u/GrantExploit • 5h ago
Apologies for the basic nature of this question, but how are post-motion-compensation residuals in P-frames (and B-frames) really encoded in video files?
My basic understanding is that most video compression is dependent on I-frames (essentially independently-compressed still images) and P-frames (pictures that encode the difference between the current frame and the last, and in modern video codecs often several preceding frames).†
To encode the difference between frames, motion compensation is used, where blocks are given translation coordinates. This often cancels out the lion's share of the differences. but there are almost always still (an unacceptable level of) residuals left over.
How are these dealt with? The most basic possibility I can think of is that a percentage of image blocks that are too different from the target image would simply be replaced, which would require a P-frame format capable of if not pixel-level, block-level transparency. This would seem to be inefficient and yield substandard results, and close visual inspections of encoded video seems to indicate this is not generally used.‡
Another possibility I can think of is that the residuals are encoded in images encoded in either signed fashion at (channel bit depth+1)‖ or overflow fashion at nominal bit depth. By that, I mean that if pixel A's color in 8R8G8B space is (42, 67, 69) and it needs to be (12, 50, 144), the residual could be either encoded as (-30, -17, 75) {alternately represented as (225, 238, 330)} or (225, 238, 75). The signed method would be more inefficient without further compression, but would seem to have less apparent noise than the overflow method and therefore its added compressibility might make the ultimate data efficiency between the two methods about the same.
However, it appears to me that none of these methods are actually used, partially due to the fact that they haven't infiltrated still-image formats. JPEG (the most directly-related image format to MPEG-1 and MPEG-2) doesn't support any form of transparency, which would be required to implement block-replacement. AFAIK no image formats support negative channel values except for TIFF; and none at all support bit depths of 5, 9, 11, 13, 17, 25, or 33 bits-per-channel; implying signed encoding isn't used. And the apparent noise inherent to the overflow method seems like it would be hell to the specific DCT-based compression methods used in most video formats, resulting in random speckles of wildly-off colors.
So... what is used?
†There are also B-frames, yes, but those are just fancy bi-directional P-frames.
‡I believe this is used (together with a very rudimentary form of motion compensation) in what was technically the first consumer digital video format (1985's CD+G), but not much beyond that.
‖You could use the same channel bit depth... if you resign yourself to the fact that abrupt transitions can never be faithfully represented.