bilgebag's Journal: MPEG motion vectors (reuse of)

From Richard Russell @ rtforum.co.uk

MPEG2 achieves much of its bit-rate reduction by 'predicting' what the next frame will be and transmitting only the differences between the true frame and this prediction; the better the prediction the less data needs to be sent.

The simplest form of prediction is to assume that the new frame is the same as the previous one. However when objects in the picture are moving, or when the camera itself moves (e.g. a pan) such a prediction is likely to be poor. Therefore the prediction process is helped by transmitting a set of 'motion vectors' to the decoder. Basically these tell the decoder that the best prediction for each pixel in the frame is not (necessarily) the *same* pixel in the previous frame, but a pixel displaced from it vertically and/or horizontally by a specified offset.

The important point in the context of this thread is that the offset which gives the best prediction of the pixel value may not necessarily correspond to the true motion in the picture. To take an example, suppose part of the picture contains a periodic structure such as a fence, ladder or wheel. It's entirely possible that the best prediction of a pixel will come not from the same post/rung/spoke in the previous frame, but from another post/rung/spoke that happens to be similar.

The way the 'motion vectors' are usually generated is to search all (or at least a subset) of the possible offsets and transmit whichever gives the smallest error for that block. That gives optimum compression performance, but if the 'vectors' are then used to predict the positions of objects part way between one frame and the next the results can be poor.

Finally, since the offsets are sent only to the nearest pixel, even if they do correspond to the true motion they aren't really accurate enough to predict a slow-moving object's position part way through a frame.

So using MPEG2 'motion vectors' to assist frame-rate conversion is fraught with problems. How well it works may depend on detailed characteristics of the encoder which otherwise aren't relevant (for example it's likely to work better with an encoder that uses Phase Correlation than with one that uses Block Matching). When it goes wrong, it can go so badly wrong as to make the end result far worse than not doing the motion compensation at all!