Applied Motion-based Video Indexing

In movie or video sequence, motion dynamically changes the viewing perspectives of the spectators and creates interesting and intriguing effects that are rarely matched by those commonly experienced in still images. It is often through motion that the content in a video is expressed and the attention of the viewers captivated. Despite its indubitable importance, however, motion continues to be neglected in most content-based analysis (or video indexing). This is due to the view that the motion parameters must be computed before any of the motion-based competence can be accomplished, and yet from the above, we know that this estimation of motion parameters is very ill-posed.

Our basic work in motion research enables us to develop appropriate spatio-temporal representations that are directly relevant to the tasks at hand without going through the ill-posed process of motion computation. Instead of one strict hierarchy, a variety of visual processes, computed in parallel and using motion features of varying amount of complexity, are constructed (Figure below). One example of such modules is the computation of the rate of approach of surround scene points. This can be used to index situations where something is rapidly approaching the camera, indicating a threatening or aggressive situation, e.g. movement through a tight passageway. We argue that through a variety of such motion competencies each having different computational requirement, a multi-faceted usage of motion information for video indexing can be effected.

Our use of motion in video production is also guided by theories that evolved from cinematography. That is, there is an intimate relationship between cinematography and motion-based indexing. Often, motion is used so that the producer’s intentions can be conveyed across to the audience. Thus, indexes can be formed based on re-establishing the intentions of these shots via motion features, often resulting in high level indexes.

Fig 3. A hierarchy of descriptors