Video Stabilization with a depth camera

Shuaicheng Liu1         Yinting Wang1,2         Lu yuan3         Jiajun Bu2         Ping Tan1         Jian Sun3

1. National university of Singapore         2. Zhejiang University         3. Microsoft Research Asia

 

Abstract:

Previous video stabilization methods often employ homographies to model transitions between consecutive frames, or require robust long feature tracks. However,the homography model is invalid for scenes with significant depth variations, and feature point tracking is fragile in videos with textureless objects, severe occlusion or camera rotation. To address these challenging cases, we propose to solve video stabilization with an additional depth sensor such as the Kinect camera. Though the depth image is noisy, incomplete and low resolution, it facilitates both camera motion estimation and frame warping, which makes the video stabilization a much well posed problem. The experiments demonstrate the effectiveness of our algorithm.

Download:

Paper                Supplementary material

Datasets: [cube] [boy] [library1] [library2] [foodcore1] [foodcore2] [gym] [Mcdonald] [corridor]

Cube example: Boy example:

 

 

Frame generation pipeline:

We use the color and depth images in (a) to generate the projection in (b) and the motion field in(c). Many pixels aremissing because of the incomplete depth image. Hence, we warp the color image by the ‘content-preserving’ warping in (d) according to the green control points and a regular grid. This warping generate a color image (e) and a motion field (f). We then generate a complete motion field (g) by combining (c) and (f). The final video frame (h) is created by warp the original frame with (g).

Pipeline:

 

 

Results