Toward Free-viewpoint Video Synthesis

Shankar Krishnan, Amy Reibman, Vinay Vaishampayan (AT&T Research), Roger Brockett and Alan O'Connor (Harvard University)

What if we could photograph an object or a scene with a special camera, download the captured data to a special algorithm, and then explore that scene or object in a simulation from any viewpoint?

Free viewpoint technology allows a user to see a real scene from an arbitrary point-of-view by combining views captured from a small number of cameras. This is an active area of research both in the computer graphics and the video processing communities. The problem can be approached by model-based or image-based methods. Image based methods trade off a denser set of images for less information about the underlying geometry. Our experimental system uses an image-based visual hull approach. This system consists of several parts which perform the following tasks: camera calibration, foreground/background segmentation, computation of the new depth field, and texture mapping from the reference images to the synthetic view. Some of these steps are illustrated above using a coffee mug as sample data. Our goal is to attain sufficient performance to handle live video sequences of complex scenes.