Transcript Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi.

Interactive Video Tours
MSR Interactive Visual Media Group
//msrweb/vision/IBR
Rick Szeliski, Sing Bing Kang, Matt Uyttendaele,
Simon Winder, Antonio Criminisi
Being There:
Video-Based Reality Project
A new computer-mediated interactive medium
Goal: Combine the
interactivity of 3D games and virtual worlds
with the
realism and richness of film and video
Being There:
Video-Based Reality Project
How?
Capture real-world video footage with an omnidirectional camera and add interactive
video/graphic elements
Application: Video-Based Walkthroughs
[IEEE CG&A May/June 2004]
Virtual Tours
 Move camera along a rail (“dolly track”) and
play back a 360 video
 Applications:
– Homes and architecture
– Outdoor locations
(tourist destinations)
Surround video acquisition system
OmniCam (six-camera head)
OmniCam
 Built by Point Grey Research (Ladybug)
 Six camera head
 Portable hard drives, fiber-optic link
 Resolution per image: 1024 x 768
 FOV: ~100o x ~80o
 Acquisition speed: 15 fps uncompressed
Acquisition platforms
 Robotic cart
 Wearable
Calibration: Lens distortion
Original (distorted) image
Output (corrected, perspective) image
Calibration: Vignetting
Before correction
After correction
Stitching
(Only 4 of 6 images shown here)
Problem with simple feathering
Multiperspective plane sweep
Without MPPS
With MPPS
Feature tracking for calibration and
ego-motion
Points tracked in all frames in all 6 cameras
Edges tracked in all frames in all 6 cameras
Stabilization
Align
frameto-frame
and
distribute
Dheading
Before motion stabilization
After motion stabilization
Augmented surround video
A movie clip from the
original surround data.
The augmented movie clip.
Surround augmented with video
Demo
A Virtual Home Tour
A second scenario:
Bellevue Botanical Gardens




Head-mounted outdoor scene acquisition
Map control
Localized audio for a richer experience
Complex path navigation
(greater freedom of motion)
Data acquisition
 Walk through the garden with the Ladybug
camera mounted on helmet
 Acquisition done in about 15 minutes
 44 Gb worth of video and audio data
Map control
• The garden map
• Drawing the
acquisition path
• Mapping video frames
positions on map
• Placing sound sources
(background and dynamic)
• Output is a descriptive
XML file
Audio processing and streaming
• Some audio sources
• Current ego-position
All audio sources are:
1. Attenuated (based on
distance from current
position),
2. Mixed together,
3. Streamed to sound card
(DirectX Audio).
Two types of sound:
background and dynamic
Bifurcation handling
 Hypothesize, align,
choose best pair as
bifurcation
 Choice of path
depends on current
heading and
bifurcation point
orientation
Current
heading
Bifurcation handling
 Hypothesize, align,
choose best pair as
bifurcation
 Choice of path
depends on current
heading and
bifurcation point
orientation
Current
heading
Demo 2
A walk in Bellevue Botanical
Garden
3rd scenario: High-End Home
(architectural walkthrough)
 High-Dynamic range (HDR) imaging
 Complex indoor path navigation
High dynamic range imaging
Combined
Overexposed
Underexposed
 Combine multiple exposures to obtain a high
dynamic range (HDR) image.
High dynamic range imaging
All frames
Underexposed frames
Overexposed frames
 HDR imaging is challenging for videos.
 Image registration is necessary.
Demo 3
A High-End Home
Features and statistics





Rapid scene acquisition
Map control
Localized audio for a richer experience
Complex path navigation
High data rates:
– 44 Gb worth of video and audio data in 15
minutes
Current/future work
 Scene Modeling
– Motion estimation and map generation
– More interactive 3D elements
 Compression & selective decoding
– Tiled video representation
 Rendering
– Better use of GPU
 Interactive DVD deployment