Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi.
Download
Report
Transcript Interactive Video Tours MSR Interactive Visual Media Group //msrweb/vision/IBR Rick Szeliski, Sing Bing Kang, Matt Uyttendaele, Simon Winder, Antonio Criminisi.
Interactive Video Tours
MSR Interactive Visual Media Group
//msrweb/vision/IBR
Rick Szeliski, Sing Bing Kang, Matt Uyttendaele,
Simon Winder, Antonio Criminisi
Being There:
Video-Based Reality Project
A new computer-mediated interactive medium
Goal: Combine the
interactivity of 3D games and virtual worlds
with the
realism and richness of film and video
Being There:
Video-Based Reality Project
How?
Capture real-world video footage with an omnidirectional camera and add interactive
video/graphic elements
Application: Video-Based Walkthroughs
[IEEE CG&A May/June 2004]
Virtual Tours
Move camera along a rail (“dolly track”) and
play back a 360 video
Applications:
– Homes and architecture
– Outdoor locations
(tourist destinations)
Surround video acquisition system
OmniCam (six-camera head)
OmniCam
Built by Point Grey Research (Ladybug)
Six camera head
Portable hard drives, fiber-optic link
Resolution per image: 1024 x 768
FOV: ~100o x ~80o
Acquisition speed: 15 fps uncompressed
Acquisition platforms
Robotic cart
Wearable
Calibration: Lens distortion
Original (distorted) image
Output (corrected, perspective) image
Calibration: Vignetting
Before correction
After correction
Stitching
(Only 4 of 6 images shown here)
Problem with simple feathering
Multiperspective plane sweep
Without MPPS
With MPPS
Feature tracking for calibration and
ego-motion
Points tracked in all frames in all 6 cameras
Edges tracked in all frames in all 6 cameras
Stabilization
Align
frameto-frame
and
distribute
Dheading
Before motion stabilization
After motion stabilization
Augmented surround video
A movie clip from the
original surround data.
The augmented movie clip.
Surround augmented with video
Demo
A Virtual Home Tour
A second scenario:
Bellevue Botanical Gardens
Head-mounted outdoor scene acquisition
Map control
Localized audio for a richer experience
Complex path navigation
(greater freedom of motion)
Data acquisition
Walk through the garden with the Ladybug
camera mounted on helmet
Acquisition done in about 15 minutes
44 Gb worth of video and audio data
Map control
• The garden map
• Drawing the
acquisition path
• Mapping video frames
positions on map
• Placing sound sources
(background and dynamic)
• Output is a descriptive
XML file
Audio processing and streaming
• Some audio sources
• Current ego-position
All audio sources are:
1. Attenuated (based on
distance from current
position),
2. Mixed together,
3. Streamed to sound card
(DirectX Audio).
Two types of sound:
background and dynamic
Bifurcation handling
Hypothesize, align,
choose best pair as
bifurcation
Choice of path
depends on current
heading and
bifurcation point
orientation
Current
heading
Bifurcation handling
Hypothesize, align,
choose best pair as
bifurcation
Choice of path
depends on current
heading and
bifurcation point
orientation
Current
heading
Demo 2
A walk in Bellevue Botanical
Garden
3rd scenario: High-End Home
(architectural walkthrough)
High-Dynamic range (HDR) imaging
Complex indoor path navigation
High dynamic range imaging
Combined
Overexposed
Underexposed
Combine multiple exposures to obtain a high
dynamic range (HDR) image.
High dynamic range imaging
All frames
Underexposed frames
Overexposed frames
HDR imaging is challenging for videos.
Image registration is necessary.
Demo 3
A High-End Home
Features and statistics
Rapid scene acquisition
Map control
Localized audio for a richer experience
Complex path navigation
High data rates:
– 44 Gb worth of video and audio data in 15
minutes
Current/future work
Scene Modeling
– Motion estimation and map generation
– More interactive 3D elements
Compression & selective decoding
– Tiled video representation
Rendering
– Better use of GPU
Interactive DVD deployment