Transcript Sequential Sfm

Recent work in image-based rendering from
unstructured image collections and remaining
challenges
Sudipta N. Sinha
Microsoft Research, Redmond, USA
Image-based maps
•
http://www.photosynth.net/view.aspx?cid=82e0166f-0367-47a8-abf4-87a075bb347e
Key Steps
• Structure from motion (Sfm)
• Robust depth-map estimation
• Rendering
Recent results
• Structure from motion (Sfm)
A multi-stage linear approach to structure from motion
Sinha, Steedly & Szeliski, RMLE –ECCV workshop 2010
• Robust depth-map estimation
Piecewise planar stereo for image-based rendering
Sinha, Steedly & Szeliski, ICCV 2009
• Image-based navigation
Image-based walkthroughs from incremental and partial scene
reconstructions Kumar, Ahsan, Sinha & Jawahar, BMVC 2010
Sequential Sfm
Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02,
Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Sequential Sfm
Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02,
Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pair
Pose estimation, triangulation
Refinement
Sequential Sfm
Fitzgibbon’98, Pollefeys’98, Nister’01, Schaffalitzky’02,
Vergauwen’06, Snavely’06, Snavely’07, Agarwal’09, Gherardi’10
Initial seed pair
Pose estimation, triangulation
Refinement
Linear multi-stage approach to structure from
Sinha et. al. 2010 (ECCV-RMLE workshop)
motion
Contributions
•
Vanishing point (VP) constraints reduces drift in rotations
–
–
more accurate than [Govindu’04, Martinec’07] for urban scenes.
Faster pairwise matching + geometric verification
• New practical linear structure and translation estimation
–
–
–
–
more stable than the known linear method [Rother’03]
robust to outliers in 2D observations
easy to parallelize
faster than sequential Sfm
– much faster than L∞ - methods
Linear multi-stage approach to structure from
Sinha et. al. 2010 (ECCV-RMLE workshop)
motion
Images
Feature Extraction
Vanishing Point
(VP) Detection
interest pts
VPs
relative rotations
Pair Matching
2 – VP + 2 point RANSAC
VP tracks
relative pose estimates
VP tracks
relative
rotations
Global
Rotation
Estimation
Linear Reconstruction
global camera
orientations
2-view
Reconstruction
Final Bundle Adjustment
Robust
Alignment
Global Scale
& Translation
Estimation
Full Sfm
initialization
Results
Timings
Break-up of Timings
Comparison with sequential Sfm
Comparison with sequential Sfm
STREET sequence
BUNDLER (65 cams, 22K pts)
OURS (65 cams, 52K pts)
before Bundle Adjustment
HALLWAY sequence
BUNDLER (139 cams, 13K pts)
OURS (184 cams, 27K pts)
Piecewise Planar Stereo for image-based rendering
Sinha et. al. ICCV 2009
Feature matching
Graph-cut based
energy minimization
Piecewise Planar Stereo for image-based rendering
Sinha et. al. ICCV 2009
Planar Stereo Results
Piecewise Planar Stereo for image-based rendering
also handle non-planar scenes now ...
Image-based walkthroughs from incremental and
partial scene reconstructions Kumar et. al. BMVC 2010
• Skip global scene reconstruction (Sfm) step,
• Generate several overlapping, partial
reconstructions instead.
• During navigation, jump
Fort sequence (~5800 images)
between local coordinate frames.
• Scales easily, also parallelizable
• Incremental matching & reconstruction
(images appear over time)
Existing issues in unstructured Sfm
• Accuracy vs. Connectedness
• Reliable results from sparse, unstructured imagery
– wide-baseline matching is still difficult
• Representations:
– metric vs. topological reconstructions ? hybrid ?
• Reconstructing Indoors
– Bottlenecks: doorways, corridors.
– fewer features, non-Lambertian surfaces
Dynamic Image-based Maps: Challenges
• Acquisition
– Images vs. video
– Short-term dynamics vs. long-term dynamics
• Need truly incremental Sfm
– Start with scratch but keep going … ?
– Interleaving matching, Sfm and dense stereo
– Hybrid matching (2D—2D , 2D – 3D, 3D – 3D)
Dynamic Image-based Maps: Challenges
• Temporal appearance changes
– Illumination:
• day/night, seasons, weather, lights on/off
• Cyclic, predictable
– Albedo changes
• Store-fronts, ads-billboards,
• irreversible
• Geometric changes:
– temporary vs. permanent
• Mid-level features for higher level recognition
Questions ?