Image Stitching Shangliang Jiang Kate Harrison

What is image stitching?

What is image stitching?

Introduction • Are you getting the whole picture?

– Compact Camera FOV = 50 x 35 °

Introduction • Are you getting the whole picture?

– Compact Camera FOV = 50 x 35 ° – Human FOV = 200 x 135 °

Introduction • Are you getting the whole picture?

– Compact Camera FOV = 50 x 35 ° – Human FOV = 200 x 135 ° – Panoramic Mosaic = 360 x 180 °

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images • 2D Rotations ( q , f ) – Ordering  matching images

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images • 2D Rotations ( q , f ) – Ordering  matching images

Recognizing Panoramas • 1D Rotations ( q ) – Ordering  matching images • 2D Rotations ( q , f ) – Ordering  matching images

Recognizing Panoramas

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching – SIFT Features – Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

SIFT Features • SIFT features are… – Geometrically invariant to similarity transforms, • some robustness to affine change – Photometrically invariant to affine changes in intensity

Overview • Feature Matching – SIFT Features – Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Nearest Neighbor Matching • Find k nearest neighbors for each feature – k  number of overlapping images (we use k = 4) • Use k-d tree – k-d tree recursively bi-partitions data at mean in the dimension of maximum variance – Approximate nearest neighbors found in O(nlogn)

Overview • Feature Matching – SIFT Features – Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching – Random Sample Consensus (RANSAC) for Homography – Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching – Random Sample Consensus (RANSAC) for Homography – Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

RANSAC for Homography

RANSAC for Homography • SIFT features of two similar images

RANSAC for Homography • SIFT features common to both images

RANSAC for Homography • • • • Select random subset of features (e.g. 6) Compute motion estimate Apply motion estimate to all SIFT features • Compute error: feature pairs not described by motion estimate Repeat many times (e.g. 500) • Keep estimate with best error

RANSAC for Homography

Overview • Feature Matching • Image Matching – Random Sample Consensus (RANSAC) for Homography – Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

Probabilistic model for verification • • Potential problem: • Two images don’t match… • … but RANSAC found a motion estimate Do a quick check to make sure the images do match • MAP for inliers vs. outliers

Finding the panoramas

Finding the panoramas

Finding the panoramas

Overview • Feature Matching • Image Matching – RANSAC for Homography – Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment – Error function • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment – Error function • Image Compositing • Conclusions

Bundle Adjustment • New images initialised with rotation, focal length of best matching image

Bundle Adjustment • New images initialised with rotation, focal length of best matching image

Error function • Sum of squared projection errors – n = #images – I(i) = set of image matches to image i – F(i, j) = set of feature matches between images i,j – r ij k = residual of k th feature match between images i,j • Robust error function

Overview • Feature Matching • Image Matching • Bundle Adjustment – Error function • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Blending

Gain compensation

How do we blend?

Linear blending Multi-band blending

Multi-band Blending • Burt & Adelson 1983 – Blend frequency bands over range  l

2-band Blending Low frequency ( l > 2 pixels) High frequency ( l < 2 pixels)

3-band blending Band 1: high frequencies

3-band blending Band 2: mid-range frequencies

3-band blending Band 3: low frequencies

Panorama straightening Heuristic: people tend to shoot pictures in a certain way

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

Conclusion

Algorithm

AutoStitch program

AutoStitch.net

Open questions • Advanced camera modeling – radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash … • Full 3D case – recognizing 3D objects/scenes in unordered datasets

Credits • Automatic Panoramic Image Stitching Using Invariant Features, 2007 – Matthew Brown and David G. Lowe (Uni. of British Columbia) • Recognising Panoramas, 2003 – Matthew Brown and David G. Lowe (Uni. of British Columbia) – 2003 – Thanks for the slides!

• Image Alignment and Stitching: A Tutorial, 2006 – Richard Szeliski (Microsoft)

Questions?