Transcript ppt

Geometry Directed Browser for
Personal Photographs
IIIT Hyderabad
Aditya Deshpande, Siddharth Choudhary, P J Narayanan,
Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar
Center for Visual Information Technology
IIIT Hyderabad
Photo-Browsing
• Digital Photography
- No hard copy
- Capture photographs and relive later on display device
• Photo-Browsers are tools to view digital photographs.
E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc.
IIIT Hyderabad
• Photo Browsing model has not
evolved much.
We use SfM and other 3D computer vision techniques to provide
intuitive Geometry Directed Photo Browsing.
Related Work
IIIT Hyderabad
• Face Detection & Tagging on Social Networking Sites.
•
[Zhang et al. MM’03], Automatic annotation of family albums.
•
[Davis et al. MM’05], Additional contextual data viz. time of
capture, geo-tag, indoor/outdoor scene, co-occurring faces.
Above techniques only improve photo-browsing experience of social
engagements.
Our Goal
IIIT Hyderabad
• Apart from social engagements, a large chunk of user’s
personal photographs consist of tourist places & monuments.
•
[Snavely et al. IJCV’08, SIGGRAPH’06] (Photosynth)
- CPC Storage, local reconstruction to add new camera’s
•
Choudhary et al., Li et al., Sattler et al., Irschara et al. etc.
- Localize new query images w/o exhaustive search.
We combine SfM-Reconstruction + Localization to provide intuitive
browsing of user photos in 3D space of the monument.
Assumptions
• Our target platform is an off-the-shelf laptop or a desktop.
• User is expected to click around 5-50 photographs for a
particular monument.
• The system should localize these user photographs in a
reasonable time.
IIIT Hyderabad
• The system should provide a smooth visualization / transitions
of all user photos and ~105 points of the monument.
System Design
IIIT Hyderabad
(1) Heavy SfM Reconstruction
done offline in the cloud
(3) User uploads personal photo’s
through a camera / phone
(2) GDBPackage : reconstruction
+ addnl. information downloaded
to local disk
(4) System registers user’s
photos to the point cloud and
provides 3D visualization.
System Block Diagram
GDBPackage
1
Registration
Module
Estimated
Camera’s
IIIT Hyderabad
User Photos
System is divided in two parts :
1. Registration / Localization Module
2. Visualization Module
Visualization
Module
2
Localizing User Photo’s
IIIT Hyderabad
• Trivial if photograph is taken from GPS enabled device
and is geo-tagged!
•
What if no geo-tag information?
•
Two Localization Approaches :
Image based search in a geo-tagged Image Dataset
[Panda et al.] Geo-locate digital heritage site photos.
Using structure information in SfM Dataset
[Irschara et al. CVPR’09], match to nearby similar images.
[Li et al. ECCV’10], visibility prioritized 3D-2D matches.
[Sattler et al. ICCV’11, ECCV’12], visual words to find 2D-3D
matches.
Localization - Choudhary et al.
• [Choudhary et al. ECCV’12]
- Triangulate a seed point in the user photograph.
- Further 3D-2D search is guided by visibility probabilities.
- Find ~20 independent matches.
- Use RANSAC to estimate camera parameters.
Up Vector
IIIT Hyderabad
View Direction
Probability Guided 3D-2D correspondence
3D Position
Advantages of Localization Method
• Data for Localization is stored in GDBPackage :
(1) Cover Set
(2) Visibility Matrix
(3) Bi-Partite Visibility Graph
• CPC images need not be stored, data requirements are minimal.
IIIT Hyderabad
• The method is fast and localizes images at the rate of 1sec/photo.
Non-Localizable Photographs
IIIT Hyderabad
• In some cases the images lack sufficient monument geometry
for localization to work :
- Occluded by people.
- Noisy images of nearby scenery/smaller monuments.
- Zoomed in images of smaller monument structures etc.
Zoomed In View of
Small Structure
(Pantheon Dataset)
Completely Occluded
by People
(Colosseum Dataset)
Non-Localizable Photographs
• Photographs have time of capture stored in their EXIF-tags.
• A non-localized image is placed at a position that is weighted
average of its immediate known predecessor and immediate
known successor in time.
IIIT Hyderabad
• Similarly, linear interpolation is also done for the view-direction
vector to get the complete camera pose.
• The above method will not give the exact location, but placing it
in temporal neighborhood suffices for display purposes.
Visualization Module
• 3D Viewer
• Mouse Navigation
• Button Navigation
• Add Screenshot
IIIT Hyderabad
• Delete Path
• Generate PhotoTour
• 2D Viewer
3D Photo Browser :
Geometry Directed Photo-Browsing
• Initial Mode : 3D Model and small preview (thumbnails) of
user photographs.
• Select Mode : Animate to clicked photo and detailed view.
IIIT Hyderabad
• Linear quaternion interpolation of Rotation Matrix for smooth
transitions between images.
• Smooth transitions give a feel of the geometric space of the
monument.
3D Photo Browser :
Generating Custom Photo Tours
• User can save the current viewpoint (“Add Screenshots”)
• Once a set of viewpoints are saved, he can smoothly animate
over viewpoints. (“Generate Photo-Tour / Animate Path”)
• User can delete the viewpoints and generate a new photo-tour.
IIIT Hyderabad
• Photo-Tours are a good way to creatively view personal
photo’s taken at a tourist place.
Results
Monument
# Photos
# Registered
Photos
Reg. Time
(secs per photo)
Colosseum
24
21
1.01
Colosseum
19
14
0.97
Pantheon
35
22
1.13
Stone Chariot
(Hampi)
17
17
1.07
IIIT Hyderabad
(a) Localization Module
(b) Visualization Module
Conclusion and Future Work
• Minimal System
Requirements.
• Intuitive 3D Visualization
of User Photographs.
IIIT Hyderabad
• Pipeline for 3D personal
photo-viewing from
SfM reconstruction.
3D Photo-Viewing & Localization
App
• Port our system to a mobile phone and have a touch/gesture
interface.
Thank you. Questions?
More Results
IIIT Hyderabad
(a) Hampi Dataset (Stone Chariot)
(b) Pantheon Dataset
IIIT Hyderabad
Platform Details
Item
Specification
CPU
Intel ® CORE ™ i5
Clock Speed
2.44GHZ
RAM
4GB
GPU
Intel ® HD Graphics Accelerator