Localization in indoor environments by querying omnidirectional visual maps using perspective images

Miguel Lourenco, V. Pedro and João P. Barreto ICRA 2012

•

Standard Image-based Indoor Localization (1/3)

How can a robot equipped with standard camera perform indoor localization?

Query image • Establishing correspondences between the query image and a database of geo-referenced images [Cummins08, Chen11, Hansen01] Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 2

Standard Image-based Indoor Localization (2/3)

SIFT descriptors Quantization + td-idf weighting Image Database + Inverted file Querying • Building a detailed database of environments is troublesome Number of Images Storage Time Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 3

Problem Statement

Query image • A complete coverage of the environment can be performed with an para-catadioptric camera • Distortion increases the appearance difference between the images • Our Contribution: A new model-based SIFT method for matching between hybrid imaging systems Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 4

Presentation Outline

• Matching in Hybrid Imaging Systems  Comparison / drawbacks of the standard approaches  Improvements to the SIFT detector and descriptor • Image-based IL using Hybrid Imaging Systems  Comparison of several database description schemes  Comparison of two searching approaches : BoV vs GVP Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 5

Matching in Hybrid Imaging Systems

• Using SIFT [Lowe04] on both para catadioptric and perspective images provides poor matching results [Puig08] 10% inliers • A straightforward solution is to apply SIFT in a virtual camera perspective (VCP) [Schönbein11] • Standard approaches either render a Polar [ Puig08 ] or a Cylindrical panorama [ Krishnan08 ]

Polar Cylinder

Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 6

Implicit cylindrical rectification - cylSIFT

• Render synthetic views require to reconstruct the image signal  Interpolation artifacts severely affect SIFT performance [ Lourenco12 ] • Based on our previous work [Lourenco12] we propose to perform the cylindrical rectification implicitly inside SIFT framework -

cylSIFT

• How does SIFT work ?

  Image salient points detected in a scale space framework SIFT descriptor is computed based on local image gradient Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 7

Implicit cylindrical rectification – cylSIFT detector

• Render the cylinder before applying SIFT adds extra computational time and interpolation artifacts Rectification > 2sec (Matlab ) * • We avoid the reconstruction artifacts by using an adaptive Gaussian filter [Lourenco12] Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 8

Standard vs Adaptive Gaussian smoothing

• • Inherent properties of the standard Gaussian filter  Space invariant filtering  Decouple convolution in X and Y directions Simplification of the adaptive filter • Advantages of the Simplified Adaptive Filter   Isotropic filter that can be decoupled for each image radius A filter bank can be computed offline and loaded into

Implicit cylindrical rectification – cylSIFT description

• Non-linear distortion modifies the local structures in the image and, by consequence, the gradients are affected • • Changes in local gradients of the image deteriorates SIFT descriptor performance Proposed Solution: Compute gradients in the omnidirectional image and implicitly correct them using the Jacobian matrix of the cylindrical mapping function Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 10

Detection and Matching evaluation

Set A Set B Set C Set D • cylSIFT completely avoids interpolation of the image signal  Better repeatability and matching with less computational burden • cylSIFT has similar performance to the VCP approach  The VCP requires a priori knowledge of the view to render to minimize viewpoints changes between the query and VCP image Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 11

•

Standard Image-based Indoor Localization (3/3)

How can a robot use a standard camera for performing indoor localization?

Query image • Compare of two image searching schemes  Standard Bags of Visual Words (BoV)  Geometry preserving Visual Phrases (GVP) [Zhang11] Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 12

BoV: Standard Bags of Visual Words (BoV)

Length: Dictionary Size … • Images are represented as the histogram of words •

Drawback:

Discard the spatial relation between words  Spatial layout can be relevant for disambiguate situations of perceptual aliasing Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 13

GVP: Encoding weak geometric constraints

[Zhang11]

I’ I

A B A B



y



y

' 

y

-3 -2 -1 0 1 2 3 4

A B

-1 0 1 5 

x

 2

x

' 

x

Offset space 3 4 • • Group of Visual Words in a certain layout form a Visual Phrase GVP incorporate geometric constraints at the searching step Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 14

•

Indoor Location Recognition - Experimental Setup

Our database covers 2 teaching buildings of our campus  118 para-catadioptric images  451 perspective images are used to query the database • The environment suffers from high perceptual aliasing Query image ?

Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 15

Retrieval Results

BoV – TOP 1 GVP– TOP 1 Re-ranking - Top 5 • • cylSIFT takes full advantage of its matching capabilities  Interpolation avoidance assure more distinctive descriptors 10% improvement of the localization success when compared with a state-of-the-art approach (Cylinder + GVP) [Chen11] • Indoor recognition systems benefit with the usage of GVP  Robustness against perceptual aliasing (spatial layout matters) Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 16

Take home messages

• Interpolation artifacts affect image retrieval  cylSIFT offers better retrieval performance in hybrid imaging systems at a marginal computational cost when compared to the standard SIFT algorithm • cylSIFT can be useful for other applications than localization  Hybrid fundamental matrix estimation [Puig08] • First work that uses a hybrid imaging systems for image retrieval without the need of rectification Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 17

Thanks for coming

Questions?

Code and dataset releases: http://arthronav.isr.uc.pt/~mlourenco/OmniSearch/ Miguel Lourenço– Institute for Systems and Robotics, Faculty of Science and Technology, University of Coimbra - 16 May 2012 - Slide 18