Transcript Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization Gabriel Takacs, Vijay Chandrasekhar, Thanos Bismpigiannis, Bernd Girod Stanford University Radek Grzeszczuk, Natasha Gelfand, Wei-Chao Chen,

Outdoors Augmented Reality on
Mobile Phone using Loxel-Based
Visual Feature Organization
Gabriel Takacs, Vijay Chandrasekhar,
Thanos Bismpigiannis, Bernd Girod
Stanford University
Radek Grzeszczuk, Natasha Gelfand,
Wei-Chao Chen, Yingen Xiong, Kari Pulli
Nokia Research Center, Palo Alto
Video Demonstration
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
2
Outline
• System Overview
– Image matching on the cell phone
• Data Organization
– Server groups images by location
• Data Reduction
– Server clusters, prunes and compresses descriptors
• Image Matching Results
– Qualitative and quantitative
• System Timing
– Low latency image matching on the cell phone
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
3
System Overview
GPS
Server
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
4
Image Matching
Query Image
AffineTest
SURF-64
Ratio
Descriptors
Matching
RANSAC
Prefetched Data
Database Images
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
5
Data Organization
Kernel
Loxel
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
6
System Block Diagram
Group Images
by Loxel
Extract
Features
Cluster
Features
Geometric
Consistency Check
Match
Images
Prune
Features
Compress
Descriptors
Loxel-Based
Feature Store
Extract
Features
Compute
Feature Matches
Camera
Image
Geometric
Consistency Check
Feature
Cache
Server
Network
Geo-Tagged
Images
Device
Location
ANN
Display Info for
Top Ranked Image
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
7
Feature Descriptor Clustering
•
•
•
•
Match all images in loxel
Form graph on features
Cut graph into clusters
Create representative meta-features
meta-feature
Example Feature Cluster
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
8
Feature Descriptor Clustering
Images of the same landmark
meta-feature
single feature
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
9
Database Feature Pruning
•
•
•
•
Rank images by number of meta-features
Allocate equal budget for each landmark
Fill budget with meta-features by rank
Fill any remaining budget with single features
4x Reduction
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
10
Feature Descriptor Pruning
Budget: 500
100
200
All
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
11
Feature Compression
• Quantization
– Uniform, equal step-size
– 6 bits per dimension
• Entropy coding
– Huffman tables
– 12 different tables
S dx
S dy
S|dx|
S|dy|
Original
Feature
Compressed
Feature
Quantization
Entropy
Coding
• 64-dimensional SURF
– 256 bytes uncompressed
– 37 bytes with compression
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
12
ZuBuD Dataset (~1000 Images)
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
13
Stanford Dataset (~2500 Images)
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
14
Matching Results
Query
Rank 1
Rank 2
Rank 3
Rank 4
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
15
Matching Performance
True Matches
False Matches
~10 images / loxel
~10 images / loxel
~1000 images / loxel
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
16
Timing Analysis
Nokia N95
332 MHz ARM
64 MB RAM
100 KByte JPEG over 60 Kbps Uplink
Downloads
Upload
Upload
Geometric
Consistency
Feature
Matching
Extract
Features
Extract
Features
All on Phone Extract Features All on Server
on Phone
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
17
Conclusions
• Image matching on mobile phone
• Use loxels to reduce search space
• 27x reduction in data sent to phone
– Clustering
– Pruning
– Compression
• ~3 seconds for image matching on N95
Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization
18
Questions