Mid Project Presentation
Download
Report
Transcript Mid Project Presentation
SCALABLE IMAGE MATCHING
MID PROJECT REPORT
David
Strickland
ENGN 256
Spring 2013
REFERENCE PAPER: INVERTED INDEX COMPRESSION
FOR SCALABLE IMAGE MATCHING
Chen, D.M.; Tsai, S.S.; Chandrasekhar, V.; Takacs, G.;
Vedantham, R.; Grzeszczuk , R.; Girod, B., "Inverted Index
Compression for Scalable Image Matching," Data Compression
Conference (DCC), 2010 , vol., no., pp.525,525, 24 -26 March
2010
VOCABULARY TREE + INVERTED INDEX
The Vocabulary Tree is a tree -structured vector quantizer
constructed by hierarchical k-means clustering of feature
descriptors.
Inverted Index: Each node has two lists
Image IDs
Array of counts
1Image
from Chen et al.
PROCESS RECAP
1.
2.
3.
4.
5.
Detect Features
Extract Feature Locations and Descriptors
Quantize Descriptors into a Vocabulary Tree
Score Database Images using Inverted Index
Pairwise Match on top scoring Images
SCHEDULE I: VT/II IMPLEMENTATION
Week 1: Research Vocabulary Tree / Inverted Index,
Determine which libraries to use
Week 2: Implement Feature Locator/Descriptors
Week 3: Implement Quantization of Descriptors in V T
Week 4: Implement Database scoring scheme using Inverted
Index
Week 5: Milestone: Mid Project Presentation, Combine
Previous parts, Pairwise Match to retrieve a single image
LIBRARY CHOICES
VLFeat
Includes hierarchical integer k means methods
VLFeat is available for MATLAB or C
Also includes SIFT and dense SIFT feature detection
FreeImage
C++ library that handles image input/output
OpenSURF
C++ SURF implementation
OpenCV
Required by OpenSURF
Dirent.h
Provides POSIX bindings for windows C++, useful for file I/O
LANGUAGE CHOICE: MATLAB OR C++
MATLAB
+ I/O is simple
+ Integration is easy
+ VLFeat tutorials are all for the MATLAB bindings
+ Data is easy to handle, array manipulation is simple
- Proprietary, would require MATLAB to use/modify
- Not as fast as C++
C++
+ No license required
+ Extremely fast, as you control everything
- Integration is difficult (different data structure schemes)
- You have to control everything
Language Choice: C++
Builds character
FEATURE DETECTION
SURF
C++: OpenSURF
Requires OpenCV, which handles the image I/O
SURF features used in the paper (Chen et al.)
SIFT
Dense SIFT feature detection
Included in VLFeat library
Same # of features for every image, but large # of features
Analysis/Comparison of DTII results when using SURF features
vs DSIFT features would provide new/useful information
PAIRWISE IMAGE MATCHING
Simple Nearest Neighbor algorithms can be applied to the
features of the two images to calculate the number of
matching images
Whichever image has the highest fraction of matches is the
best match
Methods for this are included in some of the libraries
The Dictionary Tree + Inverted Index scoring method will
produce the similarity scores for all the database images
The highest scoring images can then be scored using pairwise
matching to find the best image match
DICTIONARY TREE CREATION
HIKM – hierarchical k means clustering is done via VLFeat’s
library
All the feature descriptors are used to create hierarchical clusters
Creates the dictionary tree, each node is a cluster center
1Image
from http://www.vlfeat.org/overview/hikm.html
INVERTED INDEX CREATION
Inverted Index
The descriptors of each feature of each image traverse the tree to
build the inverted index
Each leaf node visited adds to the associated image’s array of counts
Each leaf node has its own inverted index
1Image
from Chen et al.
SIMILARIT Y SCORING & MEMORY USAGE
When matching an image, each image i k1 in the database
of N images is given a similarity score
For each node visited by query descriptors the node’s
inverted list of images all have the scores incremented :
Where:
CURRENT PROGRESS
Implemented:
Image I/O, SURF feature detection
Read/Write SURF features to file
HIKM (Dictionary Tree) Creation
Inverted Index Creation
Image Scoring Methods
Pairwise matching for SURF method
Combined everything together for Image Matching via DTII
To Do:
Read/Write HIKM tree to file
Read/Write Inverted Index to file
Compression
Add additional feature types (e.g. dsift) + analysis
CHALLENGES
File I/O needed to be handled manually in C++
Dif ferent libraries format data dif ferently
Very little documentation available for some libraries
Dictionary Tree & Inverted Index creation times are large
Finding SURF features for every image is time consuming
DT + II is very large, memory management is important
Saving Information (variables etc.) to disk is non -trivial
SCHEDULE II: COMPRESSION & ANALYSIS
Week
Week
Week
Week
6:
7:
8:
9:
Inverted Index Image ID storage
DSIFT feature version of the DT+II
Soft Binned Tree, Analysis
Final Project Presentation
INVERTED INDEX COMPRESSION
Encode each inverted index’s Image IDs by consecutive
dif ferences
Inverted index compression techniques can significantly reduce
memory usage by up to 5X without any loss in recognition accuracy
Reorder database to minimize dif ferences
Minimize:
SOFT-BINNED FEATURE DESCRIPTOR
HISTOGRAMS
Classify a feature descriptor to k nearest tree nodes instead
of just nearest tree node
Soft-binned tree gives improvement in classification accuracy
Disadvantage:
Each database feature now appears in k different inverted lists
Results in larger lists
REFERENCES
1 David M. Chen, Sam S. Tsai, Vijay Chandrasekhar, Gabriel
Takacs, Ramakrishna Vedantham, Radek Grzeszczuk, Bernd
Girod, “Inverted Index Compression for Scalable Image
Matching”