Transcript Image Matching and Retrieval by Repetitive Patterns Petr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum Center for Machine Perception, Czech Technical.

Image Matching and Retrieval by Repetitive Patterns
Petr Doubek, Jiri Matas, Michal Perdoch and Ondrej Chum
Center for Machine Perception, Czech Technical University in Prague, Czech Republic
Repeated Patterns Similarity
Abstract
Detection of repetitive patterns in images is a well-established computer vision problem. However, the detected patterns are rarely used
in any application. A method for representing a lattice or line pattern by shift-invariant descriptor of the repeating tile is presented. The
descriptor respects the inherent shift ambiguity of the tile definition and is robust to viewpoint change. Repetitive structure matching is
demonstrated in a retrieval experiment where images of buildings are retrieved solely by repetitive patterns.
Motivation
Outline of the Algorithm
• Multiple occurrences of a local patch pose a problem to one-toone matching algorithms (local matches are ambiguous)
• The presence of repeated local patches is in most cases nonaccidental and therefore very distinctive
• Regular and even non-regular repetitive patterns give a rise to
geometric constraints
Detection
1. Detect repeated elements, find the lattice of the repeated
pattern, rectify the lattice and calculate the mean tile
2. Compute shift invariant tile descriptors
Image Matching and Retrieval
3. For each pair score with the most similar patterns
Nearest neighbours and
neighbourhood sectors
10x10 43px 12x10 27px 6x9 39px
2x6 53px
16x1 15px 7x11 63px
12x10 57px
0.56 0.37 0.07 0.08 0.06 0.42
j
i
where dk and dl are shift invariant tile descriptions and
j
j
j
i
i
i
pk = (pk;1 ; pk;2 ), pl = (pl;1 ; pl;2 ) are pairs of peaks in
17x4 73px
RGB
colour histograms.
0.00
0.01
0.00
0.00
0.03
0.02
Image Retrieval by Repetitive Patterns
• Tested on two datasets with ~230 images in image
retrieval of about 50 buildings
A cluster of repeated
elements
Example: repeated patterns of two images and their similarity
Experiment 1
Experiment 2
• Performance of shift invariant representation
• Detection and matching of repeated patterns tested on image retrieval
• Ground truth for each query Gi = set of images of • Two publicly available datasets PSU-NRT(subset) and Pankrac+Marseille
building i, was manually marked
(http://cmp.felk.cvut.cz/data/repetitive)
• Top three matches for some of the queries
Repetitive Pattern and Lattice Detection
1. Detection of repeated elements. In our implementation
affine covariant regions (MSERs and Hessian Affine)
described by SIFT
2. Agglomerative clustering of SIFTs. Each cluster
hypothesise a repeated pattern
3. For each element in the repeated pattern, find spatial
nearest neighbours in each of the spatial sectors
4. Find dominant vanishing points by Hough transform and
form a 2D lattice
5. Rectify lattice and divide pattern into tiles
• For a pair of images i,j with sets of detected repeated patterns
Ci and Cj similarity sk,l of two patterns k,l is computed as
Dataset
Query
Top three best matches
.
A lattice from two vanishing
points (corresponding to red
and green directions)
Shift-Invariant Tile Representation
• Each repeated pattern is represented by an “average” appearance of a tile – a mean tile(a more complex representation possible)
• After discovering lattice of a repeated pattern an intrinsic shift ambiguity remains
• We propose two shift-invariant representations:
• the magnitude of Fourier coefficients of a tile
• “zero-phase” normalization: tile shifted so that phase of the first harmonic equals zero
Original image
Detected lattice
Detected tiles
Zero-phased tiles
Conclusions
• Image retrieval can benefit from repeated patterns if
they are detected and handled properly
• Proposed approach is able to detect 1D and 2D
lattices under affine transformation
• Shift invariant descriptors addresses tile ambiguity
• We have shown retrieval based solely on repeated
patterns, however it can be combined with standard
bag-of-words retrieval approaches
References
• T. Tuytelaars, A. Turina, and L. Van Gool, “Non-combinatorial
detection of regular repetitions under perspective skew”,
PAMI, vol.25, no.4, pp. 418-432, April 2003
• P. Doubek, J. Matas, M. Perdoch and O. Chum, “Detection of
2D lattice patterns of repetitive elements and their use for
image retrieval”, technical report, CTU-CMP-2009-16, 2009
• T. K. Leung and J. Malik, “Detecting, localizing and grouping
repeated scene elements from an image”, ECCV, 1996, pp. 546555
The authors were supported by Czech Science Foundation Project 102/07/1317 and by
EC project FP7-ICT-247022 MASH.