Transcript 슬라이드 1

Learning Object Relationships via Graph-based Context Model
Heesoo Myeong, Ju Yong Chang, and Kyoung Mu Lee
Department of EECS, ASRI, Seoul National University, Seoul, Korea
http://cv.snu.ac.kr
PROPOSED METHOD
INTRODUCTION
Similarity graph
Why Context for Scene Understanding?
 Incorporating contextual information is crucial for scene understanding
 Recently, object-object relationships have shown better performance than sceneobject relationships [Rabinovich et al., ICCV07]
Previous Works & Limitations
Building
Building
Building
Building
?
?
Query image
Road
Prefer frequently
appeared objects
Not invariant to
the number of pixels/regions
Graph-based and Exemplar-based
context model
Building
Utilize object relationships adaptively
according to the visual appearance of
objects
Crosswalk
Retrieved
training image
(Exemplar)
Crosswalk
Our context model
Context link
on the similarity graph
 A novel view for representing
object relationships
Learn object-object relationships between
all pairs of regions across whole object
class pairs
Semi-supervised
context link prediction
𝜙𝑖𝑗 (𝑐𝑖 , 𝑐𝑗 )
building
𝑖,𝑗
car
road
car
crosswalk
Retrieved training images
(Exemplars)
85.4
EXPERIMENTS
?
85.2
87.5
building
car
Quantitative Results on Standard Datasets
road
car
car
?
 Jain et al. dataset (Jain et al., ECCV10):
Similarity edge
60.7
 250 training images, 100 test images, 19 labels
Test image
66.7
building
 2,488 training images, 200 test images, 33 labels
 K-nearest neighbor similarity graph is constructed among regions from both the
query image and the corresponding retrieved image set
sidewalk
Input image
3. Context Link Description
86.0
90.7
Baseline classifier
SuperParsing
Ours
sky
tree
bison
field
40.5
Per-class recognition rate
42.9
56.4
sky
4. Context Link Prediction
Baseline MRF
1
Decompose context link prediction problem into two independent label propagation
subproblems [Lu and Ip, ECCV10]
water
0.6
sand
0.4
Input image
0
1
(a) SIFT flow dataset
0.8
Label propagation
0.4
0.2
0
 Applying label propagation to context link prediction
, 𝑏𝑢𝑖𝑙𝑑𝑖𝑛𝑔, 𝑐𝑎𝑟 =?
Ground truth
74.7
74.6
99.8
Baseline classifier
SuperParsing
Ours
Conclusion
We have proposed a novel framework for modeling image-dependent contextual
relationships using graph-based context model
0.6
 A novel context link view of contextual knowledge
Our approach
0.8
0.2
,
Ground truth
53.9
Results on Jain et al. Dataset
 Two regions (𝑠𝑖 , 𝑠𝑗 ) labeled with (𝑐𝑎 , 𝑐𝑏 ) can be viewed as a directional 𝑐𝑎 , 𝑐𝑏 -type
link between two node 𝑠𝑖 and 𝑠𝑗 on the similarity graph
Given a set of object classes 𝒞 = {𝑐1 , 𝑐2 , … , 𝑐𝐾 }, we define Q𝑎𝑏 to represent the
(𝑐𝑎 , 𝑐𝑏 )-links within the retrieved images such that
1 if 𝐺(𝑠𝑖 ) = 𝑐𝑎 , 𝐺(𝑠𝑗 ) = 𝑐𝑏 , (𝑠𝑖 , 𝑠𝑗 ) occurred in a single image
[Q𝑎𝑏 ]𝑖𝑗 =
0
otherwise
where 𝐺(𝑠𝑖 ) represents the ground truth class of region 𝑠𝑖
𝐿
car
plant
Table 1: Per-pixel classification rates and (average per-class rates)
2. Graph Construction
76.3
sky
 SIFT Flow dataset (Liu et al., CVPR09):
Label propagation
 Nonparametric context model scalable to large datasets
sky
Where 𝜓𝑖 (𝑐𝑖 ) represents data term and 𝜙𝑖𝑗 (𝑐𝑖 , 𝑐𝑗 ) is learned context scores 𝐿 by our
approach
 The similarity graph naturally
reflects visual similarity
Our Contributions
𝜓𝑖 𝑐𝑖 + 𝜆
𝑖
 For a test image, retrieve T most similar training images using global features
Our Approach
Building
𝒥 c =
1. Image Retrieval
Conventional
context model
Results on SIFT Flow Dataset
 Use Fully connected Markov Random Field (MRF) model:
Building
Object relationships are usually formulated
as co-occurrence statistics or spatial relation
[Rabinovich et al., ICCV07, Gould et al.,
IJCV08, Jain et al., ECCV10]
No appearance info. has been taken into
account during context modeling process
Inference
(b) Jain et al. dataset
Experimental results demonstrate that the proposed context model overcome the
limitation of conventional context models relying on object label agreement and gives
richer appearance-based context information