Transcript Relative Attributes

Relative Attributes
Presenter: Shuai Zheng (Kyle)
Supervised by Philip H.S. Torr
Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)
What is visual attributes?
• Attributes are properties observable in images that have
human-designated names, such as ‘Orange’, ‘striped’, or
‘Furry’.
Learning Binary Attributes
• In PASCAL VOC Challenge, we learn to predict
binary attributes. (e.g., dog? Or not a dog?)
O. Parkhi, A.Vedaldi C.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011.
Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS 2007.
Problems within Binary Attributes
• Given an attribute it is easy to get labelled
data on AMT(Amazon Mechanical Turk).
• But, where do attributes come from? Can we
find a easier way to ask more people rather
than experts to tag the images?
Problems within Binary Attributes
Some tags are binary while some are relative.
Is furry
Legs shorter
than horses’
Has four-legs
Tail longer
than donkeys’
Mule
Has tail
Labeling data
What is relative attributes?
• Relative attribute indicates the
strength of an attribute in an image
with respect to other image rather
than simply predicting the presence of
an attribute.
Advantages of Relative Attributes
• Enhanced human-machine communication
• More informative
• Natural for humans
8
Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each
other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image
Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each
other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image
Learning Relative Attributes
For each attribute
Supervision is
open
Learning Relative Attributes
Learn a scoring function
Image
features
Learned
parameters
that best satisfies constraints:
12
Learning Relative Attributes
Max-margin learning to rank formulation
2
4
6
1
3
5
Based on [Joachims 2002]
Rank Margin
Image
Relative Attribute Score
13
Learning binary attributes v.s.
Learning relative attributes
Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each
other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image
Relative Zero-shot Learning
Training: Images from S seen categories and
Descriptions of U unseen categories
Age: Hugh Clive Scarlett
Smiling:
Jared
Miley
Miley
Jared
Need not use all attributes, or all seen categories
Testing: Categorize image into one of S+U categories
16
Relative Zero-shot Learning
Can predict new classes based on their relationships to
existing classes – without training images
Smiling: Miley
Jared
Smiling
Age: Hugh Clive Scarlett
Jared Miley
S
Clive
Miley
H
J
Age
Infer image category using max-likelihood
17
Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each
other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image
Automatic Relative Image Description
Density
Novel
image
Conventional binary description: not dense
Dense:
Not dense:
19
Automatic Relative Image Description
Density
Novel
image
more dense than
less dense than
20
Automatic Relative Image Description
Density
C C H H H C
Novel
image
F H H
M F
F I F
more dense than Highways, less dense than Forests
21
Contributions
• Propose a model to learn relative attributes
– Allow relating images and categories to each
other
– Learn ranking function for each attribute
• Give two novel applications based on the model
– Zero-shot learning from attribute comparisons
– Automatically generating relative image
Datasets
Outdoor Scene Recognition (OSR) Public Figures Face (PubFig)
[Oliva 2001]
[Kumar 2009]
8 classes, ~2700 images, Gist
6 attributes: open, natural, etc.
8 classes, ~800 images, Gist+color
11 attributes: white, chubby, etc.
Attributes labeled at category level
http://ttic.uchicago.edu/~dparikh/relative.html
23
Baselines
• Zero-shot learning
– Binary attributes:
bear
Direct Attribute Prediction furry
[Lampert 2009]
big
– Relative attributes via
classifier scores
4
+
• Automatic image-description6+
turtle rabbit
1
–
2
–
3
–
5
+
– Binary attributes
24
Relative Zero-shot Learning
Binary
attributes
Rel. att.
(classifier)
Rel.
att.(ranke
r)
An attribute is more discriminative when used relatively
25
Automatic Relative Image Description
Binary (existing):
Relative (ours):
Not natural
More natural than insidecity
Less natural than highway
Not open
Has perspective
More open than street
Less open than coast
Has more perspective than highway
Has less perspective than insidecity
26
Automatic Relative Image Description
18 subjects
Test cases:
10OSR, 20 PubFig
27
Traditional Recognition
Dog
Chimpanzee
Tiger
Tiger
???
28
Attributes-based Recognition
Dog
Furry
White
[Lampert 2009]
[Farhadi 2009]
[Kumar 2009]
[Berg 2010]
[Parikh 2010]
…
Chimpanzee
Black
Big
Zero-shot learning
Describing objects
Face verification
Attribute discovery
Nameable attributes
…
Tiger
Striped
Yellow
Striped
Black
White
Big
Attributes provide a
mode of
communication
between humans and
machines!
29
Conclusions and Future Work
Relative attributes learnt as ranking functions
– Natural and accurate zero-shot learning of novel
concepts by relating them to existing concepts
– Precise image descriptions for human
interpretation
Enhanced human-machine communication
Attributes-based recognition is an interesting
direction for the future object/scenes recognition.
30
Created by Tag clouds
Cheers! – Shuai Zheng (Kyle)
[email protected]