Transcript ShadowDraw Real-Time User Guidance for Freehand Drawing

ShadowDraw
Real-Time User Guidance for Freehand Drawing
Yong Jae Lee
U. of Texas at Austin
Larry Zitnick, Michael Cohen
Microsoft Research
Drawing
Subject 1
Subject 2
We need help..
Tracing
• Difficult to find photos
• Limits creativity
Drawing a face
• Learn a set of rules
• Requires significant training
Our Idea: ShadowDraw
ShadowDraw
Creating shadows
Collection of images…
+
=
Shadow
• Shadow gives gist of many images simultaneously
Creating shadows
What if the only input is the user’s drawing?
Drawing
Shadow
• Requires partial matching and dynamic updates
Speed
Real-time updating of shadows is critical
• 0.1 seconds – “feels right”
• 0.5 seconds – “useful”
• 2 seconds – “useless”
Overview
Database (offline)
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Database Image
Edges
For each
sub-window
….
Min Hash
Query time (online)
Verify
User Drawing
For each
sub-window
Histogram
(ID x dx x dy)
Fine Alignment Spatial Scoring
Shadow
Database
Database
Database (offline)
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Sketch, Img_ID, x, y
Database Image
Edges
Edge descriptor:
Position
Orientation
Linear length
[Zitnick, ECCV 2010]
For each
sub-window
….
Min Hash
Inverted file structure:
Efficient (sub-linear) retrieval
Stores image index and offset
Matching
Query time (online)
Verify
User Drawing
For each
sub-window
Histogram
(ID x dx x dy)
Fine Alignment Spatial Scoring
Shadow
First stage: Use inverted file structure to find candidate set
Second stage: Refine alignment and compute blending weights
Fine alignment
Verify
User Drawing
For each
sub-window
Histogram
(ID x dx x dy)
Hough transform (3D)
Fine Alignment Spatial Scoring
Shadow
Break into three 1D problems:
ΔX
ΔY
ΔY
ΔS
ΔX
ΔS (scale)
Blending weights
*
=
+
Pen Strokes
*
Top matches
Shadows are the composite
of many images
Weights
Blending weights
*
=
+
Pen Strokes
*
Top matches
Shadows are the composite
of many images
Weights
Rendering
+
=
Pen Strokes
+
Pen Position
*
Output
Shadow
• Higher contrast near pen position
• Main focus is user’s drawing
Related work
• Interactive drawing interfaces
– Teddy [Igarashi et al., 1999]
– Fluid Sketches [Arvo and Novins, 2000]
– 3D drawing system [Igarashi and Hughes, 2000]
– iCanDraw [Dixon et al., 2010]
• Drawing studies
– Where do people draw lines? [Cole et al., 2008]
User studies
• 30,000 images, 20 categories
• 16 drawers (8 men, 8 women), 8 evaluators
• 5 objects (shoe, face, bicycle, butterfly, rabbit)
User studies
With ShadowDraw
Without ShadowDraw
Good drawers
User studies
With ShadowDraw
Without ShadowDraw
Bad drawers
User studies
With ShadowDraw
Without ShadowDraw
Average drawers
User studies
• Significant improvement for “Average” group
User studies
After training all users improved:
Poor skill
Good skill
• Subject’s personal style is maintained!
Future work
•
•
•
•
Matching against drawings and not photographs
Temporal strokes
Color, shading, etc.
Drawing “priors”
Not future work:
• Changing people’s strokes
Live demo