Transcript slides - Clemson University

Forward-Backward Correlation
for Template-Based Tracking
Xiao Wang
ECE Dept.
Clemson University
Introduction
 Object tracking: An important computer vision problem






Security and surveillance
Medical therapy
Retail space instrumentation
Video abstraction
Traffic Management
Video editing
 Template-Based Tracking


A classic technique
Idea of template-based tracker
Related Work
 Jepson et al. Robust Online Appearance Models for Visual
Tracking, CVPR 2001
 Ho et al. Visual Tracking Using Learned Subspaces, CVPR
2004
 Davis et al. Tracking Rigid Motion using a Compact-Structure
Constraint, ICCV 1999
 Avidan et al. Ensemble Tracking, CVPR 2005
Overview of the Approach
Forward
Correlation
Module
Next Frame
No
Gradient Module
Update Template
Textured
Background?
Yes
Backward
Correlation Module
Template-Based Tracking
 Template Selection: first frame vs. previous frame
 Motion Model:
 Similarity transformation
scaling
displacement
Template-Based Tracking

Cross Correlation: SSD
displacement
reference image
search image
Template-Based Tracking
 Similarity measure: s(Δx, Δy)
 Correlation Coefficient: c(Δx, Δy)
Mean of template
Mean of image region
Forward Correlation
 Forward Correlation:
 Reference frame: previous frame
 Goal: find transformation vector (dx, dy, α)
 Approach: cross-correlation
Put into
correlation
coefficient
framework
Template Update:
Forward Correlation
 Drifting Problem:



Forward correlation
approximates rotation
with translation.
Forward correlation
does not check the
reliability of the
template.
We need a mechanism
to question the
assumption of forward
correlation.
Out-of-plane rotation
Previous frame
Current Frame
Backward Correlation
 Consider our problem as motion segmentation
 Goal of motion segmentation
 Why is motion segmentation of video sequences difficult?
 Under-constrained
 Occlusion & Disocclusion
 Image noise
 A two-step procedure:
 Determine the motion vectors associate with each pixel or feature
point.
 Group pixels or feature points that perform common motion.
Backward Correlation
 Kanade-Lucas-Tomasi (KLT) feature tracker
 Idea: minimize the dissimilarity of feature windows in
two images
 Assumption: mutual correspondence
Backward Correlation
 Now consider the dissimilarity under the template window.
 Decompose the template window into 2 partitions:
foreground
background
Rewrite dissimilarity as:
low
high
Backward Correlation
 Background is moving at a different velocity than the foreground.
 Foreground pixels have similar velocity and generate low SSD error.
 Correlation between background pixels using foreground velocity
generates high SSD error.
 Goal: group foreground pixels which are moving at similar velocities
Reference frame I(x)
Current image J(x)
Difference image
D(x)=[I(x)-J(x+d)]2
Backward Correlation
 Formulations for backward correlation
Set of
template
candidates
Correlation
coefficient
(likelihood)
Untextured Backgrounds
 Limitation of backward correlation:
 Fails if background has little texture.
 Why? --- Examine the assumption.
 Backward correlation has no reason to prefer the
foreground to the background which is untextured.
low
Also low if untextured
Untextured Backgrounds
 Likelihood of backward correlation: textured vs. untextured
Foreground
Textured background
Template
containing
background
pixels
Untextured background
Gradient Module
 Motivation:
 Seek a module focusing on the boundary of the target being tracked.
 An edge-based segmentation problem.
 Prior information: an ellipse model.
 Gradient Module:
Unit vector normal
at pixel i
Intensity gradient
Combining Modules
 Gradient module and backward correlation module
have orthogonal failure modes.
 Textured or Untextured?

Use sum of the gradient magnitude of the
neighborhood region.
 Combination of forward correlation module and
backward correlation module is straightforward.
 Combination of forward correlation module and
gradient module requires the normalization of the
matching scores.
Combining Modules
 Normalize the matching score (likelihood):
Finial state is decided by:
Adaptive Scale
 Vary the scale by ± 10 percent during search process.
 Filter the result to avoid oversensitive scale adaptation.

Comaniciu et al. Kernel-based object tracking, TPAMI 2003
Size of the best state
given by the alg.
Size of the object in the
previous frame
Experimental Results:
Cluttered Background
 Traditional template-based tracker slides off target:
Experimental Results:
Cluttered Background
 Our algorithm remains locked onto target:
Experimental Results:
Cluttered Background

Tracking error plot:
Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed)
Error in x direction
Error in y direction
Experimental Results:
Untextured Background
 Tracking results of traditional template-based tracker:
Experimental Results:
Untextured Background
 Tracking results of our algorithm:
Experimental Results:
Occlusion
Experimental Results:
Tracking a vehicle
Conclusion
 Presented an extension to template-based tracking.
 Achieved robustness to out-of-plane rotation.
 Effective tracking in both textured and untextured
environment.
 Remaining challenges:



Robustness when scale changes.
Use motion discontinuities to improve performance.
Analysis of parameter sensitivity for untextured
backgrounds.
Questions?
Thank You!