Transcript Segmentation of Vehicles in Traffic Video Tun-Yu Chiang Wilson Lau

Segmentation of Vehicles in
Traffic Video
Tun-Yu Chiang
Wilson Lau
Introduction
 Motivation



In CA alone, >400 road monitoring cameras, plans to install more
Reduce video bit-rate by object coding
Collect traffic flow data and/or surveillance e.g. count vehicles
passing on highway, draw attention to abnormal driver behavior
 Two different approaches to segmentation:


Motion Segmentation
Gaussian Mixture Model
Motion Segmentation

Segment regions with a coherent motion
Coherent motion: similar parameters in motion model

Steps:
i.
ii.
Estimate dense optic flow field
Iterate between motion parameter estimation and
region segmentation
Translational model  use motion vectors directly as parameters
iii.
Segmentation by k-means clustering of motion
parameters
Optic Flow Estimation

Optic Flow (or spatio-temporal constraint) equation:
Ix · vx + I y · vy + I t = 0
where Ix , Iy , It are the spatial and temporal derivatives

Problems
i.
Under-constrained: add ‘smoothness constraint’ – assume flow
field constant over 5x5 neighbourhood window
 weighted LS solution
ii.
‘Small’ flow assumption often not valid:
e.g. at 1 pixel/frame, object will take 10 secs (300 frames@30fps)
to move across width of 300 pixels
 multi-scale approach
Multi-scale Optic Flow Estimation
 Iteratively Gaussian filter and
sub-sample by 2 to get ‘pyramid’
of lower resolution images
 Project and interpolate LS
solution from higher level which
then serve as initial estimates for
current level
 Use estimates to ‘pre-warp’ one
frame to satisfy small motion
assumption
 LS solution at each level refines
previous estimates
 Problem: Error propagation
 temporal smoothing essential at
higher levels
Level 2
1 pixel/frame
Level 1
2 pixels/frame
4 pixels/frame
Level 0 (original resolution)
Results:
Optic flow field estimation
Results:
Optical flow field estimation
 Smoothing of motion vectors across motion (object) boundaries due to


Smoothness constraint added (5x5 window) to solve optic flow equation
Further exacerbated by multi-scale approach
 Occlusions, other assumption violations (e.g. constant intensity)
 ‘noisy’ motion estimates
Segmentation
 Extract regions of interest by thresholding magnitude of motion vectors
 For each connected region, perform k-means clustering using feature
vector:
[vx, vy, x, y, R, G, B]
motion
vectors
pixel
coordinates
color
intensities
 Color intensities give information on object boundaries to counter the
smoothing of motion vectors across edges in optic flow estimate
 Remove small, isolated regions
Segmentation Results




Simple translational motion model adequate
Camera motion
Unable to segment car in background
2-pixel border at level 2 of image pyramid (5x5 neighbourhood window)
translates to a 8-pixel border region at full resolution
Segmentation Results
 Unsatisfactory segmentation when optic flow estimate is noisy
 Further work on



Adding temporal continuity constraint for objects
Improving optic flow estimation e.g. Total Least Squares
Assess reliability of each motion vector estimate and incorporate into
segmentation
Gaussian Background Mixture Model
 Per-pixel model
 Each pixel is modeled as
sum of K weighted
Gaussians. K = 3~5
 The weights reflects the
frequency the Gaussian is
identified as part of
background
 Model updated adaptively
with learning rate  and new
observation
X t  X r , t
X g ,t
X b ,t 
T
P( X t )   wk ,t k ,t X t ,  k ,t ,  k ,t 
K
k 1
k ,t  N  k ,t ,  k ,t 
k ,t  k ,t ,r k ,t , g k ,t ,b   k ,t   k2,t  I
T
Segmentation Algorithm
New
observation
Matching and model
updating
foreground
 Update Formula
 Matching Criterion
X t   k ,t


background
2
  

If no match found: pixel is
foreground
If match found: background
is average of high w / 
ranking Gaussians.
Foreground is average of
low ranking Gaussians
Update weights:
wk  1     wk    M k
2
k ,t
Update Gaussian:
Match found:

m,t 1  1     m,t  X t
 m2 ,t 1  1      m2 ,t   X t  m,t  X t  m,t 
T
No Match found:
Replace least possible Gaussian
with new observation.
Segmentation Result 1
• Background: “disappearing” electrical pole, blurring in the trees
• lane marks appear in both foreground/background
Segmentation Result 2
Cleaner background:
beginning of original sequence is purely background, so background
model was built faster.
Segmentation Result 3
Smaller global motion in original sequence:
Cleaner foreground and background.
Parameters matter
  affects how fast the
background model
incorporates new
observation
 K affects how sharp the
detail regions appears
Artifacts: Global Motion



Constant small motion caused by hand-held camera
Blurring of background
Lane marks (vertical motion) and electrical pole (horizontal
motion)
Global Motion Compensation
 We used Phase
Correlation Motion
Estimation


Block-based method
Computationally
inexpensive comparing
to block matching
Segmentation After
Compensation
Segmentation After
Compensation


Corrects artifacts before mentioned
Still have problems: residue disappears slower
even with same learning rate
Q&A
Mixture model fails when …
 Constant repetitive
motion (jittering)
 High contrast between
neighborhood values
(edge regions)
 The object would
appear in both
foreground and
background
Phase Correlation Motion
Estimation
 Use block-based Phase
Correlation Function
(PCF) to estimate
translation vectors.
 1 x    2 x  d 
1  f   2  f  e
T
j 2 d f
1  f  2*  f 
 f  
 e j 2 d
1  f  2  f 
T
f
PCF x   F 1   f    x  d 
Introduction
Our Experiment
 Obtain test data
We shoot our own test sequences at intersection of
Page Mill Rd. and I-280.
 Only translational motions included in the sequences

 Segmentation
Tun-Yu experimented on Gaussian mixture model
 Wilson experimented on motion segmentation
