Transcript PowerPoint 프레젠테이션 - Carnegie Mellon University

Vision-based 3D Bicycle Tracking using
Deformable Part Model
and Interacting Multiple Model Filter
May 11, 2011
Hyunggi Cho1, Paul Rybski1,2, and Wende Zhang3
1Electrical
and Computer Engineering
School of Engineering
Carnegie Mellon University
2Robotics
Institute
School of Computer Science
Carnegie Mellon University
3The
Electrical and Controls
Integration Lab.
General Motors R&D
Outline
 Motivation and Overview
 Bicycle Detection
 Bicycle Tracking
 Experimental Results
 Conclusion and Future work
Motivation
 Motivation
-
In 2009, 630 bicyclists were killed
and 51,000 were injured in traffic
accidents in the United States*.
-
Bicyclists and pedestrians are the
most vulnerable traffic participants.
-
There is less research on bicyclist
detection and tracking compared to
that of pedestrians.
*http://www.nhtsa.gov
Movie clip: Bicycle messengers in New York City (Youtube)
Sensors on Cars
Source : http://www.tartanracing.org
System Overview

Track bicycles using a single video camera mounted on a vehicle
output : position & velocity
Input : video
System

System block diagram
Bicycle
Detector
Bicycle
Tracker
Bicycle’s
position & velocity
Bicycle Detection – Deformable Part Model HOG Detector

Eight view-based bicycle detection
root filters
coarse resolution
part filters
finer resolution
deformation
models
P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained
Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
DPM HOG Detector – Object hypothesis

Bicycle detection process
P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained
Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
DPM HOG Detector – Performance Analysis

Examples of bicycle detection
Test images from Google image
DPM HOG Detector – Performance Analysis

Precision-Recall Curve (VOC2009 + Ours)


Training Set : 350 positive / 3300 negative
Test Set : : VOC2009 ‘val’ dataset

Terminology
Eight view
Recall (True Positive Rate)
recall 
TP
TP

P (TP  FN )
Precision
precision 
TP
(TP  FP)
TP
True Positive
FN
FP
P
False Negative
AP
Average Precision
( Area Under Curve )
False Positive
Total No. of Positive
Overview of our single bicycle tracking system

Kalman filter-based tracking
2D image space
state space :
x  [x
y x y ]
x
y
Prediction stage
Update stage
Overview of our single bicycle tracking system
 Dynamic system model
Model
xk 1  Fxk  wk
: Dynamic equation
zk  h( xk )  vk
: Measurement equation
x0 ~ N (0,  0 )
: Initial state
wk ~ N (0, Qk )
: Process noise
vk ~ N (0, Rk )
: Measurement noise
 Motion model : constant velocity
 x  0
 y  0
 
 x 0
  
 y 0
0
0
0
0
1
0
0
0
0   x  0
1  y  0

0  x  1
  
0  y  0
xk 1  Fxk  wk
0
0 ~
v
0

1
x
y
Overview of our single bicycle tracking system
 Measurement model : perspective projection
Image
Horizon
height
u   f
v  0
  
 w  0
h1 
0
f
0
uc  
vc   R
1  
|
u a11 xk  a13 yk  a14

w a31 xk  a33 yk  a34
zk  h( xk )  vk
 xk 
 
0
t    
y 
  k 
1
h2 
 a11
a
 21
a31
a12
a22
a32
a13
a23
a33
x 
a14   k 
0
a24   
y 
a34   k 
1
v a21 xk  a23 yk  a24

w a31 xk  a33 yk  a34
R
rotation matrix
t
translation vector
f
focal length
uc , vc
optical center
IMM - Choosing a model set


Model Set I
CA


Constant Velocity
Coordinated Turn


Constant Velocity
Simplified Bicycle with
CV and CY angle
CV
CV
CV
GM

Constant Velocity
Constant Acceleration
Model Set III
GM


Model Set II
CT
SB
IMM - Choosing a model set

Constant Velocity model :
x  [x
 x  0
 y  0
 
 x 0
  
 y 0

Simplified Bicycle model :
0
0
0
0
y x y ]
1
0
0
0
0   x  0
1  y  0





0 x 1
  
0  y  0
0
0 ~
v

0

1
IMM - Performance analysis


IMM Tracking performance

We tested the IMM method on the GM bicycle dataset

Test Set : 6 sequences with a stationary GM test vehicle

Data statistics : Size : 320x240 , FPS : 10~12 , No. of bicycles : 1
Details of six bicycle sequences ( SM vs. IMM )
Seq.
ego-vehicle
bicycle
RMSE(SM)
RMSE(IMM)
‘seq1’
stationary
laterally
0.0183
0.0216
‘seq2’
stationary
longitudinally
6.6207
6.6196
‘seq3’
stationary
randomly
0.1515
0.1443
‘seq4’
moving
laterally
2.3493
2.3860
‘seq5’
moving
longitudinally
7.0884
6.860
‘seq6’
moving
randomly
11.0929
10.6281
IMM - Performance analysis

Sequence 3 case
Multiple bicycle tracking using a Rao-Blackwellized particle filter


Multiple bicycle tracking problem

Single bicycle tracking : We solved this problem.

Data association : Given a measurement, which target produced it, if any ?

Unknown number of targets : How many bicycles are there ?
In our multiple bicycle tracking case
p(r0:t , s0:t | y1:t )  p(r0:t | y1:t , s0:t ) p(s0:t | y1:t )
Kalman filter
rt  [rt ,1    rt ,T ]T
st  {et , ct }
Particle filter
Joint state vector
ct : Data association indicator
et
:Target visibility indicator
Simo Särkkä, Aki Vehtari, and Jouko Lampinen (2007). Rao-Blackwellized Particle Filter for Multiple
Target Tracking. Information Fusion Journal, Volume 8, Issue 1, Pages 2-15
Multiple bicycle tracking using a Rao-Blackwellized particle filter

Particle filter for data association problem


All possible events between two measurements

Only one target can die

yk
is associated with :
(a) Clutter
(b) One of the existing targets
(c) A newborn target
Example
yk 1 and yk
y2
: Target
t
y3
: Measurement
t
t-1
t-2
: Trajectory
y1
t-1
t-2
Experimental Results

Tracking performance

We tested our detection/tracking system on our bicycle dataset

Test Set : A challenging sequence from a moving Boss (so called ‘Free for all’)

Data statistics : Size : 320 x 240 , Frame rate : 13~15 frame per second

Sensor coverage area
15 m
4m
5m
0m
Minimum pixel size
HOG Detector : 32x64
4m
Experimental Results - data collection

US Bicyclists Crash Types – Top 10covering 61% database samples
Rank & Rate
Description
10
Motorist Overtaking-Other
The motorist was overtaking a bicyclis
ts.
(3.9%)
9
(4.3%)
8
(4.4%)
Illustration
Bicyclist Left Turn in front of traffic
The bicyclist made a left turn in front o
f traffic travelling in the same direction
.
Ride Out At Midblock
The bicyclist entered the roadway at a
shoulder or curb midblock location.
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication
No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Rank & Rate
Description
7
Motorist Right Turn
The motorist was making a right turn
and the bicyclist was riding in either
the same or opposing direction.
(4.7%)
6
(5.1%)
5
(5.9%)
4
(6.9%)
Illustration
Ride Out At Residential Driveway
The bicyclist entered the roadway from
a residential driveway or alley.
Motorist Left Turn– Facing Bicyclist
The motorist made a left turn while
facing the approaching bicyclist.
Ride Out At Midblock
The motorist was entering the
roadway from a driveway or alley
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication
No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Rank & Rate
Description
3
Ride Out At Intersection - Other
The crash occurred at an intersection,
signalized or uncontrolled, at which
the bicyclist failed to yield.
(7.1%)
2
(9.3%)
1
(9.7%)
Illustration
Drive Out At Stop Sign
The crash occurred at an intersection
at which the motorist was facing a stop
sign.
Ride Out At Stop Sign
The crash occurred at an intersection
at which the bicyclist was facing a stop
sign or flashing red light.
We categorized the upper scenarios into 4 different classes in
terms of bicycle motion patterns !!!
W.H. Hunter, W.E. Pein, and J.C. Stutts, Bicycle Crash Types: A 1990's Informational Guide, Publication
No. FHWA-RD-96-104, Federal Highway Administration, Washington, DC, April, 1997
Experimental Results - data collection
Experimental Results – Performance analysis

Scenario – Random moving case (‘Free for all’)
2D Bounding box
{view (x coordinate, y coordinate)}
Uncertainty level
Experimental Results – Performance analysis

Scenario – Random moving case with 3D visualization
2D Bounding box
{view (x coordinate, y coordinate)}
Uncertainty level
Summary and Future Work
 Summary

Data collection
- Based on bicycle accident statistics

Detection part
- Applied DPM HOG detector into a multiple bicycle tracking system

Tracking part
- Incorporate Interacting Multiple Model (IMM) algorithm into our multiple
bicycle tracking system to exploit several types of motion models
- RBPF data association algorithm
 Future work

Real-time C++ implementation ( > 10fps)

Integration the system into the perception system of our autonomous
vehicles at CMU
Q&A
Single bicycle tracking using an IMM

Main idea of Interacting Multiple Model (IMM)

True motion of a bicycle cannot be exactly modeled by just one model, only be sufficiently
approximated by using several motion models for representing dynamic driving behaviors
of a target (i.e., maneuverings of a bicycle).

The IMM filter runs several motion models in parallel and estimates a state by computing
a weighted sum of several filter results which are based on different motion models.
Integral HOG Detector - Performance Analysis II
Vision-based Bicycle Detector

Related works
Publication
Sensors
Features
Attention Focusing Stage
Classification stage
Gavrila
IV2004
Stereo
Edge map
Stereo-based depth
Chamfer matching
Texture classification
Papageorgiou
IJCV2000
Monocular
Haar wavelet
Can add motion/stereo
modules for preprocessing
SVM classifier on Haar
wavelet features
Viola & Jones
CVPR2001
Monocular
Haar-like wavelet
NA
AdaBoost
Dalal & Trigg
CVPR2005
Monocular
HOG
NA
Linear SVM on HOG
Zhu & Avidan
CVPR2006
Monocular
Integral HOG
NA
AdaBoost with linear
SVM as a weak classifier
Miko.
ECCV2004
Monocular
SIFT-like
orientation feature
NA
AdaBoost
Wu & Nevatia
ICCV2005
Monocular
Edgelets
NA
AdaBoost with hardcoded mid-level features
Felzenszwalb
CVPR2008
Monocular
HOG
NA
Deformable part model
with LSVM