Transcript No Slide Title

Panoramic Stereo Vision
of Cooperative Mobile Robots
for Localizing 3D Moving Objects
Zhigang Zhu, K. Deepak Rajasekar
Allen R. Hanson, Edward M. Riseman
Department of Computer Science
University of Massachusetts at Amherst
[email protected]
http://www.cs.umass.edu/~zhu
Funded by
AFRL/IFTD F30602-97-2-0032 (SAFER)
DARPA/ITO DABT63-99-1-0022 (SDR Multi-Robot)
DARPA/ITO Mobile Autonomous Robot S/W (MARS)
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
1
Outline
• Motivation
• Panoramic Vision Sensor
• Cooperative Panoramic Stereo
• 3D Match Algorithm
• Experimental Results
• Conclusion & Discussion
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
2
1. Motivation
• Task: Human Search and Rescue
–
–
–
–
multiple moving platforms - cooperation
distributed sensors- self calibration
multiple moving objects & rapid response
unknown environments
• Research Issues: Panoramic Stereo
–
–
–
–
–
2015/7/17
panoramic stereo geometry
dynamic calibration of cameras on moving platforms
view planning - adaptive viewpoints and baselines
real-time detection and 3D localization
match primitives and algorithms btw. 2 views with
large perspective distortions
Omnidirectional Vision 2000
UMASS-Amherst
3
2. Panoramic Imaging Sensor
- geometric mathematical model
for image transform & calibration
P1
panoramic annular lens (PAL)
* 40 mm in diameter, C-mount
P
* view: H: 360, V: -15 ~ +20
* single view point (O)
2015/7/17
B
Hyperboloidal
mirror
O
pinhol C
e
Ellipsoidal
mirror
p p1
Omnidirectional Vision 2000
UMASS-Amherst
4
2.1 Cylindrical panoramic un-warping
Two Steps:
(1). Center determination
(2) Distortion rectification
2-order polynomial approximation
Circular to cylindrical transformation
after eliminating radial distortion
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
5
2.2. Virtual cylindrical camera calibration
Find three parameters
by three points:
H
v
- Effective Focal Length
Fv
v0
O
- Horizon Circle
H0
D
Fv
v0
- Camera Height:
H0
2015/7/17
ground plane
Omnidirectional Vision 2000
UMASS-Amherst
6
3. Panoramic Stereo
• Cooperative panoramic stereo is
constructed by cameras on two
moving platforms
•Four Issues:
Target
Image 1
Image 2
(1) Self-Calibration by seeing each other
_
- Baseline B and relative viewpoints
(2) Distance by Triangulation
Baseline
Camera 1
- Match the images of a target
(3) Error Analysis and View Planning
Camera 2
- What is the best view configuration?
(4) Size-Ratio Method
- Alternative way in co-linearity
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
7
3.1. Dynamic Calibration
• Image size of known cylinder determines baseline
• Image bearings determine relative orientation
PAL 2

B  R / sin( )
2
PAL 1
Why cylinder?
- view-invariant
- easy to detect
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
8
An example: self-calibration and triangulation
Pano 1:
Image of the 2nd robot
Images of a person
Pano 2:
Image of the 1st robot
Results: B = 180 cm, D1 = 359 cm, D2 = 208 cm
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
9
3.2. Error Analysis and View Planning
D
D2
D1  1 B  D1 cot(1  2 ) 1 
2
B
sin(1  2 )
• Three error sources
B2
B 

2R
– Calibration error:
• the baseline error is roughly proportional to the square of the
baseline itself
– Matching error: 1  2
• view difference in O1 and O2 will introduce a "matching error"
– Triangulation error
• decreases for long baseline and good viewing geometry, but
meanwhile the calibration error and matching error may increase
• QUESTION:
– For a given camera/target distance,where should the
second camera be placed to minimize triangulation error?
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
10
Best viewpoint and baseline: error map
>45
8m
Optimization problem of
two variables: B and 1
O 2?
Error
level
30
O1
T
20
-8 m
-8 m
8m
Error map when O2 in different locations
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
10
11
Best viewpoint and baseline : three conclusions
(1) Accuracy in distance is not simply proportional to baseline in
the case of dynamic calibration
1
B
68.0
59.0
27.8
0
D1
2m
6m
(11.5R) (34R)
11.5 m
(64R)
2.9 m
2.1 m
1.2 m
D1
0 2m
6m
11.5
(2) The two curves can be used to find the best viewpoints &
baselines for difference distances
(3) There is a relatively large region with errors that are less than
twice the minimum error
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
12
3.3. Size-Ratio method in co-linearity
Object
Camera 1
O1
Width
D1
O2
Camera 2
D2
• Sizes of an object in a pair of images tell the distances
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
13
4. 3D Match Algorithm
• Performed on objects extracted from images
• Four steps:
(1) Moving object detection and tracking
(2) Head detection and localization
(3) Stereo match based on 3D features
(4) Improving match by temporal tracking
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
14
4.1. Object Detection &Tracking
Detection and tracking multiple moving people
by motion analysis and region grouping
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
15
4.2. Head detection and localization
Head location is one of the reliable primitives for 3D match
- usually visible
- easy to detect
- symmetric
>>Further extension:
appearance-based partial match
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
16
4.3. Stereo Match based on 3D features
Why not 2D match?
(1) large perspective distortion
(2) low image resolution
Matching primitives of an object blob :
(1) Intensity of blob (2) bearing of head ->D
(3) width of blob ->W (4) point at top of head -> H
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
17
[3D match algorithm]
(1) Measures for each
assumed match:
- Intensity similarity: ris
Are image intensities
consistent?
- Ray convergence: rrc
Do rays thru. two images
converge in 3D space?
- Width consistency: rwc
Are image widths consistent
with assumed geometry?
- Height consistency: rhc
Are image heights consistent
with assumed geometry?
(3). Match Selection
Image 1
Image 2
O
O
1
2
object 2
object 1
(2). Overall Match “Goodness”
r(i, j)  ris(i, j)rrc(i, j)rwc (i, j)rhc(i, j) [0,1]
•Choose maximum
•remove object images from match hypothesis
•repeat
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
18
4.4. Current work:
Improving match using temporal match consistency
Time T
m(i’,j’) in
Frame t-1
Best temporal
match rt(i,i')
i'
j'
Spatio match
rs(i,j) in frame t
i
Improved measure:
Best temporal
match rt(j,j')
j
m(i, j)  rs (i, j )  rt (i, i' )m(i' , j' )rt ( j' , j)
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
19
5. Experimental Results
• Stationary platforms but dynamic
calibration
• Three processes:
– two clients: people detection & tracking
– the server: 3D match & GUI
Camera 1
Camera 2
Platform1 :
Detection
& Tracking
Platform 2 :
Detection
& Tracking
• Synchronized image capture
– network time
– light signal and camera sync clock
• Tests: people walked on a pre-defined
track
internet
Server:
3D Match & GUI
3D estimates
– one person: performance of localization using
triangulation & size-ratio
– two person: performance of 3D match for
multi-objects
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
20
One -person sequence (256 frames, 5 Hz)
Image 1
• 2D map of 50cm grid
camera1
camera 2
• Walked along a
rectangular path
Co-linearity
Triangulation
• 6 turn around
• Distance estimates
using two methods
Better estimation here, error ~ ±10 cm
Image 2
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
21
two -person sequence (200+ frames )
Image 1
•Two people
camera1
camera 2
- 3D match
•opposite direction
- inaccurate OK
•the same path
• move forward,
- track when meet
turn, meet,
- exclude false
apart...
- consistent !
5% match error
Image 2
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
22
Error case 1: detecting error causes inaccurate size measure
Image 1
camera1
camera 2
Image 1
Match “goodness”
1
2
3
1 0.37 0.43 0.62
2 0.00 0.00 1.00
Image 2
Improvement:
Appearance-based partial match?
Image 2
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
23
Error case 2: “Greedy” Match causes “winner takes all”
Image 1
camera1
camera 2
Image 1
A global optimazation algorithm
(e.g. Dynamic Programming) will fix
this kind of error
Match
“Goodness”
1
2
1 0.00 0.92
2 0.89 1.00
Image 2
Image 2
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
24
6. Conclusion & Discussions
• Conclusion:
– cooperative panoramic stereo : view planning & adaptive
baseline
– dynamic calibration method : moving platforms
– 3D match algorithm: in perspective distortion and low resolution
– real-time implementation: Now 5Hz
• Further work:
– view planning : moving platforms or multiple camera scheduling
– general framework for match and track: Bayesian Network?
– incorporate pan/tilt/zoom cameras with panoramic cameras
• accurate self-calibration
• accurate localization
• human identification by face recognition
2015/7/17
Omnidirectional Vision 2000
UMASS-Amherst
25