Transcript Concurrent Stereo Matching (CSM)

Stereovision
Image Noise
John Morris
Department of Computer Science,
Tamaki Campus
The University of Auckland
Iolanthe at 14 knots on Cockburn Sound, Perth
Stereo Image Noise Sources
• Signal noise
•
•
•
Electromagnetic interference eg cross-talk
Quantum behaviour of electronic devices
eg resistor shot-noise
Quantization: digitization of real-valued signals
• Geometric sources
•
•
•
Discrete pixel sensors with finite area
Occlusions
Perspective distortion
• Opto-Electronic sources  Single camera sources
•
•
•
•
 Stereo (2-camera) sources
Sensitivity variations between cameras
Different ‘dark noise’ levels
Real lenses
Depth-of-focus
• Optical Surface properties 
Note that we use the term ‘noise’ for all problems!
2
Stereo Image Noise Sources
Optical Surface Properties
 Lambertian scatterers
 A “perfect” scatterer scatters light uniformly in all directions
 Most correspondence algorithms assume perfect
(Lambertian) scatterers
 This means that surface patches will appear with the same intensity –
independent of viewing angle ..
 Simple matching
• Intensities should be the same (perturbed by random noise!)
 Reflectors
3
Stereo Image Noise Sources
Ordering Constraint
 Imaged points appear in the
same order in both images
 Often used to simplify
matching algorithms
 Particularly dynamic
programming
 Violated by ‘poles’ – narrow
objects in front of planes
Scene points and image points in same order
4
Stereo Image Noise Sources
Scene: A B C
D
Ordering Constraint
 Imaged points appear in
the same order in both
images
 Violated by ‘poles’ –
narrow objects in front of
planes
Left: a b d
Right: d b c
5
Electronic Noise
Antennae (Receivers)
 Wires act as antennae for EM waves
 ‘Wire’ includes discrete wires
but also
 Tracks on circuit boards
 Interconnects on chips
Transmitters
 Any wire with a changing current emits EM waves
6
Electronic Noise
Digital circuits  very rapid transitions (switching
events)
High frequency signals
Ideal digital signal
Real digital signal
‘Instaneous’ rise or fall ≡ infinite frequency
 perfect radiator
Crosstalk
‘Fast’ rise or fall
 high frequency
 very good radiator
 One wire is influenced by neighbouring wires
Signal driven into purple wire
EM coupling
Signal picked up on green wire
7
Electronic noise
 Quantum effects
 Resistor ‘shot’ noise
e-
 Resistive element is composed of discrete atoms
 Always in motion for all T > 0oK (absolute zero)
 Moving atoms ‘collide’ with electrons moving to form the
current
Random fluctuations in current
or
 Noise as effective resistance changes
 Similar effects in all current carrying or producing devices
e-
• Transistors
• Capacitors
• Inductors, etc
8
Electronic noise
 Digitization noise
 Analogue signal
 Taking all possible values
• At least at a macroscopic level!
 Digital signal
 Represented by a range of integers
• 0 .. 255 (8 bit signal)
• 0 .. 4095 (12 bit signal)
• -2048 .. 2047 (12 bit signed signal)
 A to D converter
 Decides to which integer value to map a real value
 Discretization
 Values which differed (in real domain) become the same (in integer domain)
9
Opto-electronic noise
Stereo Problem
Cameras have different gain settings
 Amplifiers are not ‘matched’ perfectly
Current
Different slopes
Gains differ
Different offsets
Dark currents differ
Light Intensity
Sensors have different ‘dark current’ characteristics
 All sensors produce some electrons (current) with no light
 Quantum ‘tunneling’ out of the sensor device
10
Electronic Noise
Summary
11
Geometric noise
‘Pixelisation’ of images
 Sensor is divided into discrete regions – pixels
 ‘Edges’ in images don’t conveniently fall onto pixel
boundaries
Blue
object
Real image
has blurred
edges
Red object
12
Geometric noise
Stereo Problem
Occlusions
 Points visible from one
camera only
Points which it is impossible
to match
MR
ML
B
B
Perspective distortion
 Field of view in one camera
differs from other
 Left and right images contain
different numbers of pixels
Impossible to match all pixels
correctly
B Binocularly visible
ML Monocular Left
MR Monocular Right
13
Perspective problem
Case 1 – ‘Fronto-planar’ object
Object with face
•  to image planes
•  to optic axes
• extent, a
In image plane,
AL = AR = az / f
Case 2 – Angled object
AL > AR
Effect exacerbated by
• Increased baseline
• Increased angle of object
•Verged axes,   
14
Optical effects
Real lenses
 Finite depth of field
Real lens
Image
plane
a
 Image is ‘in focus’ over finite region only
• Due to deviations of real (thick) lens from
ideal shape
 Depth of field increased by decreasing
aperture
• Always work with a small aperture!
• f-stop
• Ratio of focal length to aperture
• ‘f-stop’ = f / a
• ‘f’ = 1.4 – aperture wide open
• ‘f’ = 22 – aperture closed down
f
Mechanical
aperture
For accurate work (objects in focus over a wide range)
Use more light to permit narrower aperture
15
Optical effects
Depth of focus
 Finite depth of field
Real lens
Image
plane
a
 Image is ‘in focus’ over finite region only
• Due to deviations of real (thick) lens from
ideal shape
 Depth of field increased by decreasing
aperture
• Always work with a small aperture!
• f-stop
• Ratio of focal length to aperture
• ‘f-stop’ = f / a
• ‘f’ = 1.4 – aperture wide open
• f’ = 22 – aperture closed down
Mechanical
aperture
Depth of field
f=22
f=1.4
For accurate work (objects in focus over a wide range)
Use more light to permit narrower aperture
16
Lambertian Scatterers
‘Perfect scattering’
 Intensity of scattered light is the
same in all directions
Observers at all angles see the
same intensity
Characteristics of a scattering
surface




Optically ‘rough’
Surface roughness ≈ 
 = wavelength of incident light
700-400 nm for visible light
17
Reflectors
Perfect reflector
 100% of incident intensity reflected
 Angle of incidence i = angle of
reflection r
All reflected intensity seen at one
angle
r i
Properties of a reflecting surface
 Optically smooth
 Surface roughness  
18
Real Surfaces
 Reflector
 Most of incident intensity reflected
 Bright ‘spot’ seen over a small angle
 Properties of a reflecting surface
Specular
reflection
r i
 Optically smooth
 Surface roughness  
 Specular reflections are the result of
partly reflecting surfaces
Smooth surface
 Bright spots in images
from
 Water surfaces
 Polished glass, plastic, …
19
Real Surfaces
 Scattering surface
 Reasonably uniform distribution of intensity
with angle
 Intensity of the same patch varies with angle
 Intensity mis-match between L and R images
 Systematic mis-match between views of the
same patch
 One is always less intense than the other
 Shape of reflectance curve varies with
surfaces
 Random mis-match between different
patches
 Reflectance curves can be -dependent
also
 Different views of the same patch appear to
have different colours
20
Real Surfaces
Grating effects
 In extreme cases
 Regular surface variations with
dimensions 
Grating effects
 Spectrum generation
 Small variations in view angle 
large colour variations
 Holograms
Systematic mis-match between
different patches
21
Ordering constraint
 If M appears to the left
of N in the left image,
it should appear to the
left of N in the right
image
 Violated by ‘poles’
22
Stereo Correspondence is an ill-posed problem!
or
There are always multiple solutions!
23
Corresponding signals and possible surface profiles
left scan-line signal
View direction
right - scanline signal
•Single surface
reconstruction
•Extreme disjoint
variant
corresponding
Areas
(possible matches)
24
Effect of Noise
but …
What happens if we use ‘noise-free’ images?
L Image - ‘corridor’ set
Synthetic (ray traced)
 Precise ‘ground truth’
is available
25
Noise-free Image Matching
Mismatch is the matching error
Examine one scan line – line
152pixels shifted by known d
R image
(from ground truth)
 Zero difference with L pixel?
 Even ‘noise-free’ images have
matching errors!
Intensity
Mismatch
IL(x)-IR(x-dx)
Disparity
(from ground truth)
26
Real Image Matching
Tsukuba – line 173
Intensity
?
Mismatch
IL(x)-IR(x-dx)
Occlusion
Disparity
(from ground truth)
27
Noise-free Image Matching
Examine one scan line – line 152
Intensity
Mismatch
IL(x)-IR(x-dx)
Edge effects
Disparity No change in d,
(from ground truth)  no occlusion!
28
Pixel-wise correspondences – ‘Tsukuba’ pair (line 173)
Distribution
of signal
differences
Grey-coded
signal
differences
29
Dealing with the noise
Window-based correlation
Epipolar line
Window in
reference image
Possible windows in
other image
Algorithm
For each pixel in the reference image
For each possible disparity, d
Calculate signal difference between windows
centred on (x,y)L and (x-d,y)R
Choose best matching window
Output disparity
30
Matching in the presence of noise
Correlation algorithms
 Match over a window
 Correlation functions:
SSD (sum squared differences)  (IL(x+j.y+k)-IR(x+i-d,y+k))2
SAD (sum absolute differences)  |IL(x+j.y+k)-IR(x+i-d,y+k)|
Average matching errors
Handles random intensity fluctuations
• Electronic, quantization, … noise
 Doesn’t handle
• Occlusions
• Pixelisation
• Intensity anomalies in one window compared to the other
• Gain and dark noise variations
• Perspective
• Tries to assign all pixels in a window to the same disparity
• Non-Lambertian surfaces & specular effects




31
Improving correlation
Normalized correlation functions
 Normalize pixel intensities to mean within a window
Some tolerance for gain and dark noise changes
 Often not worth the additional computational effort!
 Better: Correct the images at source!
• Align amplifiers, calibrate electronics, …
Other variations of cost function
 Several hundred papers in the literature!
32
Correlation algorithms
Generally poor matching
 Mismatches (small windows) or
 Blurred edges due to large windows
Simple (almost trivial!) code
Adaptive windows improve performance 
33
Correlation-Based Methods
 Matching Performance
 Success of correlation methods depends on whether the image window in
one image exhibits a distinctive structure that occurs rarely in the search
region of the other image.
 How to choose the size of the window, wh?
 Too small a window
• may not capture enough image structure and
• may be too noise sensitive
 many false matches
 Too large a window
• makes matching less sensitive to noise (desired) but also
• decreases precision
(blurs disparity map)
 An adaptive searching window has been proposed
34
Correlation Methods
Input – Ground truth
3x3 window
Too noisy!
7x7 window
Sharp edges
are blurred!
Adaptive window
Sharp edges and
less noise
35
Correlation algorithms
Generally poor matching
 Mismatches (small windows) or
 Blurred edges due to large windows
Simple (almost trivial!) code
for each pixel position,w
for each disparity, d
Cwd = (ILw+j-IRw+j-d)2
Choose min(Cwd) over d  d for position w
 Fast enough?
36
Correlation algorithms
 Not fast enough for small images
 For nn image, d disparity values and hw window – O (n2dhw)
 For small (300x300) images and low depth accuracy (5%  d = 20 )
 If n = 3x102, d = 20, h=w=9, then t(n) = c  81  2 x 10  9 x 104 ≈ 1.5
x 107
 For a 3GHz processor, cycles / pixel ≈ 3  109 / 1.5 x 107 ≈ 200
 ??? Enough to






Compute indices,
Fetch two pixels,
Subtract
Square or absolute difference
Find minimum from d costs
Repeat for adaptive window?
but ……
37
Correlation algorithms
 Moderate resolution images
 For moderate (1Mpixel) images and reasonable depth accuracy (1%  d
= 100 )
 If n = 103, d = 100, h=w=9, then t(n) = c  81  102  106 ≈ 1010
 For a 3GHz processor, cycles / pixel ≈ 3  109 / 1010 ≈ 0.3
 Inherently parallel Cwd = (ILw+j-IRw+j-d)2
 Hardware acceleration needed!!
 MMX (Intel Graphics Processing Extensions) helps
 Optimized for simple arithmetic on pixels
but
 MMX pipeline needs to be filled efficiently
 Max 16 operations in parallel
 Excellent for 4  4 windows 
 Usually 9  9 needed!
38
Correlation algorithms
Easy to implement in hardware
 Match for each disparity at the same time
 Add minimum circuit
 Simple circuit ..
 But O (hw) computations for each disparity
Relatively large circuitry
39
SAD
Algorithm
Block
Diagram
•
Note:
High ‘fan-in’ for
SAD calculator
block
Strains FPGA
routing resources
Large buffers
needed for
scanlines
Consequence of
window size
40
Correlation methods - summary
 Simplest code
 Poorest performance
 Adaptive windows don’t help much!
 Medium speed
 Window size is an important factor
 Simple hardware realization
but
 Expensive in resource use
 Handle random noise only
 Window is just treated as vector of pixels
 No spatial information used
 Occlusions ignored
41
Dynamic Programming Stereo
 Attempts to find the ‘best path’ (sequence of disparity values)
 Can recognize occlusions!
 Averages noise over a scanline
 Essentially local
 Always moves ‘forward’ in a scanline
 Solution generated by backtracking through predecessor array
• Doesn’t adjust values in backtrack
 Uses the ordering constraint
 Readily adapted to
 allow for gain and offset changes
 Perspective distortion
 ‘Stubborn’
 Incorporates a penalty for occlusions
 Tends to make ‘streaks’ in disparity images ..\resources\DP_Gonzalez.pdf
 Can be improved by using neighbouring scan lines
 Requires fewer scan line buffers than correlation window
42
Dynamic Programming Result
Note the horizontal streaks!
It’s like a bureaucrat: once a DP algorithm ‘decides’
to adopt a disparity value – it doesn’t want to change its mind!
Adding inter-scanline constraints (using a neighbouring scanline)
generally improves this!
43
Dynamic Programming Performance
Better matching than correlation methods
 ‘Global’ along scan lines
 Recognizes occlusions
 Uses ordering constraint
 Uses spatial information
Time complexity





O(n2d)
Faster than correlation methods in software
Uses memory (typical of DP algorithms)
For n=103, d=102, t = c 108
c = ~30 cycles / pixel on 3GHz machine
Not enough for real time in software!
44
Dynamic Programming
Basic idea





Given a very large problem
Solve small problems first,
Save solutions to the small problems,
Use them to solve larger problem
Continue until large problem is solved
Dynamic programming uses storage space (for
solutions to the small problems) to gain speed
45
Dynamic programming stereo
Small problems are the costs of a ‘path’ to a certain
pixel
Costs are based on
 Difference between pixel intensities
 Cost for an occlusion
 Sometimes - Discontinuity cost
 Assume that disparity changes (depth changes) are relatively rare
 Alternatively – assume most surfaces are smooth
46
Dynamic programming stereo
Monocular Right Pixels
Left scan line
Start
Right scan line
This path would
indicate all
matches at zero
disparity
Monocular Left pixels
Optimum path marks
matching pixels
Diagonal path – smooth
surface – no change in
disparity
Jumps – changes in
disparity
End
47
47
Dynamic programming stereo
Monocular Left Pixels
Left scan line
Start
Right scan line
Maximum disparity
Monocular Right pixels
DP algorithm calculates
cumulative costs for all
possible paths – using
costs for previous pixel to
estimate costs for the
current pixel
Compute costs
for matches in
this region
End
48
48
Dynamic programming stereo
Monocular Left Pixels
Left scan line
Start
Right scan line
As cost for each pixel at
each disparity is
determined, save the
Monocular Right pixels
predecessor
Use the predecessors
to back track
Need w 
predecessors –
Biggest resource cost!
Predecessor array
stores disparity of best
previous point
Maximum disparity
Compute costs
for matches in
this region
End
49
49
50
Symmetric Dynamic Programming Stereo
Compute costs for
the Cyclopean image
centred between OL
and OR
Use visibility
constraints If disparity changes by
n pixels, must be n MR
or ML points between
the two B points
Only need to save
state in predecessor
array .. not disparity!
51
Symmetric Dynamic Programming Stereo
Conventional Stereo
 Match pixel in L image with pixels to R of it in R image
SDPS
 For pixel position, x, in Cyclopaean image,
match pixels at xL+d/2 with xR-d/2
 Even x positions have even disparity,
odd x position have odd disparity
 Calculate in two phases: odd and even
 Resultant image is 2w pixels wide
 Backtrack fills in extra pixels in B regions
52
SDPS
Cost equations – see paper
Pixel mismatch is absolute difference - |gL-gR|
 Any dissimilarity measure could be used
Occlusion cost is fixed constant
 Results are not very sensitive to the actual value
Predecessor array only stores previous state
 ML, B or MR – 2 bits only
53
SDPS transitions
ML
MR
This pixel, at
disparity d0,
can only have 3
states – ML, B, MR
Arrows indicate
costs used to
evaluate cost to
reach each state.
Visibility constraints
allow only these
transitions.
Note half integral x positions – SDPS
generates 2w points/line
54
System Architecture
L
Camera
R
Camera
FPGA
Interface
LVDS/
CameraLink
Line Buffers
Distortion Removal
Image Alignment
PC
Stereo
Matching
Corrected
Images
Host
Higher order
Interpretation
Control
Signals
Disparity
Depth
Depth
Map
55
SDPS hardware – Gidel FPGA card
56
SDPS hardware – Disparity calculator block
Small and compact –
easily replicated to
provide large disparity
range
57
SDPS hardware – proposed system
58
FPGA hardware
Field Programmable Gate Arrays
 Soft hardware
 ‘Programmed’ in a high level language – VHDL
 ‘Compiled’ programs soft loaded onto FPGA
 No cost to change the circuit!
Tradeoffs
 Flexible
 Pipeline or inherent parallelism easily achieved
• Compare GPU
 Slower than state of the art CPUs – n00 MHz vs n GHz
 Memory may be a problem
59
FPGA Hardware
 FPGA = Field Programmable
Gate Array
 ‘Circuit’ is stored in static RAM
cells
 Changed as easily as reloading
a new program
60
FPGA Hardware
 Why is programmability important?
or
 Why not design a custom ASIC?
 Optical systems don’t have the flexibility of a human eye
 Lenses fabricated from rigid materials
 Not possible to make a ‘one system fits all’ system
 Optical configurations must be designed for each application




Field of view
Resolution required
Physical constraints
…
 Processing hardware has to be adapted to the optical configuration
 If we design an ASIC, it will only work for one application!!
61
Other correspondence algorithms
Belief propagation
 DP Stereo extended in all directions
Labels each pixel with a state – its disparity
Minimizes an energy function
 Two terms
 Data term – pixel difference
 Smoothness term – penalizes changes (in disparity)
Iterates until convergence (or cutoff)
 Exchanging messages with neighbouring pixels
62
Belief propagation
Best general performance
Simple BP being overtaken by segmentation-based
algorithms
Memory intensive
 Needs to store costs for each label for each pixel in a region
 Compare last two pixels in scan line for SDPS
Iterates – so slow
63
Algorithm Comparison
Symmetric DP stereo
SCSM
Graph cut
CSM
Symmetric BP
Ground Truth
64
Middlebury Benchmark (MB)
Tsukuba
Algorithms
Sawtooth
all
untex
disc
all
untex
disc
MB-SBPO
0.97 1
0.28 3
5.45 3
0.19 1
0.00 1
2.09 3
CSM
1.15 3
0.80 12
1.86 2
0.98 13
0.62 25
1.69 2
SCSM
0.97 1
0.74 11
1.80 1
0.96 12
0.60 24
1.57 1
Venus
Algorithms
all
Map
untex
disc
0.02
2.77
all
MB-SBPO
0.16
CSM
1.18 12
1.04 10
1.48 3
3.08 34
7.34 18
SCSM
1.15 11
0.91 9
1.38 2
3.05 33
7.03 17
* MB-SBPO
3
3
1
0.16
disc
1
2.20
1
rank among
40
algorithms
– symmetric belief propagation algorithm (best-performing Middlebury
benchmark)
65