Transcript Section 4 Automated DSM generation Nicolas Paparoditis

Institute for
Photogrammetry and
GeoInformation
Section 4
Automated DSM generation
Image Matching, Surface Reconstruction, and DSM Analysis towards object
extraction
Nicolas Paparoditis
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Automatic Surface reconstruction from VHR imagery: Where are we now?
• Research has been improving in the last years
• Data sources and technologies have been improving
• New tools coming from computer vision and image processing are starting to get integrated
• What is working in production? How close are we to production?
• What is still research at short, medium and long term?
• I will show many case studies from the MATIS laboratory but similar work exists worlwide
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Surface reconstruction from image matching how does it work?
• A surface is generally described by:
• a set of samples (obtained by total station, GPS, photogrammetry)
• a function to interpolate in between the samples (e.g. triangulation)
• Image matching algorithms work in the same way
• Matching image samples/features
• Reconstructing the surface covering the features
• Problems with VHR images:
•discontinuities, steep slopes, and occlusions
•homogeneous areas, repetitive textures
•specular effects
•DIFFICULT IN URBAN AREAS
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Our Road Map
• Image feature matching techniques and impact of multi-viewing
- points, slopes
- depth discontinuities
- edges, segments
• Surface reconstruction strategy
- raster-based optimisation techniques
- DSM analysis and segmentation
- feature-based interpolation techniques
- model-based reconstruction
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
How do we match features : from image space or object space?
From object space: vertical line locus constraint
O1
O2
C1
C2
Zmax
dz
z
Zmin
y
x
(X,Y)
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
How do we match features : from image space or object space?
From image space: epipolar geometry constraint
O2
P1 (i1,j1)
O1
P1 (i1,j1)
E
C2
O2
(imin,j1)
E’
(imax,j1)
(imin,j1)
O1
C1
C2
C1
Zmin
Zmax
Epipolar Resampling
Image Matching: a 1D problem? Yes, and no.
Cleaner in 2D due to no perfect estimation of image poses
All matches can be reinjected in the bundle adjutment
1
0,8
0,6
0,4
0,2
0
Surface 3D 1
0,05 0,1 0,15 0,1
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
0,8-1
0,6-0,8
0,4-0,6
0,2-0,4
0-0,2
Institute for
Photogrammetry and
GeoInformation
P1(i1,j1,g
1)
O1
C
O2
1
P2
(i2,j2,g2)
34 56 134 100 103 105 120 107
60 56 55 53 60 56 55 53
C
46 67
54 65
67
66
87
54
94
75
56 54 56 55 57 54 56 55
59 56 54 54 59 56 56 54
57 60
63
54
60
76
80
80
57 56 60 54 57 56 59 54
2
75 69 75
51
54
56
67
73
56 55 54 51 60 56 55 53
90 73 85
93 80 76
65
48
58
54
57
75
63
86
87
89
55 57 57 58 57 54 56 55
54 56 56 54 59 56 56 54
102 101 103 101
53 55 54 53 57 56 59 54
ds(z)
95 76 94 108
100 110 109
87 91 93
Ortho-window image
Ortho-window
image
I1
I2
z
x
Image Right
Image Left
Matching points
y
(X,Y)
Score  Cov( Ik ( x1, y1), Il ( x 2, y 2))
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching points on (steep) slopes
• Least squares matching or adaptive window
shape matching
[Baltsavias 91]
O2
P1 (i1,j1)
O1
C1
C2
ds(z,a,b)
z
x
y
(X,Y)
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching points close to depth discontinuities
(i,j)
(i', j')
(i'' ,j'')
• Adaptive shape matching
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching points close to depth discontinuities
• Adaptive shape matching
1 metre satellite
simulation
[Paparoditis & al 98] [Cord & al 99]
Classical template
Adaptive shape matching
matching
13x13 window size
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching (interest?) points from multiple views
3x3 window size
Correlation score
Robust filtering
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching (interest?) points from multiple views
3D view of robust points
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching segments
• From two views, to achieve robustness, we need a global matching technique: relaxation, dynamic
programming
• From more views, the problem is better-posed; the geometry of the segment itself is self-sufficient
C1
C2
S1
C3
S2
S3
2
1
3
D
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching segments
[Taillandier & al 02]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching segments
[Taillandier & al 02]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching edgels
[Jung & al 02]
[Karner & al 05]
O2
O1
O3
~P
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching edgels
[Jung & al 02]
[Karner & al 05]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching edgels
[Jung & al 02]
[Karner & al 05]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Matching edgels
[Jung & al 02]
[Karner & al 05]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
What Surface Reconstruction?
From a
• Reality
topographic
to a
• Raster-based
cartographic
Reconstruction
Degree of a priori knowledge
• Feature-based
interpolation
• Model-based
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Raster-based and energy minimisation-based matching techniques
• Matching by the research of a field of parallax/surface G minimising E(G) :
E (G)   A I k  k  x, y, G x, y   F G 
Data attachment
Regularisation term
1. Differential approaches (e.g level sets)
+ Precise and possibly quick
- Very good initialisation, criteria and variables differentiable
2. Combinatory approaches (graph theory)
+ No initialisation, no derivation
- Discretisation necessary
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Data attachment term
Ik ( x, y , Z )
Il ( x, y, Z )
( x, y , Z )
Score 
Matching volume
Score( x, y, Z ) 
 Cov( I , I )
k
l
1 k l  N
 Cov( I ( x, y, Z ), I ( x, y, Z ))
k
l
1 k l  N
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Looking for a minima of E
Energy
E (Z )  1  Score( x, y, Z )   * (Z )
Data attachment
Regularisation term
Gradient
 ( Z )  Z ( x 1, y )  Z ( x , y )  Z ( x , y 1)  Z ( x , y )
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Looking for a minimal E can be seen as a a minimal length path problem
• Dynamic programming on a
profile of the surface
In object space
• Matching conjugate epipolar lines
[Baillard 97]
• Artefacts due to dissimetry in the
processing of image lines and
columns
In image space
• Figural continuity constraint
as a solution to minimize
problems
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Roy&Cox Optimal flow algorithm [Roy & Cox 98]
z
A 3D graph can be constructed such as:
•
•
•
•
the nodes are the possible values of Z (voxels in Fig);
the edges are the pairs of neighboring Z (in xy or in z);
the planimetric edges are assigned the regularization cost
the altimetric edges are assigned the data attachment cost;
Roy & Cox showed that :
• the surfaces are the set of graph cuts between the set of nodes of max. Z and min. Z
• the weight of a cut associated to a surface is exactly equal to E
E (Z )  1  Score( x, y, Z )   * (Z )
Finding the surface minimizing E can be seen as finding a minimal cut in a graph
• solvable in polynomial time with classical minimal cut and maximal flow graph theory
algorithms.
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
x
Institute for
Photogrammetry and
GeoInformation
Integration in a multi-resolution scheme [Pierrot-Dessilligny & Paparoditis 06]
To limit the combinatory, a multi-resolution approach
•
•
•
At an initial step, of resolution 2N, we explore all possible disparities.... etc.
At current step, of resolution 2K, initialisation with the previous one of résolution 2K+1;
Morphological dilatation to define research space
Matching
Matching
Matching
Matching
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Application to SPOT5-HRS
12000*60000 image
3x3 window sizes
= 0.2
3 days on 1.8 GhZ PC
1 hour on a cluster
Omnidirectionnal shading
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Application to Pléiädes simulations
tri-stereo
forward-nadir-backward,
70 cm, Toulouse
3x3 window sizes
a = 0.04
Omnidirectionnal shading
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Application to aerial digital frame camera images
6 aerial images
20 cm, Amiens
20 cm, Marseille
3x3 window sizes
a = 0.02
Omnidirectionnal
shading
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Scene analysis
Images & Orientations
Image Matching
and
Thematic interpolation
Stereo DSM
Scene Analysis and
Segmentation
Mono thematic ROI:
Ground, Building, Vegetation
Thematic 3D Features
Extraction
3D Points, 3D Line Segments, ...
Thematic 3D
Interpolation
Thematic, local models.
Global Reconstruction
Final,
complete
SURFACE
Model of the
observed
scene
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Scene analysis and DSM segmentation
Images
DSM
Segmentation
Classification
(terrain, vegetation, building, etc)
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Interpolation-based surface reconstruction (buildings)
• Using raster-based DSM to reduce search space
[Paparoditis & al 01]
[Maillet & al 02]
[Zhang & al 05]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Interpolation-based surface reconstruction (buildings)
• With edgels
[Jung & al 03]
[Karner & al 06]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Interpolation-based surface reconstruction (buildings)
• With edgels
[Jung & al 03]
[Karner & al 06]
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: Building reconstruction
• Heuristic methods
[Haala & al 98] [Vosselman & Suveg 01] [Rottensteiner 01]
• Energy minimisation methods
[Jibrini & al 02]
[Taillandier 04]
H
G
A
A
H
G
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: Building reconstruction
• Using ground footprints
databases and the DSM
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: Building reconstruction
• Finding footprints in the DSM
Object approach using marked point processes: energy minimization with a bayesian
framework
[Ortner 04] [Lafarge & al 06]
Pléïades simulations
Tri-stereo
forward-nadir-backward
70 cm, Amiens
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: Terrain reconstruction
• Robust Adaptive elastic grid interpolation methods:
• Ground features/breaklines and uncertainties can be injected in the minimisation
[Champion & al 06]
245m
165m
Initial DSM
Vegetation and Buildings Masks
(in white, points higher than 245m
(in black) over initial DSM
final DTM
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: 3D City Models
Bati3D Software
• Amiens
• 20 cm images
• 3000 ground footprints
• Automatic reconstruction
• 8 hour of editing
[Kaartinen & al 05]
EuroSDR Building Extraction Comparison
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Model-based surface reconstruction: 3D City Models
Bati3D Software
• Marseille
• 6 camera-head
• 4 IR,R,G,B
• 2 VHR tilted panchro for façades
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006
Institute for
Photogrammetry and
GeoInformation
Conclusion
• Research in this area has significantly improved and is still improving thanks to the integration of
computer vision and image processing tools
• Raster-based energy minimisation matching techniques are very efficient and operationnal at a
production level
• Image quality and multi-viewing are capital to both increase signal to noise ratio and robustness
• Multi-viewing (along and across track) is the key solution to DSM generation automation: agility
of incoming satellites and future constellations should help
• DSMs are a key feature for scene understanding and object extraction
• Feature-based interpolation techniques ensure a duality between surface and object extraction
which is very helpfull for scene management and 3D GIS
ISPRS Technical Commission I Symposium “From Sensors To Imagery”, Paris – Marne la Vallée, France, 3-6 July 2006