Transcript – MIR Multiple Image Registration Final presentation

www.nr.no
MIR –
Multiple Image Registration
Final presentation
Line Eikvil
Norsk Regnesentral
ESRIN
June 28, 2005
Outline of presentation
►
Problem/Background
►
Objective
►
Review of methods and tools
►
Selected approach
►
System
►
Examples and results
www.nr.no
Problem
►
Co-registration important in many remote sensing
applications.
►
Automatic techniques exist, but there is no one registration
technique that works equally well for all image types.
►
More than 90% of studies in remote sensing that could have
used automated approaches for registration of images do not
use them.
►
The lack of a more general tool for helping in this process
may be one of the reasons for this.
►
Useful to have a more general tool for image registration that
could be used for several applications.
www.nr.no
Objective
►
Develop a co-registration tool:
▪
for homogeneous time series of images
◦
◦
▪
which is general and can handle time series
◦
◦
◦
▪
▪
From the same/similar sensor
With comparable resolution
From different sensors
With different contents
Acquired under different circumstances
by providing a selection of different methods
and intelligence enabling selection of the most
appropriate method for each problem.
www.nr.no
Main idea
►
Integrate existing methods and tools for
registration.
►
Develop methods and functionality for automatically
choosing the most appropriate registration
approach based on image characteristics.
www.nr.no
Project activities
►
Review
▪
▪
▪
Study of the problem
Study of the methods
Study of appropriate tools
M1 Kick Off
Analyse
problem
WP2000
Review
existing tools
WP2000
Detailed design
WP5100
Overall design
WP4100
Relevant
methods
and tools
Software
requirements
WP3000
M2 SRD Acceptance
Selection of
methods and
tools WP4200
Interface
design (API/GUI)
WP4300
Architectural
design
Implementation
and integration
WP 5200/5300
Define testplan
WP 6000
Testing and
validation
WP6000
M3 SW Acceptance
M4 Final presentation
►
Selection of methods and tools.
►
Development of overall approach, necessary
intelligence and additional methods.
►
Integration with existing tools into a new tool.
www.nr.no
www.nr.no
Review and
selection of tools
Image registration process
►
The main steps in a registration process are
▪
▪
▪
▪
Feature extraction
Feature matching
Transform model estimation
Resampling
►
Feature extraction and matching are the steps that
▪ mainly determine registration performance
▪ are most dependent on image characteristics.
►
Many methods exist and tools providing wider selections of
methods are appearing.
►
BUT: no methods or tools have yet appeared that provide
automatic selection of methods.
www.nr.no
Review and selection of methods
►
Aspects considered:
▪
►
Image characteristics, application areas, maturity, complexity,
performance, suitability for the project.
Identified metrics/features
▪
▪
▪
Correlation-based. Pixel-wise cross-correlation. Invariant to
linear contrast changes, robust to noise. Sensitive to clutter,
occlusion and nonlinear contrast changes.
Mutual information. Measures dependency while the actual form
of the dependency does not have to be specified. Robust to
noise and can handle multi-sensor registration.
Wavelet/edge based. Can be used as preprocessing or to find
feature points. Wavelets suitable for multi-resolution
approaches.
www.nr.no
Review and selection of tools
►
Classes of tools considered:
▪
▪
▪
►
Choice of tools based on:
▪
▪
▪
►
Image registration tools
General remote sensing tools
General image processing tools
Functionality provided
Expected ease of integration
Licensing and availability.
Selected tools:
▪
▪
ENVI/IDL
Insight Toolkit
www.nr.no
www.nr.no
Approach
Overall approach
►
The approach subdivides the images into rectangular regions
allowing for elimination of unsuited regions and the use of
different matching methods for different regions.
▪ Features are extracted from each region
▪ Based on the features, the expected performance (the
rating) of each method for that region is estimated.
▪ Regions with low ratings are discarded.
▪ For each of the remaining regions the method with the
highest rating is used to perform local matching.
▪ Finally, a global transform is estimated from the set of
local transforms.
www.nr.no
Feature extraction
►
The images are subdivided into
rectangular regions.
►
Features are extracted from a pair of
regions.
▪
▪
▪
►
Texture features
Statistical features
Differences between these
The features from the two regions
are merged into a joint feature vector
Fixed
Moving
Feature
extraction
X = [x1, …, xn]
www.nr.no
Overview of features
►
Features are selected to say something about:
▪ The information content in the region
▪ Difference between fixed and moving
►
Texture features
▪ GLCM and gradient features
◦
◦
►
Statistical features
▪ Region and zone means, variance, entropy
◦
►
from moving image
difference between fixed and moving image
difference between fixed and moving image
A total of 26 features are extracted from each region
www.nr.no
Region and method rating
►
From the extracted features a
neural net is used to predict the
performance of each method for
each region.
►
The prediction is based on the
extracted features.
►
The net is defined as follows:
▪
▪
▪
N input nodes (N = nof features)
One layer of hidden nodes
M output nodes (M = nof methods)
Features:
X = [x1, …, xn]
Region/method
rating
Ratings:
R = [r(m1), .., r(mm)]
www.nr.no
Training of the neural net
►
Regions and features are extracted
from a set of image pairs with known
geometric displacement.
►
All different matching methods are
applied to each region.
►
The distance between the estimated
transform and the true transform is
computed for each region and
method.
►
Features and truncated distances are
used to train the net.
Features
Distance from
known distortion
using each method.
www.nr.no
Region and method selection
►
►
►
►
Regions with low ratings are
discarded.
For the remaining regions, selection
is performed to retain a good
distribution over the image.
For each of these regions the
method with the best score is
selected.
Local region matching can then be
performed with the selected method.
Ratings:
R = [r(m1), .., r(mm)]
Region/ method
selection
1
1 1 1
1 2 1
1
1 1 2 2 2
1
2 2 1
1 1 2
1
1 3 1
1
1
2
1
1
1
1
1
1
3 3
1 2 2 3
1
1 1
1
2
2
www.nr.no
Methods for region matching
Methods fetched from ITK
►
Metric
▪
▪
▪
►
Normalized cross-correlation
Mean squares
Mutual information (different varieties)
Moving
Image
Optimizer
▪
▪
▪
Gradient Descent
Regular step gradient descent
Genetic algorithm
►
Transform
►
Interpolator
Fixed
Image
Metric
Optimizer
Interpolator
Transform
www.nr.no
Choice of matching methods
►
We define a matching method as:
▪ a combination of a metric and an optimizer.
►
A selection of 10 different methods is used.
Method ID
Metric
Optimizer
M1
Mattes MI
Regular Step Grad. Descent
M2
Normalized Correlation
Gradient Descent
M3
Mean Squares
Regular Step
M4
Mattes MI
One Plus One Evolutionary
M5
Normalized Correlation
Regular Step Grad. Descent
M6
Viola-Wells MI
Gradient Descent
M7
Viola-Wells MI
Regular Step Grad. Descent
M8
Mean Squares
Gradient Descent
M9
Mean Squares
One Plus One Evolutionary
M10
Norm. Viola Wells MI
Regular Step Grad. Descent
www.nr.no
Transform estimation
►
►
►
►
The selected matching method is used
to estimate the transform for each of
the selected regions.
The set of estimated transforms is
analysed to remove obvious outliers.
Control points are computed for each
region based on the estimated
transforms.
A global transform is computed from
the set of control points.
Set of region
transforms
Outlier removal
Reduced set of
region transforms
Control-point
computation
Set of control
points
Estimation of
global transform
and image
resampling
www.nr.no
Overview of process
Fixed
Moving
Selected
regions
and
methods
Feature
extraction
X = [x1, …, xn]
Region/method
rating
1
1 1 1
1 2 1
1
1 1 2 2 2
1
2 2 1
1 1 2
1
1 3 1
1
3 3
1 2 2 3
1
1
2
1
1
1
1
1
1
2
2
1
1 1
Region/ method
selection
Ratings:
R = [r(m1), .., r(mm)]
Region
matching
Set of region
transforms
Outlier removal
Reduced set of
region transforms
Control-point
computation
Set of control
points
Estimation of
global transform
and image
resampling
www.nr.no
www.nr.no
System
Overview: MIR tool
MIR tool
Interface
Input Checker
User Interface
Communicates with
Interface
Session
Config. Files
Batch Interface
Transform Estim
Extractor
Selected parts:
Region/Method
C++
(Insight)
Rater
Region
Matcher
Global
Transform
Estimator
C
Retrieves
setup from
Starts
Main system:
ENVI/IDL
Feature
Region and Method Selector
Region/Method
Selector
User
Process
Control
Region and
Method
Selector
Interface
User Interface
Transform
Estimator
Retrieves method
setup from
Method
Config. Files
Image IO
Batch Interface
Region and Method Selector
Transform Estimator
Feature
Extractor
Region
Matcher
Region/Method
Rater
Global
Transform
Estimator
Region/Method
Selector
www.nr.no
System features
►
Runs under Linux
►
Provides a GUI and a batch interface
►
Operates in two modes
▪
▪
►
Automatic selection of methods
User-selection of methods
Method configuration possible through parameterfiles.
www.nr.no
Graphical User Interface
www.nr.no
Graphical User Interface
www.nr.no
www.nr.no
Examples and
results
Test examples
►
NOAA-AVHRR
▪
►
Landsat
▪
►
Challenges: Clouds, varying snow cover
Challenges: Clouds, phenology
ERS1
▪
Challenges: Varying soil moisture, crop maturity
www.nr.no
NOAA-AVHRR
May 31, 2003
July 7, 2003
www.nr.no
www.nr.no
NOAA-AVHRR: Regions
www.nr.no
NOAA-AVHRR: Region selection
www.nr.no
NOAA-AVHRR: Region selection
www.nr.no
NOAA-AVHRR: Method selection
M1
M4
M5
M6
M7
M9
www.nr.no
Landsat
Aug 19, 1995
July 31, 1994
www.nr.no
Landsat
www.nr.no
Landsat: Region/method selection
M1
M4
M5
M9
www.nr.no
ERS1
April 30, 1993
July 9, 1993
www.nr.no
ERS1
April 30, 1993
July 9, 1993
www.nr.no
ERS1
April 30, 1993
July 9, 1993
M4
www.nr.no
Summary of results
►
Results are promising
▪
▪
Works on a range of different images without
specific tuning.
Able to discard areas not suited for registration:
◦
▪
Clouds, Ocean, etc.
Able to perform registration under varying
conditions:
◦
Varying snow cover, differences in phenology, crop
maturity, soil moisture etc.
www.nr.no
Summary
►
A general tool for co-registration has been
developed.
►
The tool provides:
▪
▪
a selection of registration techniques
intelligence for automatic selection of:
◦
◦
►
the regions of the image to be used, and
the registration technique to be used for each region.
It has been tested on time-series of optical and
radar images and results are promising.
www.nr.no
Possible further work
►
Larger-scale testing and tuning
►
Automatic selection of bands
►
Inclusion of additional methods
▪
Wavelet-based/edge-based
►
Possible to extend to multi-sensor?
►
Include tools for training
www.nr.no