Transcript Stabilization and Georegistration of Aerial Video Over Mountain

Stabilization and
Georegistration of Aerial
Video Over Mountain
Terrain by Means of LIDAR
IGARSS 2011, Vancouver, Canada
July 24-29, 2011
Mark Pritt, PhD
Lockheed Martin
Gaithersburg, Maryland
[email protected]
Kevin LaTourette
Lockheed Martin
Goodyear, Arizona
[email protected]
Problem: Georegistration
 Georegistration is the assignment of 3-D geographic
coordinates to the pixels of an image.
 It is required for many geospatial applications:



Fusion of imagery with other sensor data
Alignment of imagery with GIS and map graphics
Accurate 3-D geolocation
 Inaccurate georegistration can be a major problem:
Correctly
aligned
Misaligned
GIS
2
Solution
 Our solution is image registration to a high-resolution
digital elevation model (DEM):


A DEM post spacing of 1 or 2 meters yields good results.
It also works with 10-meter post spacing.
 Works with terrain data derived from many sources:




LIDAR: BuckEye, ALIRT, Commercial
Stereo Photogrammetry: Socet Set® DSM
SAR: Stereo and Interferometry
USGS DEMs
3
Methods
 Create predicted images from the DEM, illumination
conditions, sensor model estimates and actual images.
 Register the images while refining the sensor model.
 Iterate.
Aerial Video
Sensor
Illumination
Occlusion
Scene
Shadow
Predicted
Images
4
Methods (cont)
Predicted
Image
from DEM
The algorithm identifies tie
points between the
predicted and the actual
images by means of NCC
(normalized cross
correlation) with RANSAC
outlier removal.
Predicted
Image from
Aerial Image
Registration
Tie Point
Detections
5
Methods (cont)
 The algorithm uses the refined sensor model as the
initial guess for the next video frame:
Initial
Camera
Register
• Estimate
camera
model
• Use camera
focal length
& platform
GPS if avail.
• Predict
images from
DEM and
camera
• Register
images with
NCC
Refine
• Compose
registration
fcn & camera
• LS fit for
better cam
estimate
• Iterate
Next
Frame
• Register to
previous
frame
• Compose
with cam of
prev. frame
for init. cam
estimate
Iterate
• Iterate for
each video
frame
Finish
• Trajectory
• Propagate
geo data
from DEM
• Resample
images for
orthomosaic
 The refined sensor model enables georegistration.
 Exterior orientation: Platform position and rotation angles
 Interior orientation: Focal length, pixel aspect ratio, principal point
and radial distortion
6
Example 1: Aerial Motion Imagery
Inputs:
Aerial Motion Imagery over
Arizona, U.S.
1/3 Arc-second
USGS DEM
Area: 64 km2
Post Spacing: 10 m
16 Mpix, 3.3 fps, panchromatic
7
Example 1 (cont)
Problem: Too shaky to find moving objects
Zoomed to full resolution (1 m)
8
Example 1: Results

Outputs:



Sensor camera models
Images georegistered to DEM
Platform trajectory
9
Example 1 Results (cont)
ATV
Vehicle
Pickup
Truck
Human
Video is now
stabilized, and as a
result, moving
objects are easily
detected.
10
Example 2: Oblique Motion Imagery
Inputs:
Oblique Motion Imagery Over
Arizona, U.S.
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
16 Mpix, 3.4 fps, pan
11
Example 2: Results
Target
Tracking
Stabilized
Video Inset
Map
coordinates
Aligned
Map
Graphics
Orthorectified
Video
Background
LIDAR DEM
Aligned
Map
Graphics
12
Example 2 Results (cont)
 How fast does the algorithm converge?
IMAGE 591
Num tie
points
RMSE
Mean Δx
Mean Δy
Sigma Δx
Sigma Δy
319
318
282
17.4
1.4
-3.8
15.8
6
4.8
-0.7
-0.1
4
2.6
2.9
0.1
0
2.5
1.5
Tie Point Residuals
Image Pixels
Num tie
points:
RMSE:
Mean Δx:
Mean Δy:
Sigma Δx:
Sigma Δy:
Camera Iteration
1
2
3
20
18
16
14
12
10
8
6
4
2
0
RMSE
mean
sigma
1
2
Camera Iteration
1
2
3
3
Tie Point Residuals
3
681
687
681
2.7
1
0.9
2.1
0.9
0.6
0
0
0.5
0.2
0.3
0
0
0.3
0.1
The initial error
is high, but it
decreases after
only several
iterations.
Camera Iteration
Image Pixels
IMAGE 1
2.5
RMSE
mean
sigma
2
1.5
1
0.5
Subsequent
frames have
better initial
sensor model
estimates and
require only 2
iterations.
0
1
2
3
Camera Iteration
13
Example 3: Aerial Video
Inputs:
Aerial Video Over
Arizona, U.S.
720 x 480 Color 30 fps
LIDAR DEM
Area: 24 km2
Post Spacing: 1 m
14
Example 3: Results
Background
Image
Draped Over
DEM
Map
coordinates
Orthorectified
Video
Aligned
Map
Graphics
15
Example 3 Results (cont)
Map Graphics Stay Aligned with Features in Video
16
Example 4: Thermal Infrared Video
Inputs:
MWIR Video Over White
Tank Mountains in Arizona
Commercial
LIDAR DEM
Post Spacing: 2 m
1 Mpix, 3.3 fps
17
Example 4: Results
Video Mosaic
Georegistered and
Draped Over Mountains
in Google Earth
Video
Mosaic
Background
LIDAR DEM
Inset:
Original
Video
with Map
Graphics
Overlay
18
Demo
Click picture to play video
19
Conclusion
 We have introduced a new method for aerial video
georegistration and stabilization.
 It registers images to high-resolution DEMs by:



Generating predicted images from the DEM and sensor model;
Registering these predicted images to the actual images;
Correcting the sensor model estimates with the registration results.
 Processing speed is 1 sec per 16-Mpix image on a PC.
 Absolute geospatial accuracy is about 1-2 meters.

We are developing a rigorous error propagation model to quantify
the accuracy.
 Applications:



Video stabilization and mosacs
Cross-sensor registration
Alignment with GIS map graphics
20