Transcript Motivation - Acgsc Inc

Vision Based Following of
Locally Linear Structures using
an Unmanned Aerial Vehicle
Sivakumar Rathinam, Zu Whan Kim,
Raja Sengupta
Center for Collaborative Control of
Unmanned Vehicles
University of California, Berkeley
1
Motivation
 Aim: Enable UAV use for infrastructure monitoring
• Traffic monitoring, aqueduct inspection, pipeline monitoring ….
2
Our Technology
 Keep the vehicle over the structure using with vision in the loop
 Complement GPS waypoint navigation
• Waypoint navigation to get the vehicle over the structure
• Lock it on using vision for accurate imaging
3
Motivation
 Unmanned Aerial Vehicles for Traffic Surveillance
• The Ohio Department of Transportation, The Florida Department of
Inter-Office Cargo
Transportation, The Georgia Department of Transportation
Delivery
Forest Fire Monitoring
 Lane changes, Average inter-vehicle distances, Heavy vehicle counts,
Accidents, Vehicle trajectories, Type of vehicles etc.
 The road should be in view.
 Coifman et. al, Surface Transportation Surveillance from
Unmanned Aerial Vehicles
“The turning radius of the fixed wing UAV is such that changing
directions at waypoints can take some time and space until the vehicle
regains its course. When traversing roadway links of lengths less than
400 ft, large portions of the links went unobserved.”
4
Motivation
Hanshin Expressway, Japan 1995
Alaska pipeline
 The visual feedback compensates GPS inaccuracies and tracks
the structure even it is shifted from the assumed location.
5
Generalization: Vision Based Following of Locally
Linear Structures
(Closed Loop on the California Aqueduct, June 2005)
6
Results
Tracking the California Aqueduct
 The average error of the position of the vehicle from the curve was 10
meters over a length of 700 meters of the canal.
 Algorithm ran at 5 Hz
7
Current UAV Platform Configuration
 Wing-Mounted Camera allowing
for vision-based control,
surveillance, and obstacle
avoidance
 Ground-to-Air UHF Antenna for
ground operator interface
 GPS Antenna for navigation
 802.11b Antenna for A-2-A comm.
 Payload Tray for on-board
computations and devices
 Payload Switch Access Door for
enabling / disabling on-board
devices
8
Current Payload Configuration
 Off-the-shelf PC-104 with
custom Vibration Isolation
 Orinoco 802.11b Card and
Amplifier for A-2-A comm.
 Analog Video Transmitter for
surveillance purposes
 Printed Circuit Board for
Power and Signal Distribution
among devices.
 Umbilical Cord Mass
Disconnect for single point
attachment of electronics to
aircraft.
 Keyboard, Mouse, Monitor
Mass Disconnect for access
to PC-104 through trap door
while on the ground.
9
Problem
 Follow a given curved structure based on visual feedback.
Overhead View
10
Following a Structure using
Visual Feedback
1.
Structure detection
a. Learn the structure from a one example
b. Real time structure detection of the structure
c.
2.
Curve fitting
Tracking
a. Transformation of image to ground coordinates
b. Control the vehicle to follow the structure

Hardware in the loop setup and evaluation

Experiments
11
Basic Detection Idea
Locally linear: Structure should look approximately linear in each image
12
1a. Learning the Structure from One
Example
Rectify image
-Finding the
vanishing point
13
1a. Learning the Structure from One
Example
•Mean intensity will show high
variation at the boundary
•The variance in between the
boundary points should be low
•Done off-line
•Can be automated or manual
mean
Road Template
variance
14
1b. Real Time Detection in each Image
Road Template
For every 4th horizontal
scan line pick several
boundary hypotheses
-Each hypothesis is a
pair of features (high local
intensity gradient)
-Score each hypothesis for
match quality with learnt
profile
-Keep the best three
hypotheses for each
scan line
15
1b. Real Time Detection in each Image
a,b
Corr(Ih’(p),L)
Road Template
For every 4th horizontal
scan line pick several
boundary hypotheses
-Each hypothesis is a
pair of features (high local
intensity gradient)
-Score each hypothesis for
match quality with learnt
profile
-Keep the best three
hypotheses for each
scan line
16
1c. Curve Fitting
Road Template
RANSAC for Curve Fitting
Pick four scan lines at random
and four center hypotheses
i.e., one from each line
Fit a cubic spline
Score the cubic spline
Pick the spline with the best score
Set of supporting scan line
matches
17
Cal Road Detection on MLB Video
(No Control)
Generic corridor
detection by onedimensional
learning
•Roads
•Aqueducts
•Perimeters
•Pipelines
•Power Lines
18
2a. Transforming Image to Ground
Coordinates
 Height is measured by the pressure sensors.
 Use accelerometers and the gyros in the avionics package to calculate
the transformation
• Roll and pitch
 Internal calibration parameters
X

Y

 Z



Coordinates
19
2b. Controlling the Vehicle to Follow the
Structure
 Find a connecting contour that joins the current position to the desired
curve and follow that path
• Position and slope at the origin and the look ahead distance
(Soatto 2000)
20
Literature review – Vision Based Road
Following Systems
 VITS (1988)
• Tracked roads at 13 miles/hr
 Dickmanns (1992)
• Tracked roads in autobahn at speeds up to 62 miles/hr
 Taylor et.al (1999)
• Tracked roads at speeds up to 75 miles/hr
 Eric Frew et.al (2003)
• Unmanned Aerial Vehicle
• Tracked roads at around 44 miles/hr
21
More dangerous stuff……
Obstacle Avoidance
Experiment flown on a Sig Rascal airframe with a Piccolo avionics package
and vision processing on an onboard PC104.
An 8.5 foot diameter balloon was used as the obstacle (distance currently
calculated using GPS).
22
Flight Demonstration
23
Related Work
Vision-based obstacle avoidance has been studied primarily in the context of
mobile ground robots.
• Lenser ’03, Ohya ’00, Lorigo ‘97,
Vision based navigation of UAVs
• Saripalli ’02, Shakernia ’02, Furst ’98 – Landing with known markings
• Sinopoli ’01, Doherty ‘00 – Visual landmark navigation (terrain avoidance) for
helicopter
• Ettinger ’02, Pipitone ’01, Kim ’03 – Pose estimation for aircraft
Obstacle/Collision Avoidance for UAVs
•
•
•
•
Mitchell ‘01 – Aircraft avoiding known aircraft
Sigurd ’03 – Aircraft with magnetic sensors
Sastry ‘03 – Helicopters avoiding known helicopters/obstacles
How ’02 – MILP for Obstacle Avoidance
Vision based obstacle avoidance
• Barrows ’03 – Biomimetic reactive control
24
Related Research
Ground robots
• Fixed baseline stereo – JPL, many others
• Monocular map construction – Lenser (CMU), Kim (Berkeley)
• Cooperative stereo - CMU
Optical Flow
• Helicopter ground following – Srinivasan/Chahl (Australia)
• Corridor following - USC helicopter
• Micro UAV obstacle avoidance – Centeye
UAV depth map construction
• Lidar – CMU Helicopter Project, Sastry (Berkeley Helicopter Project).
• Vision + high precision IMU – Bhanu (joint with Honeywell)
Stereo Vision
• GT Helicopter
25
Requires Depth
Typically use Stereo Vision
 Given the image coordinates of a feature in one image
• if one can find the image coordinates of the feature in the
other image (feature matching), and
• if one knows the rotation and translation of the two image
planes then one knows the world coordinates of the feature
(Ego-motion Estimation)
26
Problem with Depth Estimation by Stereo Vision
0
Z+
Z
Zz
Increased accuracy requires increased camera separation
27
Accurate Depth Estimation is a Problem
Range error due to pixel errors is
dZ
dp

Z
2
B f
.
28
Approach
UAVs flying at low altitudes must autonomously avoid obstacles
Strategy
• Segment the image into sky and non-sky

Non-sky in the middle  OBSTACLE
• Strategy 1

Aim at the sky
• Strategy 2

If it looms faster than a threshold and is in the middle  AVOID
Else do NOTHING
29
Tailored to………………..
For most UAV applications (>50 m), the obstacles of concern will be
large objects such as towers, buildings or large trees
For these cases, the problem of obstacle detection is different from that
of ground vehicles in environments cluttered with many obstacles.
VS
30
Segmentation at Moffet Airfield
Results for multiple regions found (only largest regions shown, dark
blue represents all small regions)
31
Sky Segmentation
32
Flight Demonstration
-100
0
100
200
300
400
0
-50
avoidance with GPS
Balloon
-100
y position (m)
-150
direction of flight
-200
-250
-300
-350
-400
-450
autonomous control
started
-500
x position (m)
33
Vision Processing
 Classification: balloon/horizon correctly found in ~ 90% of images
 Time results: ~2Hz (120ms SVM, 200-600 ms horizon)
34
Flying Low
Helicopter pilots fly low
FAA requires see and avoid
Find the freeway and follow it
35
Used Sectionals to build a Manhattan
model at 300 feet (approx.)
Simulation testing of Control
36
Cal UAV: Target Capabilities
Obstacle Avoidance
Simulation testing of Control
• Flight through Manhattan model (300+ feet)
37
End 2
38