Transcript Slide 1

Status of AiTR/ATR
in
Military Applications
James A. Ratches
CERDEC NVESD
January 2007
UMDAROATR
Outline
• Definition
• Importance & Scenarios
• Performance Assessment
• Problem Statement
• Way Forward
• Summary & Conclusions
Military Definition
• Generic term to describe automated/semi-automated
functions carried out on imaging sensor data to perform
operations ranging from cuing a human observer to
complex fully autonomous object acquisition and
identification
• Machine function:
- Detection - Classification - Recognition - Identification - Friend or Foe
• Aided Target Recognition (AiTR)
- Machine makes some level of decision and annotates the image
- Human makes higher level decision. e.g. to identify and fire
• ATR is fully autonomous
- No human in-the-loop after weapon firing, e.g. fire-and-forget seeker
• ATR/AiTR may use information from other sensors to make
decision by fusing information
AiTR (aided)
ATR (autonomous)
Scenarios Where AiTR Essential
Rapid wide area search for close combat
in high clutter, against difficult targets
(occlusion, defilade, CC&D) and variable
target signatures
Urban terrain; 360 degree situational
awareness, short ranges, human intent,
transmission limitations
UAV & UGV transmissions
BDA
over limited bandwidth
road
debris
road
Tree trunks
New object
1030
1830
Scouts in Overwatch-Objects of interest and scene changes
Detection of
Dismounts & intent,
& bunkers
Missile Scenarios Where ATR Essential
1st
Waypoint
(Tower)
Fire Units
Obstacle
2nd
Waypoint
(Mountain)
Field Of View
Target of
Opportunity
4 Km
Navigate To
Emplacement Site
- Power Up
- Computer Initialization
- Intelligence Preparation of
Battlefield
- Plan Missile Routes
if necessary
Missile Auto-Navigate To
Target Search Point
(Enroute Recon)
-Receive Target
Information
Through C2 Network
- Verify Target Selection
- Route & Salvo Selection
-Launch Missile(s)
-Acquire GPS Satellites
-Update GPS Position
-Calibrate the Inertial
System
- Navigate to Target Area
Detect, Recognize &
Identify Target
(Engage Autotracker)
- Start Search (Wide FOV)
- Locate Target (Narrow FOV)
- Lock On
- Aimpoint Update
- In-Flight Intelligence
- Target Marking
- Target Reporting
to C2 Network
Warhead Function
On Impact
AiTR Annotates Images – Not Maps
Which pixels in image correspond to targets?
Lab/Field Measured Performance
ROC Curves
1
0.9
Algo 1 Algo 1 Algo 2 Algo 2 -
Probability of Detection
0.8
0.7
Clutter levels:
In Open
Occluded
In Open
Occluded
0.6
High
Hunter Ligget
Medium
Yuma
Low
Grayling
0.5
0.4
0.3
Hunter Ligget
0.2
0.1
Yuma
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Grayling
0.8
False Alarms per Square Degree
Effects of Occlusion
Effects of Clutter
Manual FLIR (30 FOR) Search Time > 60 Sec.
0
H MD
V IS
H MD
M ODE
H MD
WP
NAV
U PD T
N AV
AP LT
NA V
WP
+
Aided Target Search Time less than 4 Sec.
SCRN BRT
SLAVE
W
N
29
S
X
Y
+
TS D
HO ME
F T
T SD
SC LE
E
NB
2320
445 6
NB52
002250
TSD
TAC
T SD
NAV
A
B
1
TS D
CN TR
T SD
W NDW
CTRS
+
WIDE AREA TARGET CUEING WITHIN 4 SECONDS
Target Acquisition Sensor Suite (TASS)
SWIR CAMERA
AiTD/R
•Assess Maturity Ground Based
AiTD/Rs in Varied Environmental
Conditions
Gimbaled Scan
FLIR
•Long Range Target
Detection
• 2nd Gen B-Kit (LWIR)
Evaluations yield
ROC curves
•Long Range Target Identification
•Leverage ACT II LIVAR and CETS
Program - EBCCD Technology
•4.5” Aperture
MTI Radar
•Utilize
AN/PPS-5D
Laser Illumination/
Designation
• Assess maturity of
SOA AiTR in gimbal
scanning mode in
the field
• SOA single color/
shape LWIR based
algorithms from
COMMON SENSOR
multiple sources
Long Range Scout Surveillance System
2nd Gen FLIR Modified f/ Gimbaled Scan • Include urban bkgds
and man targets
AiTR/ATR Continues to Be Tested in Realistic Environments
Overall Assessment
1. DOD investment in AiTR has resulted in quantifiable
level of performance documented in ROC curves
2. Performance measured under favorable conditions
3. Order of magnitude improvement in search time with
AiTR over human only
4. Discrimination levels above detection have not been
vigorously pursued*
5. Detection performance can have degradation for suboptimal conditions*
- high clutter - low contrast - obscuration - extended ranges
6. Training target sets have been typically for < 10 targets
7. There are no human detection algorithms
SOA AiTR Algorithms Have Known Limitations
* Especially for ground-to-ground
Need for Robust AiTR/ATR
For future combat scenario must be robust
- High false alarm rate renders aid useless and operator
will turn it off (AiTR)
- Ground-to-ground presents high clutter
- Target variability increases complexity
- Low signature targets can be expected
- Partial occlusion & defilade obscures the target
- CC&D need to be mitigated
- Detect human threats in urban terrain
- Final ID can be man-in-the-loop (AiTR)
Robust AiTR/ATR Critical for Ground-to-Ground
Manned-shoot first
Close Fight Unmanned-autonomous
operation
The AiTR/ATR Problem
• ~$100M investment to realize SOA AiTR
• Humans can still do better than SOA
AiTR (Except for speed)
• Robust AiTR required
- Potential target set is large with wide range of
environmental and operational variations
-AiTR for humans and urban terrain
• New university concepts have not migrated
to industry and military developers
ARL-SEDD
DATA
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
Aided
Manual
20
40
Search Time (sec)
AiTR/ATR Cannot Do As
Well As The Human
Alone-However,
It Can Do It Faster
-Improvement that approaches
human performance will be
an enabling force multiplier
60
Perceived Impediments to ATR/AiTR
• Required computational power
• High cost, power and size
• Proprietary issues
• Tactical scene complexity
• Required to be better than human
• New CONOPS will be needed to
fully utilize benefits of ATR/AiTR
Real Limitation Is The Lack of An Image ScienceWhat Is Important in An Image?
Possible Paths to Improvement
• 3D LADAR – if can cover/search field of regard
- Otherwise, use for higher level discrimination
• Multi/hyper-spectral/look/mode sensor and
Sensor Fusion
• Untried University “New Ideas”
- Recognition by parts
- Advanced eye-brain understanding
- Gradient index flow and active contour analysis
- Frame-to-frame correlations
- Spatial contextual intelligence
- Hierarchical imaging
- Category theory
• Off-board sensor features data via low bandwidth
tactical networks
• Validated synthetic image generation to stress
algorithms during formulation
• Investment in Image Science
ARO/Duke University Workshop
Computational Sensors for Target Acquisition and Tracking
Beaufort, NC December 2-4, 2003
Representative Recommendations
• Different approach to applying eye-brain understanding to
AiTR needed
- Does not necessarily mean that we need to mimic that process
• Artists may give a unique insight into minimalist
representation
• Poor performance of AiTRs relative to humans suggest there
are better features than have been found by AiTRs
• The perspective of clutter rejection rather that object feature
extraction may present a different set of opportunities
Eye-Brain Understanding Can Still Be A Fertile Ground
of Investigation for AiTR Concepts
Progression of Algorithms
ALGORITHM
1. Statistical
a. no range
b. w/range
FEATURE
ΔT, size, perimeter, etc.
EXAMPLE
Comanche
same + target window size
CAPABILITY METRIC
- target in open & in center of FOV
- ~ 10 target set in low clutter
- baseline performance
- reduce search time (10X)
- reduce FAR (10X)
*
*
*
2. Template Matching
comparison of ROI to stored
target templates
SAIP
- expand target set, e.g. aspect,
articulation, dirurnal/seasonal, etc.
2. Model Based
comparison of ROI to stored
target model
MSTAR
- increase target set with stored data
set reduction
4. Multi-spectral
pixel value=f(λ,Δλ)
MFS3
- penetrate camouflage
- reduce FAR
5. Multi-look
target indications at GPS
coords from off-board
sensors
Dynamic
Variable
Threshold
- reduce FAR (~ 10-100X) by
by correlating target detects
- detect obscured, defilade targets
- missed target reduction (~2X)
6. Multi-mode
non-imaging sensor indications
(sound, vibration, magnetics)
ASM
algorithms
- mitigate CC&D
- reduce FAR
* Classified data on false alarms and P
d
exist for these algorithms
*
*
*
Progression of Algorithms (con’d)
ALGORITHM
FEATURE
EXAMPLE
CAPABILITY METRIC
*
7. Geographic Contour Maps
terrain slope
DTED
- FAR reduction (potential ~ 75%)
8. Advanced Eye-Brain
Understanding &
Representation
synapse maps
9. Recognition-by-Parts
target subelements
detected
- detect partially obscured targets
- missed target reduction
10. Gradient Index Flow &
Active Contour Analysis
2D chips of humans
- determine human intent
11. Frame-to-Frame
Correlations
pixel change
correlations
12. Spatial Contextual
Intelligence
target forbidden terrain
- reduce search time
by reducing search area
13. Artists Insights
hierarchical scene
characteristics
- reduce search time by
focus on search area
14. Hierarchical Imaging
activate/retard signals
- bandwidth reduction by
evaluating information before
transmission
15. Category Theory
sensor report &
locations
NN, holographic NN,
wavelets
MTI
Sf
S2
Sw
S1
- intelligent search & detect
- FAR reduction
- reduce search timelines
- detect changes in scene
- reduce search times
Sc
- geolocation accuracy
improvement
Theoretical Basis for Multi-Look
•
Different features have
different ROC curves
ROC Curves for 2100m, 2500m, and Fused
1
– Range dependent
•
•
Features from different
sensors and platform can be
passed over the network (low
bandwidth information)
Performance gain
proportional to ROC curves
Pick 2 features as example
– local variation
– wavelet
0.810X
FAR reduction
0.7
0.6
Pd
•
0.9
0.5
0.4
0.3
Fused Best: Pd Indep., Pfa Indep.
Fused: Pd Indep., Pfa Fully Correl.
Fused Worst: Pd, Pfa Fully Correl.
2100m
2500m
0.2
0.1
0
0
Pfa
Features from Off-Board Sensors Can Improve On-Board
Sensor AiTR/ATR
False Alarms Uncorrelated between Sensors
Ridge Sensor
R4
R2
R3
R2
R1
V2
Valley Sensor
LWIR-1
V2
V1
LWIR-2
LWIR-2
LWIR-1
Most false alarms
Most false alarms
Most false alarms
for LWIR-1
Plan View
Most false alarms
for LWIR-2
for LWIR-1
for LWIR-2
Side View
Category Theory
Recognition-by-Parts
Gun
barrel
Category Theory is a mathematically
sound framework:
Hot spot
geon2
Turret
geon3
geon1
-Designed for network applications
-Describes information fusion systems
and processes in an elegant language
- Captures commonality and relationships between objects
Specifications S:
S1. S2 data from sensors 1 & 2
Sc real world stimuli
Sw ground truth
Sf registration transformation
between S1 & S2
Morphisms:
arrows
Functors:
Relationships with other
categories
Sf
FLIR Image
geon4 Tracks & wheels
geon5
Engine
exhaust
Recognition is based
on recognition of
critical sub-components
called geons
Biderman (USC)
T-72 tank
Library of Geons for targets of interest
forms the basis for recognition
S1
Sw
S2
Sc
Example of a
category
O1
O2
O3
c
O4
Kokar (Northeastern)
cxaxc=(cxb)xa=cx(bxa)
Composition operation that is associative
a
b
Network supplies opportunity for sophisticated
fusion techniques to be applied to AiTR
“Image Science” Based Algorithms
SOA algorithms attempt to recognize static targets in
single frames: Need to consider more image-based, e.g.
parameters e.g., image temporal-spatial relationships.
Sensor-Scene Dynamics
Change detection & MTI
Context
Algorithms Must Extract More Contextual Information
Gradient Vector Flow (GVF)
(Active Contour Analysis)
• Higher level process or user initializes any curve close to the
the object boundary (indication of a region of interest)
• The parametric curves (snakes) then starts deforming and
moving towards the desired object boundary
• In the end it completely “shrink-wraps” around the object
Gradient of intensity
(x, y)
Human intent
GVF field is defined to be a vector field X [x(s), y(s)]
for s in [0,1]
Solve Euler equation αx''(s) - β x''''(s) - Eext = 0
to minimize energy functional E = ∫01 ½ (α│x'(s)│2 β│x''(s)│2) + Eext (x(s))ds (α and β user defined constants)
Eye-Brain Understanding Must Be Applied Faithfully
Artists Unique Insight
This painting shows how Van Gogh was able to transmit detailed
Information about a person (20-year old woman) to the viewer
Using Only ~10 brush strokes for her face.
From Falco (U of AZ)
Hierarchical Imaging &Target Representation
Elements of Network Make Localized Decisions Rather Than
Simply Sending Raw Data to A Central Processor
• Sensors
Sample n Parameters
• Network becomes large scale sensor
• Hierarchical decisions
- Local decisions determine relevant information
- Global decisions develop global model
- Each node is a virtual point detector at the next
level
- Algorithms determine what is to be shared/when
The Network Becomes The Sensor & AiTR
Conclusions
• SOA ATR/AiTR has attained a level of performance that
has some level of military value
- Targets in the open
- Low to medium clutter
- Target set ~ 10-15
- No obscuration or camouflage
- No humans or human intent
- No high value targets, e.g. bunkers
• Major new innovations are needed to get a leap
ahead in performance under operational environments
- New university concepts
- Network information
Back Up Slides
Aided Target Recognition for Intelligent Search
outer
Range x
Inner
Prescreener
Sensor
Feature
Extraction
Registration
Recognition
Neural
Net
M-35
M35 @ 0°, scale = 1.0
ZSU @ 165°, scale = 0.85
M60 @ 180°, scale = 0.85
M35 @ 270°, scale = 1.0
M35 @ 195°, scale = 1.0
Representative Configuration of SOA AiTR
3D from Optical Flow
Crater
Above
Surface
Mound
• Subtle motion provides substantial depth information
• Memory/ processing advances permit harvesting of depth
information (Target/Sensor motion)
• Algorithms have been developed that amplify motion vectors
and present them in a binocular display in real time to create
“hyperstereo” using advances in microlens technologies
Processing Motion Information Can Provide Depth (Range)
Courtesy of FOR 3D, Santa Rosa, CA
Passive Ranging with DTED Data
DTED Data
Lines of constant range
superimposed on FLIR
image at HunterLiggett
Lines of constant range superimposed
on FLIR image where earth is flat at
Yuma P.G.
Flat Earth Approximations Cannot Be Used for All Scenarios
Of Interest
Work on passive ranging and imagery by Raytheon.
Passive Ranging
• Accurate range estimation can reduce false alarm rates in AiTR
• Permits estimation of target size
• Active ranging potentially reveals position
• Most AiTR algorithms make a flat earth approximation
• Passive ranging may provide more adequate accuracy
Optical flow
DTED w/GPS
GPS
Lines of
constant
range
Eye motion
Near field
objects
Far field
objects
DTED
overlaid
on imagery
El & Az of
gun known
“ Despite often heard claims to the contrary, without range data there is no way
of knowing if the target is 100- times smaller than a pixel or 1000 times larger
than the image as a whole. – Northrop-Grumman