Transcript Briefing

Information Fusion Technical Area
Overview & Applications
Joseph A Karakowski
[email protected]
(732)460-7752
November 16, 2011
CACI Private Data; Do Not Copy or Distribute
What we will cover..
the “important” parts…..about fusion
CACI Private Data; Do Not Copy or Distribute
Agenda
• Background Technology
– Fusion Definition
– Fusion Models
• Fusion Technology Sector Applications
– Military
– Medical & Non-Military
• Personal Fusion Areas (Optional)
Questions to be Answered:
What is Fusion Technology and it’s basis?
What are some example fusion applications in specific markets?
CACI Private Data; Do Not Copy or Distribute
Some Fusion Definitions…
…the process of combining data/information to
estimate or predict the state of some aspect of the
world (Bowman)
…the process of utilising one or more data sources
over time to assemble a representation of aspects of
interest in an environment (Lambert)
…series of processes performed to transform
observational data into more detailed and refined
information, knowledge, and understanding (USArmy)
…everything is Connected… a “Global Graph”
portrays the connected world; graph nodes are the
entities; graph links are the actions or relationships
(Walsh)
CACI Private Data; Do Not Copy or Distribute
What is Fusion?
Technical methods/processes which supports, through
cognitive/perceptual modeling, the solution of a class
“Difficult Problems”
These are a first scientific step to solve these classes of
problems, which have not been solvable, up to this time.
Implementation of these processes using information
technology, has been moving forward for the last 25+
years, and will probably continue for many more years…
Some Typical characteristic of Difficult Problems:
• Multiple Goals
• Complexity, with large numbers of items, interrelations and decisions
• Dynamic, time considerations
• Cognitive/perceptual problem solving
CACI Private Data; Do Not Copy or Distribute
“The Blind Man & the Elephant”
Question: What is an Elephant?
CACI Private Data; Do Not Copy or Distribute
The “Fusion Elephant”
It’s a
Cognitive/
Perceptual
Process!
Question: What is Fusion?
Its
Biometric
Apps!
It’s
Intelligence
Apps !
It’s the
JDL
Model !
A State
Prediction
Problem!
CACI Private Data; Do Not Copy or Distribute
It’s a
Global
Graph !
A More Realistic “Fusion Elephant”
Nuclear
This is a new
technology, and
much RD&E
remains
CACI Private Data; Do Not Copy or Distribute
Major Fusion Process Models
Many Definitions…and many more models have been proposed and built!
• Joint Directors of Laboratories Model (JDL)* [1986-Pres]
• Transformation of Requirements for Information Process
(TRIP) Model [2000-?]
• Visual Situation Assessment Model () [1997]
• Salerno SA Model [2001-Pres]
• “Graph” Fusion Model [2005-Pres]
• Contextual Fusion Model * [2009-Pres]
• There are many others….
CACI Private Data; Do Not Copy or Distribute
DFG Functional Model
(JDL Model)
Process
Refinement
Level 4
Human/
Computer
Interaction
Source
Input
Preprocessing/
Predetection
Fusion
Level 0
Single Object
Refinement
Level 1
Location;
Attributes
Behavior;
Class; ID
Situation
Refinement
Level 2
Aggregate
object
refinement
Implications/
Threat
Refinement
Level 3
Situation
Intent;
interpretation Vulnerability
Courses of
Action
Database Services
Relatively static
a priori
Knowledge
Dynamic
Situation
Database
CACI Private Data; Do Not Copy or Distribute
Richard Antony, in
DFG Meeting
Minutes, W. Doig,
14 March 1997
Antony & Karakowski
Contextual Fusion Model (CFM) 2009
Conceptually organized along three(related) dimensions:
(entity, context1, context2)
AKA “Triple”
“Fusion” ..an “assessment” operation between pairs of Triples:
Lead to 8 fundamental classes of fusion operations
“Fusion as a Process” exhausts all possible “assessment”
combinations or fusion in a Triple; the result is a set of
discovered concepts & relations from the fusion of the pairs of
Triples. This provides a rich discovery space within an
existing knowledge source
CFM explicitly fuses diverse context with specific basic
entities, all within a computational JDL model framework,
resulting in a testable, expandable, and general fusion model
CACI Private Data; Do Not Copy or Distribute
Contextual Fusion Model
Context & Content
• Information for fusion requires both context and entity
• “Entity” is the specific unit of information, or node in a
graphical representation
• Context allows perception of an Entity with respect to the
information of interest
– Context gives meaning to an Entity’s “information”
– Context is required before an information entity has any
meaning
– Context must be an integral part of the fusion process
& process model, its computation paradigm
Context is knowledge that enhances the more complete
understanding of a specific entity of interest and the
desired resultant objective information product (s)
12
CACI Private Data; Do Not Copy or Distribute
Two Entities within a graph, with Two sets of
Two Entities, as their Contexts
Entities as
Context
E6/C6
E5/C5
E4/C4
E3/C3
Graph
Entity-Entiy
Relations
Entity
Graph
Nodes
E1
E2
Green = Entity
CACI Private Data; Do Not Copy or Distribute
Eight Canonical Fusion Forms
Intelligence Traditional Tracking/Correlation Application
Fusion Operation = fusion of two entities with associated context
Fusion Operation = (entity A, location A , time A) x (entity B, location B, time B)
Contextual Dimensions
Level 1
Fusion
Form
Individual
Entity
Location
Time
1
Similar
Similar
Similar
2
Similar
Similar
Dissimilar
3
Similar
Dissimilar
Similar
4
Similar
Dissimilar
Dissimilar
5
Level 2
Fusion
6
7
8
Dissimilar
Dissimilar
Dissimilar
Dissimilar
Similar
Similar
Dissimilar
Dissimilar
Similar
Dissimilar
Similar
Dissimilar
Similarity/Dissim Assessment Op
Example Concept of Fusion Result
formed for Location and time context
Fuse multiple, similar information sources;
tracking using E-sensors
Static similar object
Infeasible condition – inconsistency; a truth
maintenance signal
Entity tracking/ tracking using both (slower) Esensors and messages
Association / correlation of possible action as colocated Entities
Association / correlation of entities based on
same location only
With prior Communication info: Potential Entity
comm link (cell phone, chat, email)
Multiple different entities at different times- further
data mining and other FF conceptual
analyses may be indicated
CACI Private Data; Do Not Copy or Distribute
14
Simple Physical Context
Example - Voltage
If a voltage(entity) is viewed by itself
without any context, we just “see” a
value, either static, “semantic” or
If voltage has added the context of
apparently varying
time, “signals” are created, with the
Voltage
field of electrical electronic
engineering and associated signal
analysis….
Note the huge information content
difference between the entity of
“voltage” and the addition of the
context “time” and how context
gives much more “meaning” to the
entity (voltage)
time
.
Human Entity, for example, a much more complex entity…. This is
like a generalization from “humans” to “human signals”
CACI Private Data; Do Not Copy or Distribute
Eight Canonical Fusion Forms
Source “Voltage” (just for fun)
Fusion Operation = ( Source V1, location V1 , time V1) x (Source V2, location V2, time V2)
Contextual Dimensions
Level 1
Fusion
Form
V-Source
Compare
Location
Time
Concept of Fusion Result of Voltage Source
for Location and time context
1
Similar
Similar
Similar
Instantaneous single source value, at one place
2
Similar
Similar
Dissimilar
Time varying single source value, at one place
3
Similar
Dissimilar
Similar
4
Similar
Dissimilar
Dissimilar
5
Dissimilar
Similar
Similar
Instantaneous V-field for single source
Instantaneous time-varying V-field for single
source
Instantaneous multiple source value, at one place
Level 2
Fusion
6
Dissimilar
Similar
Dissimilar
7
Dissimilar
Dissimilar
Similar
8
Dissimilar
Dissimilar
Dissimilar
Time varying / multiple sources value, at one
place
Instantaneous V-field for multiple sources
Instantaneous time-varying V-field for multiple
sources
16
CACI Private Data; Do Not Copy or Distribute
Prior Art: Military Target Entity Model
DRs of
Organization
to
Organization
DRs of
Organizations
to Events
Level 2
Organizations
DR: Discovered
Relations thru
contextual FFs
DRs of
Individuals to
Organizations
Level 2
DRs of
Events to
Events
Events
All Relations
based on
Location & Time
Context Only
DRs of
Individuals
to
Individuals
Individuals
DRs of
Individuals
to Events
Level 2
Other forms of discovery are possible;
(I O, OE )  (EI ) eg
CACI Private Data; Do Not Copy or Distribute
17
Fusion Applications
CACI Private Data; Do Not Copy or Distribute
Some Fusion Military Application Areas
•
•
•
•
•
•
Intelligence
Bio sensing/biometrics Can support at all
levels: Hardware,
Situation Awareness
Software, Level 0 –
Level 5
Imagery
SIGINT(COMINT/ELINT)
Tracking
CACI Private Data; Do Not Copy or Distribute
Non-Military Fusion Application Areas
•
•
•
•
•
•
Networking/Cellular
Homeland Security
Medicine
Chemistry
Cognitive sciences
…many others…
CACI Private Data; Do Not Copy or Distribute
Fusion “Topics” from a
recent conference…
CACI Private Data; Do Not Copy or Distribute
Conference [Shortened-”C”]
Index of Fusion Topics I
•
•
•
•
•
•
•
•
•
•
•
Camera
Capability Acquisition Graph
CBRN data fusion
Cellular automata
Centralized processing
systems
Challenge Problem Set
Change detection
Chemical plume
Classification fusion
Classification System
Closest point approach
•
•
•
•
•
•
•
•
•
•
•
•
Clustering algorithm
Clutter
Co-ranking
Coalition formation
Coalition operations
Coarsening
Coastal radar
Cognitive Radio Networks
Collaborative systems
Collision mitigation
Color Clustering
……
CACI Private Data; Do Not Copy or Distribute
Conference [Shortened-”C”]
Index of Fusion Topics II
•
•
•
•
•
•
•
•
•
•
Combination of belief
•
functions
•
Combinatory categorial
grammar
•
Communication Decision •
Communication failures
•
Complex object recognition •
Compression
•
•
Computer security
•
Conceptual graphs
•
•
Conditional independence
Confidence management
Configuration
Conflict analysis
Confusion
Conjunctive operator
Connection Model
Context
Contradiction
Convex optimization
Convoy tracking
Cooperative systems
Coordinate registration
CACI Private Data; Do Not Copy or Distribute
Conference [Shortened-”C”] Index
of Fusion Topics III
•
•
•
•
•
•
•
•
•
•
•
•
•
Coordination
Correlation
Course Of Action
Covariance control
Credal networks
Credibility
Crop modeling
Cross correlation
Cross-cueing
Cubic Spline Curve
Cued Sensors
Cyber fusion
Cyber-security
From these three slides
one can see both very
specialized areas and
much broader areas
which currently utilize
information Fusion
technology
CACI Private Data; Do Not Copy or Distribute
Overview of Specific IF Apps from
Selected market areas
Military
1…Biometrics
2…Target Detection & Tracking
3…Chemical & Explosives
4…Image Fusion
Medical
Note: All these apps
will fall somewhere in
the fusion models and
fusion definitions
which I previously
described.
5…Breast Cancer
6…Radiology
Non-Military
7…Dept of Homeland Security
8…Cyber Security
Summaries of specific fusion papers follows…
CACI Private Data; Do Not Copy or Distribute
1. Biometrics
A Multibiometric Face Recognition Fusion
Framework with Template Protection[1]
• “A fusion framework.. which demonstrates how …algorithms that
produce hard decisions can be combined with unprotected
algorithms that produce scores or soft decisions”
Improving the recognition of fingerprint biometric
system using enhanced image fusion[2]
•“approach to increase the verification and identification of fingerprint
recognition. This was achieved by using … linear fusion techniques”
Multimodal Eye Recognition[3]
• “results show that the proposed eye recognition method can
achieve better performance…, and the accuracy of…kernel-based
matching score fusion methods is higher than PCA and LDA”
Military & Commercial
CACI Private Data; Do Not Copy or Distribute
2. Target Detection &Tracking
Level 0-2 fusion model for ATR using fuzzy
logic[4]
• “use of fusion at the lowest levels has been demonstrated.
…provides a structure for fusion of multispectral data at all
levels”
Long-duration Fused Feature Learning Aided
Tracking[5]
• “Our experiments indicate that the Long-term Hypothesis Tree
algorithm, which solves the tracklet-to-tracklet association problem,
can be used to strongly disambiguate a multitude of situations and
is a more computationally efficient algorithm than previously
proposed joint solutions”
Military
CACI Private Data; Do Not Copy or Distribute
3. Chemical & Explosives
Fusing chlorophyll fluorescence and plant canopy
reflectance to detect TNT contamination in
soils[6]
• “physiological response of plants grown in TNT contaminated soils
and … to detect uptake in plant leaves…use remote sensing of plant
canopies to detect TNT soil contamination prior to visible signs”
Sensor data fusion for spectroscopy-based
detection of explosives[7]
• “Multi-spot fusion is performed on a set of independent samples from
the same region…. Furthermore, the results … are fused using linear
discriminators. Improved detection performance with significantly
reduced false alarm rates is reported using fusion techniques”
Military Market
CACI Private Data; Do Not Copy or Distribute
4. Image Fusion
Towards Visual-Data Fusion[8]
• “Fusion for both data and visual processes are derived as
specific transforms from human linguistic requests. Visual
“understanding” occurs by human-directed perception of
summarized pattern representations within a familiar frame of
reference”
An orientation-based fusion algorithm for
multisensor image fusion[9]
• “Gabor wavelet transform … to fuse visible images and thermal
images; orientation-based fusion is superior to the results of multiscale
fusion algorithms…and can be applied to multiple (more than two)
image fusion”
Military & Commercial
CACI Private Data; Do Not Copy or Distribute
5. Breast Cancer
Investigation of PET/MRI Image Fusion Schemes
for Enhanced Breast Cancer Diagnosis[10]
• “results indicate that the radiologists were better able to perform a
series of tasks when reading the fused PET/MRI data sets using
color tables generated by our new genetic algorithm, as compared
to commonly used …schemes”
Time of Arrival Data Fusion Method for TwoDimensional Ultrawideband Breast Cancer
Detection[11]
•“A new microwave imaging method is given for breast tumor detection
using an ultrawideband (UWB) imaging system. By combining the time
of arrival (TOA) measurements from different sensors, the presence and
location of small malignant lesions can be identified”
Medical
CACI Private Data; Do Not Copy or Distribute
6. Radiology
KNOWLEDGE BASED FUZZY INFORMATION
FUSION APPLIED TO CLASSIFICATION OF
ABNORMAL BRAIN TISSUES FROM MRI[12]
• “automatically classify abnormal tissues in human brain in a three
dimension space from multispectral magnetic resonance images
such as TI-weighted. T2- weighted and proton density feature
images. It consists of four steps: data matching. information
modeling, information fusion and fuzzy classification”
New Applications of Planar Image Fusion in
Clinical Nuclear Medicine and Radiology[13]
• Fusion of multiple modalities has become an integral part of
modern imaging methodology, especially in nuclear medicine where
PET and SPECT scanning are frequently paired with computed
tomography(CT). Additional fusing of orthopedic radiographs with
photographic images of the extremities..
Medical – Add
CACI Private Data; Do Not Copy or Distribute
7. DHS
Military & Non-Military
Information Fusion for CB Defense Applications[14]
• “With appropriate algorithmic approaches and appropriately resolved
tradeoffs, information fusion can offer… the potential of reaching
performance that would be difficult, if not impossible, to attain
otherwise. Thus, information fusion represents a significant
opportunity for the CB defense and homeland security realm”
Decision-level Information Fusion to Assess Threat
Likelihood in Shipped Containers[15]
•“details an approach to the decision-level fusion of disparate
information to produce an assessment of the presence of a threat in a
shipping container”
Homeland Security Fusion Application of STEF[16]
•“fusion system provided sufficient actionable intelligence that could
have stopped a .. realistically staged terrorist attack on a US civilian
target. …provided sufficient information to allow .. arresting the
mastermind of the plot, as well as other key individuals and detaining
the lower level individuals in his network, including the suicide bomber”
CACI Private Data; Do Not Copy or Distribute
8. Cyber Security
Application of the JDL Data Fusion Process
Model for Cyber Security[17]
•
“explores the underlying processes identified in the Joint Directors
of Laboratories (JDL) data fusion process model and further
describes them in a cyber security context”
Military & Non-Military
CACI Private Data; Do Not Copy or Distribute
We have covered “the more
important parts”…a warning
CACI Private Data; Do Not Copy or Distribute
Summary & Closing
Comments
• Short background of Fusion Technology
& Models/Contextual Fusion Model
• Few examples of Fusion R&D / Apps
• A lot was left out ….
CACI Private Data; Do Not Copy or Distribute
Backups
CACI Private Data; Do Not Copy or Distribute
Some of My Personal Fusion
RDE Areas
• UGS tracking/ID L0/L1
• RADAR ID
– Signal processing L0/L1 / Fuzzy Expert
– Confirmation/Disconfirmation
• Voice Fingerprint ID biometrics L0/L1
• Visual fusion L0-L2[*]
• Semantic/contextual unstructured information -understanding & discovery L1-L3[*]
• Contextual Fusion System[2006-2010]
• General Context Fusion Model [2011-?]
CACI Private Data; Do Not Copy or Distribute
Some Fusion Publications
•
•
•
•
•
•
•
•
•
•
•
Karakowski, J.A., “An Application of Text-Independent Speaker Recognition to High
Speed Voice Surveillance”, Wide Area Surveillance Symposium, Office of Nat’l Drug
Control Policy/Counter Drug Technology Assessment Center(1993)
Karakowski, J.A., “Text Independent Speaker Recognition using A Fuzzy Hypercube
Classifier”, ICASSP97(1997)
Karakowski, J.A., “Towards Visual Fusion”, Invited Paper, Georgia Tech(1998).
Antony, R. T. and Karakowski, J. A., “Service-Based Extensions to the JDL Fusion
Model,” SPIE Defense Security and Sensing Conference (March 2008).
Antony, R. T. and Karakowski, J. A., “Fusion of HUMINT & Conventional Multi-Source
Data,” National Symposium on Sensor and Data Fusion, Session SC04 pp. 1-16 (07).
Antony, R. T. & Karakowski, J. A., (2007) “Towards Greater Consciousness in Data
Fusion Systems,” MSS National Symposium on Sensor and Data Fusion, (June 07).
Antony, R. T. and Karakowski, J. A., “First-Principle Approach to Functionally
Decomposing the JDL Fusion Model: Emphasis on Soft Target Data,” Fusion (July 08).
Antony, R. T. & Karakowski, J. A., “Discovery Tools for Soft Target Applications,”
National Symposium on Sensor and Data Fusion(2009)
Antony, R. T. and Karakowski, J. A., “First-Principles Mapping of Fusion Applications
into the JDL Model,” SPIE Defense Security and Sensing Conference (April 2009)
Antony, R.T, & Karakowski, J.A., “Multiple Level-of-Abstraction Tracking and Alias
Resolution”, National Symposium on Sensor and Data Fusion(2010)
Antony, R.T., & Karakowski, J.A., “Toward more Robust Exploitation of the Asymmetric
Threat: Binary Fusion Class Extensions”, (April 2011) SPIE.
CACI Private Data; Do Not Copy or Distribute