Transcript Document 7470777

SAMI: Situational Awareness
from Multi-modal Input
Naveen Ashish
Talk Organization






Why are we at RESCUE interested ?
Situational Awareness (SA)
– Introduction
System architecture
Research challenges
Expected outcomes and artifacts
Extraction system demonstration
Team
Naveen Ashish
Sharad Mehrotra
Nalini
Venkatasubramanian
Utz Westermann
Dmitry Kalashnikov
Stella Chen
Vibhav Gogate
Priya Govindarajan
Ram Hariharan
John Hutchinson
Yiming Ma
Dawit Seid
Jay Lickfett
Chris Davision
Quent Cassen
Bhaskar Rao
Mohan Trivedi
Rajesh Hegde
Sangho Park
Shankar Shivappa
Ron Eguchi
Mike Mio
Jacob Green
Information from Various Sources
News, video, audio footage
Pushing “Human-as-sensor”
Emergency responders
People/Victims at disaster
GIS, satellite imagery, maps
More Data ≠ More Information
Whereall
are
the
fire
What
Have
areas
medical
should
supplies
we
personnel
reached
start
evacuating
? ?
first ?
SA
Situational Awareness


Wide variety of fields
– Beginning in mid-80s, accelerating thru 90s
– Fighter aircraft, ATM, Power plants, Manufacturing
Definitions
– "the perception of elements in the environment along with a comprehension
of their meaning and along with a projection of their status in the near
future"
– "the combining of new information with existing knowledge in working
memory and the development of a composite picture of the situation along
with projections of future status and subsequent decisions as to appropriate
courses of action to take"
Knowing what is going on


Situational awareness and decision making
Areas
– Cognitive science
– Information processing
– Human factors
Abstraction of Information
Awareness
Events
Multimodal Input:
Text, Audio, Video
First-cut Architecture
Text
Audio
EVENT
EXTRACTION
VISUALIZATION and USER
INTERFACES
Video
Internet
KNOWLEDGE: ONTOLOGIES
REFINEMENT
Disambiguation
Location
Graph View
Spatial
Indexing
PDF
Histogram
Querying and
Analysis
RAW DATA
EVENT BASE
Centered around EVENTS as fundamental abstractions
Research Areas
Event Modeling
Event Extraction
Disambiguation
GIS Querying
Location Uncertainty
Graph Analysis
Event Modeling


What is an event ?
Event Representation
TIME
LOCATION
TYPE
PEOPLE
REPORT
EVACUATION
RELIABILITY
AGENCY
NAME
LOCATION
OPERATION
FROM
TO
NUMBER
Domain Knowledge
THAILAND
EVACUATION
IS-A
…….
IS-A
ROAD
EVACUATION
SOUTHERN
REGION
AIR
EVACUATION
PHUKET,
CHANGWAT

Captured as Ontologies
PHUKET
Event Extraction





Long history of information extraction
– IR (MUC efforts)
– Web data extraction
DARPA ACE
– Entities, Relations, Events
– Events in 2004
Event extraction accuracy is still low
SA Domain
– Stream of information
– Duplicated, ambiguous
– Reliability
– Conversations
Modalities
– Text
Semantics Driven Approach


Semantics Driven
Challenges
– Framework
– Ontologies


What semantics required for event extraction ?
Application
 With NLP, ML techniques

Performance
– SA specific

Duplicates, reconciliation, temporal, conversations …..
Disambiguation
Disambiguation
Uncertainty is a Challenge
Report 1: “... a massive accident involving a hazmat truck on
I5-N between Sand Canyon and Alton Pkwy ...”
Report 2: “... a strange chemical smell on Rt133 between I405
and Irvine Blvd ...”
– point-location
 in
terms of landmarks
 uncertain,
not (x,y)
– reasoning on such data
 support
all types of queries
Implications of Uncertainty in Text
How to model uncertainty?
– probabilistic model
– P(location | report)
 e.g. report says “near building A”
Queries
– cannot be answered exactly...
 use probabilistic queries
 all events: P(location  R |
report) > 0
– SA requirements
 triaging capabilities
 fast response
– top-k
– threshold: P(location  R |
report) > 
– -RQ, k-RQ, k -RQ
How to map text to probabilities?
– use spatial ontologies
A
R
B
Graph Analysis



GAAL
Inherent spatio-temporal
properties
Graphs are powerful for
querying and analysis
GIS Search
Current FGDC Search
GIS Search
Progressive Refinement of Data
Deliverables, Outcomes, Artifacts




“Vertical” thrusts
– Event extraction system (TEXT)
– Disambiguation system
– GIS search system
Overall system demonstration ?
“By-products”
– Ontologies
Computer science research areas
Databases
Semantic-Web
Information Retrieval
Intelligent Agents (AI)
http://sami.ics.uci.edu
Thank you !