Transcript Significant research accomplishments already made

MURI: Computer-aided Human
Centric Cyber Situation Awareness
Peng Liu
Professor & Director, Lions Center
Pennsylvania State University
ARO Cyber Situation Awareness MURI
1
Team
•
•
•
•
•
•
Peng Liu, Professor and Director, Penn
•
State Center for Cyber-Security,
Information Privacy and Trust
Massimiliano Albanese, Assistant
Professor, GMU
•
Nancy Cooke, Professor and Science
Director, Arizona State Cognitive
Engineering Research Institute
•
Coty González, Associate Research
Professor and Director, CMU Dynamic
•
Decision Making Lab
Dave Hall, Professor and Dean, Penn State
College of IST
•
Christopher Healey, Professor, NC State
•
# of graduate students: 16
Sushil Jajodia, University Professor
and Director, George Mason Univ.
Center for Secure Information
Systems
Mike McNeese, Professor and
Associate Dean, Penn State College
of IST
Peng Ning (on leave), Professor,
NCSU
Douglas Reeves, Professor and
Interim Assistant Dean for COE
Graduate Programs, NCSU
VS Subrahmanian, Professor and
past Director, U. of Maryland
Institute for Advanced Computer
Studies
John Yen, University Professor and
Director, Intelligent Agents Lab
# of post docs: 3
2
ARO MURI: Computer-aided Human Centric Cyber Situation Awareness
PSU, ASU, CMU, GMU, NCSU, UMD
Contact: Peng Liu, Tel. 814-863-0641, E-Mail: [email protected]
Objectives: Improve Cyber SA through:
• Cyber SA specific cognition models
• Cognition-friendly tools and analytics that fill the
gap between the sensor side and the analyst side of
cyber SA
• Cross-layer situation knowledge integration
DoD Benefit:
• Significantly improved capabilities in gaining cyber
SA in face of cyber attacks
• Significantly improved job performance of analysts
Scientific/Technical Approach
•Take a holistic approach to integrate the “human
cognition” aspects and the “cyber tools” aspects of
cyber SA
•Leverage cognition models to develop human
cognition-friendly SA techniques, tools, and
analytics
Accomplishments
• Year 5: See slide 5
Challenges
• Understanding the mental processes of analysts
• Team integration
3
• Simulation
• Measures of SA & Shared SA
Automated
Reasoning
Tools
Information
Aggregation
& Fusion
• R-CAST
• Plan-based
narratives
• Graphical
models
• Uncertainty
analysis
• Transaction
Graph
methods
•Damage
assessment
Computer network
Real
World
Multi-Sensory Human
Computer Interaction
•
•
•
Data Conditioning
Association & Correlation
Cognitive Models & Decision
Aids
• Instance Based Learning Models
• Enterprise Model
• Activity Logs
• IDS reports
• Vulnerabilities
Security Analysts
Testbed
•
•
Computer
network
•
4
Year 5 accomplishments
Research:
-- Major achievements made
-- See individual presentations
Technology
transitions:
-- See slides later on
Pub:
-- 37 (1 book, 10 journals, 20 conf., 6
Tools:
chapters) (Y1-Y5 total: 170+)
-- ARSCA  Shift Transition
-- MetaSymploit
-- NETS simulator
-- DEXTAR
-- Patrol  X-Ray
-- Switchwall
-- NSDMiner
-- CyberCog
-- PASS  PADUA
-- CAULDRON
-- etc.
-- 2 PhD thesis
-- 7 presentations
Deep collaboration with ARL:
-- 11 ARL security analysts
-- 5 researchers at ARL
-- 4 joint publications
Awards:
-- Max Albanese received the 2014
Mason Emerging
Researcher/Scholar/Creator Award
5
Cyber Operations for Mission Assurance
Sensors,
probes
What has happened?
•
•
•
What is the impact?
Why did it happen?
Computer
networks
(e.g., GIG)
What should I do?
Security Analysts
6
Cyber Situation Awareness
What has happened?
What is the impact?
Why did it happen?
What should
I do?
Enabler
Core Cyber SA
7
Cyber SA Info Processing Box
Attacks
The
Network
Data
Sources
(feeds)
Depicted
Situation
Compare
Ground
Truth
(estimates)
Job
Performance
8
Why Research is Needed?
20+ CNDSPs*, whose operations are relying on human
analysts, face critical challenges:
1. Job performance is unstable
2. Hard to get the big picture: walls between functional
domains
3. Better analytics and tools are needed to improve job
performance
* In the commercial world, similar issues exist.
9
State of the Art: Big Gap Exists
Current tools:
• Ability to create problemsolving workflows
• To see big picture
• To manage uncertainty
• To reason albeit
incomplete/noisy knowledge
• To quickly locate needles in
haystacks
• To do strategic planning
• To predict
• ……
BIG GAP
Desired cyber SA capabilities:
Vulnerability scan
Event logging
Traffic classifying
Intrusion detection
Alert correlation
Signature gen.
Taint analysis
Back tracking
Integrity check
Static analysis
Bug finding
Attack graphs
Symbolic execution
Sandbox
VM monitors
10
…
Scientific Objectives
Develop a deep understanding on:
1. Why the job performance between expert and
rookie analysts is so different? How to bridge the
job performance gap?
2. Why many tools cannot effectively improve job
performance?
3. What models, tools and analytics are needed to
effectively boost job performance?
Develop a new paradigm of cyber SA system design,
implementation, and evaluation.
11
Scientific Barriers
A. Massive amounts of sensed info vs. poorly used by
analysts
B. Silicon-speed info sensing vs. neuron-speed human
cognition
C. Stovepiped sensing vs. the need for "big picture
awareness"
D. Knowledge of “us”
E. Lack of ground-truth vs. the need for scientifically
sound models
F. Unknown adversary intent vs. publicly-known
vulnerability categories
12
Potential Scientific Advances
Understand the nature of human analysts’ cyber SA
cognition and decision making.
Let this nature inspire innovative designs of SA systems.
Break both vertical stovepipes (between compartments)
and horizontal stovepipes (between abstraction layers).
“Stitched together” awareness enables advanced
mission assurance analytics (e.g., asset map, damage,
impact, mitigation, recovery).
Discover blind spot situation knowledge.
Make adversary intent an inherent part of SA analytics.
13
Scientific Principles
Cybersecurity research shows a new trend: moving from
qualitative to quantitative science; from data-insufficient
science to data-abundant science.
The availability of sea of sensed information opens up
fascinating opportunities to understand both mission and
adversary activity through modeling and analytics. This will
require creative mission-aware analysis of heterogeneous
data with cross-compartment and cross-abstraction-layer
dependencies in the presence of significant uncertainty and
untrustworthiness.
SA tools should incorporate human cognition and decision
making characteristics at the design phase.
14
Why a Multidisciplinary Approach?
Several fundamentally important research questions
cannot be systematically answered by a singledisciplinary approach.
See next slide.
15
Q1: What are the differences
between expert analysts and
rookies?
Computer and
Information Science
of Cyber SA
Q2: What analytics and tools
are needed to effectively boost
job performance?
Q3: How to develop the better
tools?
Cognitive Science
of Cyber SA
Our
focus
Decision Making
and Learning Science
16
of Cyber SA
Technical Approach
Draw inspirations from cognitive task analysis,
simulations, modeling of analysts’ decision making, and
human subject research findings.
Use these inspirations to develop a new paradigm of
computer-aided cyber SA
Develop new analytics and better tools
Let tools and analysts work in concert
“Green the desert” between the sensor side and the
human side
Develop an end-to-end, holistic solution:
In contrast, prior work treated the three vertices of the
“triangle” as disjoint research areas
17
The proposed cyber SA framework
It is a ‘coin’ with two sides:
 The life-cycle side


Shows the SA tasks in each stage of cyber SA
Vision pushes us to “think out-of-the-box” in performing
these tasks
 The computer-aided cognition side


Build the right cognition models
Build cognition-friendly SA tools
18
Perception
Comprehension
Projection
• Simulation
• Measures of SA & Shared SA
Automated
Reasoning
Tools
Information
Aggregation
& Fusion
• R-CAST
• Plan-based
narratives
• Graphical
models
• Uncertainty
analysis
• Transaction
Graph
methods
•Damage
assessment
Computer network
Real
World
Multi-Sensory Human
Computer Interaction
•
•
•
Data Conditioning
Association & Correlation
Cognitive Models & Decision
Aids
• Instance Based Learning Models
• Enterprise Model
• Activity Logs
• IDS reports
• Vulnerabilities
Security Analysts
Testbed
•
•
Computer
network
•
19
Situation Knowledge Abstraction Perspective
Mission
Workflows
App, Net
Services
Reeves
Jajodia,
Albanese
Subrahmanian
Vulnerability
Gonzalez, Exploits
Cooke
Alerts
Yen,
Healey
OS
Liu: integration
McNeese & Hall:
multi-level
cognition
CPU
and fusion
20
Impact on DoD
Significantly enhance mission assurance through:
1. Significantly improving the job performance of
CNDSPs
2. Developing cognition-friendly SA tools to effectively
improve job performance
• Situation knowledge integration
• Situation knowledge discovery & elicitation
• Reasoning assistants, decision aids
• Better interfaces, better shift transitions
21
Y5 Team Integration
Within each theme:
• Collaboration is pervasive
• Collaboration is further deepened
• Joint research tasks
• Co-authored papers
• Tool-level integration in progress
Between themes:
• Integration along the functional perspective
• Integration along the knowledge abstraction
perspective
• E.g., Jajodia & Cooke, Coty & Cooke, Hall &
McNeese & Liu, Healey & Hutchinson, Yen & Cam
& Erbacher & Glodek & Hutchinson & Liu, Jajodia
22
& Albanese & Cam & Yen & Liu
Technology Transfer (1)
Partner: ARL
Contact: Rob Erbacher, Bill Glodek, Steve Hutchinson, Hasan
Cam, Renee Etoty, Chris Garneau
Focus: Collect the cognitive traces of CNDSP analysts
Status: -- Over two years
-- Over 30 traces collected
-- ARSCA tool is being used at ARL
-- Weekly teleconferences
-- In discussion: directly operate on ARL datasets
23
Technology Transfer (2)
Partner: ARL
Contact: Rob Erbacher, Bill Glodek, Steve Hutchinson
Focus: Shift transitions
Status: -- A user study on shift transition fully designed
-- IRB developed and approved
-- ARSCA-shift-transition tool developed
-- Shipped to ARL site and tested there
-- Pilot study is being scheduled
24
Technology Transfer (3)
Partner: ARL
Contact: Hasan Cam
Focus: Enhance the ARL petri-net model for impact assessment
-- feed outputs of CAULDRON and ARSCA into petri-net
Status: -- Proposal developed and approved
-- Just started (Nov 2014)
-- First experiment sketched
25
Technology Transfer (4)
Partner: ARL
Contact: Rob Erbacher, Christopher Garneau
Focus: (a) Investigate how the current practice of training
professional CNDSP security analysts can be enhanced
by leveraging ARSCA.
(b) A pilot study for investigating the feasibility of using
ARSCA-facilitated training procedures for supporting the
training of analysts about their analytical reasoning
process.
Status: -- Proposal developed and approved
-- Just started (Nov 2014)
-- Weekly teleconferences
26
Technology Transfer (5)
Partner: ARL
Contact: Christopher Garneau, Rob Erbacher
Focus: Human subject experiments on the cognitive effects of
different (visualization) views
Status: -- IRB developed and approved
-- User study fully designed
-- Pilot study being scheduled at Penn State
27
Tech Transfer (6)
Phase II STTR: Cooke group has been working with Sushil
Jajodia and Max Albanese (George Mason and fellow
MURI PIs) on an STTR that involves a higher fidelity
version of CyberCog, DEXTAR, in which we will integrate
CAULDRON.
- Phase II STTR through Sandia Research Corporation
- AFRL has shown interest in the test-bed we have been
developing on the Phase II STTR
Cooke group has been working on SBIR for AFRL with
Charles River Associates that involves team sensors for
cyber analysts.
28
Technology Transfer (7)
Partner: AFRL – Human Effectiveness Directorate
711th Human Performance Wing, Wright-Patterson AFB, OH
Contact: Benjamin Knott and Vince Mancuso
Focus: Human performance and measurement of cognition
Partners: Deloitte, Ernst and Young, KPMG, Price Waterhouse Coopers
Contacts: J.B. O’Kane (Vigilant by Deloitte), Jenna McAuley (EY-ASC) and others
Focus: Observe practicing analysts, test visualization toolkits and fusion tools,
measure human cognition and performance
Partner: MIT Lincoln Laboratories
Cyber Security Information Sciences Division
Contact: Stephen Rejto and Tony Pensa
Focus: Conduct human-in-the-loop experiments; evaluate MIT-LL/PSU analyst tools
29
Technology Transfer (8)
Partner:
Contact:
Focus:
Status:
NIST
Anoop Singhal
Gain awareness of stealthy info bridges in a cloud
-- One research work done
-- One NIST technical report produced
-- Paper published
Partner:
Contact:
Focus:
Status:
NEC Labs America, Inc.
Z. Qian, Z. Li
Discover long-running Idling processes in enterprise systems
-- One research work done
-- A real enterprise environment (on 24 hosts)
-- In-depth measurement study
-- Paper submitted
Partner:
Contact:
Focus:
Status:
IAI, Inc.
Jason Li
System call level enterprise cyber SA
-- A new research work done
-- One PhD dissertation
30
Tech Transfer (9)
Ethnographic studies/knowledge elicitation with network analysts
working in education, military, government, and industry domains.
Briefings provided to several companies including: Deloitte,
Lockheed Martin, Raytheon Corporation, MITRE, Computer
Sciences Corporation, and MIT Lincoln Laboratory.
Briefings to NSA, DTRA, ONR, DHS, and DoDII.
Neville Stanton, University of Southampton is the developer of EAST
modeling and is collaborating with Buchanan and Cooke on this
form of modeling applied to cyber.
31
No-Cost Extension Plan
Each PI has a research plan from their
perspectives: see the individual presentations
Set-aside project 1 with ARL
Set-aside project 2 with ARL
Team integration exercises will be held
32
Q&A
Thank you.
33