Overview of Cyber Experimentation & Test Ranges ICOTE
Download
Report
Transcript Overview of Cyber Experimentation & Test Ranges ICOTE
Overview of Cyber Experimentation & Test Ranges
ICOTE
September 25 2012
William C. Liu
Section Lead
Cyber Operations & Networking Group
BAE Systems – Technology Solutions
Arlington VA 22203 USA
Dr. Kevin M. McNeill
Engineering Fellow & Technical Director
Cyber Operations & Networking Group
BAE Systems – Technology Solutions
Arlington VA 22203 USA
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
1
Problem Domain - overview
• Software is in everything!
• Functionality defined by software (cyber)
but tight coupling to physical (kinetic) world
• Large-scale, complex cyber-physical systems
and systems-of-systems across a wide
variety of application domains
• Such systems deployed for mission-critical
operations in many domains, e.g.:
• Defense
• Critical Infrastructure
• Health-care
• Finance
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
Source: IEEE Computer Magazine 2009
• Design flaws, unintended bugs, vulnerabilities
or attack have physical effects
2
Problem Domain – Broad Challenge
• General purpose methods and tools supporting development of the software
(cyber) element of complex cyber-physical systems fail to a)
b)
c)
d)
provide assurance for mission-critical functions
provide assurance the system will satisfy safety and reliability requirements
provide assurance the system will support scalability and adaptability demands
provide support for the analysis of vulnerabilities to offensive cyber operations
• a), b) & c) belong to a broad software engineering challenge for test and
validation that is beyond the scope of this presentation!
d) introduces additional challenges for test and validation that must
be addressed by Cyber Experimentation & Test Ranges
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
3
Problem Domain – Cyber Testing & Validation
• General technical approach to testing, validation & verification of these cyberphysical systems are not adequate
• Especially with respect to analysis related to cyber operations
• Common methods and techniques do not support –
•
•
•
•
Rigorous scientific/engineering methodologies
Rapid configuration and adaptation of test environments
Flexibility of test environments to accommodate new technologies
Automation of test processes
• The result is that cyber test ranges are often expensive, unreliable and quickly
become obsolete
This is the testing challenge that is the focus of this presentation!
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
3
Why is it a problem?
• New generations of distributed ultra-large scale, systems of systems…
• Global Enterprise Networks; Command & Control Systems; Smart-Grid; Next
Generation Air Traffic Control; Cyberspace operations…
• … are complex systems…
•
•
•
•
•
Often built from rapidly evolving open-source, Internet-based services
Driven by user demands for rapid evolution (or changing mission requirements)
Have complex configuration options
Require a capability to respond to unexpected situations/environmental factors
Contain unknown software vulnerabilities and are often accessible from around
the globe via Internet
• Rigorous testing, validation & verification of these systems based on scientific
& engineering principles is not supported by legacy tools
• Cost (time) to develop, predictability of behavior in unexpected conditions, safety,
reliability, identification of vulnerabilities, verification of requirements
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
4
Focus on Cyber Experimentation & Test Ranges
The growth of cyber intrusions presents a key challenge to national and
international scale enterprises in their deployment and operation of large-scale
cyber-physical systems
• Cyber experimentation & test ranges are used to understand vulnerabilities of
computing and networking infrastructure
• Critical Challenges
• Building and operating these ranges is complex, costly, time-consuming and often
does not provide sufficient fidelity to provide any real value
• Cyber testing challenges are made even more complex by the nature of rapidly of
services provided by centralized service provides (e.g., ISP’s, ASP’s) that support
many business operations
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
5
Cyber Experimentation & Test Challenges (1)
Live testing on the operational or development networks of an Enterprise is not
recommended due to risks to business operations
• Therefore, cyber testing must be conducted on highly isolated environments
• To build a successful, useful & maintainable cyber experimentation and test
range it is necessary to address key challenges • Scope – Identify the scope of the experimentation & test to size and equip the
range infrastructure appropriately (do you need a few servers & VM’s or a data
center?)
• Central to estimating personnel and on-going costs (e.g., technology refresh, licensing
costs, training and operations)
• Replication – Enterprises use services provided by centralized providers,
potentially distributed across multiple data centers or provided by Cloud services
• It is not economically feasible to recreate these in a range with a high degree of fidelity
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
6
Cyber Experimentation & Test Challenges (2)
• key challenges (cont’d) • Proprietary – Environment under test may use proprietary software that is not
available for internal analysis or testing
• No private instances of the commercial services are allowed for analysis,
experimentation and test
• Licensing – Implementing a large-scale cyber range that to replicate an enterprise
for network vulnerability assessment may require a large number of virtual
machines running commercial software
• Vendors may require paying for a license for each instance active on the range
• Adaptability – Enterprises leverage new services that come online and become
important operationally very quickly and often change rapidly
• Cyber ranges must facilitate rapid integration or replication of new services to provide a
realistic test environment
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
7
Cyber Experimentation & Test Challenges (3)
• key challenges (cont’d) • Contamination – Resetting the range to a pristine state after testing must ensure
that no latent malware remains or can propagate off of the range
• Requires policies, procedures and technologies to verify the state of the range before,
during and after the test
• Sensors/Instrumentation and visualization of the range are critical
• This is a very hard problem
• Fidelity – Replicating the hardware, software and network environment in a way
that provides sufficient realism is necessary to ensure that the range produces test
results that are meaningful
• Difficult challenges relate to scale, replication of user traffic, emulation of user
behaviors, or use of non-standard hardware
• Operations & Personnel – Need to have a business model for the range “enterprise”
• Who are the “customers” for the range;
• What are roles and responsibilities for the range personnel;
• What is the cost/revenue model for ongoing maintenance and support;
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
7
Technologies to overcome these challenges (1)
Model-based Automation Frameworks
• To avoid complete duplication of an enterprise environment for testing cyber
range operators must use a combination of live/virtual/constructive testing
with significant emulation/simulation or analysis
• Solving this problem for a specific class of systems is feasible and well
supported by the using a model-based, adaptive testing framework
• Such a framework should support multiple levels of abstraction to accommodate
different testing requirements and phases
• It should be model-based (e.g., using domain-specific modeling languages) to
support rapid reconfiguration and update
• It should integrate with custom emulation tools for non-standard systems
• It should be implementing to use virtualization to a great extent in order to
facilitate scalability and cost management
Domain-specific modeling languages (DSMLs) allow definition of models for
individual aspects of cyber testing and support automation and rapid adaptation
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
8
Technologies to overcome these challenges (2)
Hierarchical Levels of Abstraction to Address Complexity
•
•
•
•
Isomorphic Testing – The cyber test range can be configured with analogous software
that can be shown to have features and behaviors that map onto those of the
application of interest
Behavioral Models - This level of abstraction may include some form of transaction
replay by a model of the application of interest
Simulation - For cyber testing this provides relevance in assessing scalability and
interaction of many actors across an large scale networking environment or assessing
Enterprise impact
Emulation – Provide a higher level of fidelity or adaptability and works well when
internals are a “black-box” and the interaction with the external network is important
Model-based Automation Frameworks, built with virtualization technologies and
supporting various levels of abstraction significantly reduce the time, cost and
complexity of conducting cyber experimentation and test exercises while supporting
scientific and engineering rigor
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
9
Automation of the test processes (1)
Define Test
Needs
Iterative
planning
meetings with
approvals and
requirements
analysis
Most cyber experimentation and testing is a
sequence of highly manual processes that are
very time consuming, costly and prone to
error
Plan Test
Informal
document driven
process with high
potential for
ambiguity (e.g.,
emails)
Configure Range
Define Test
Scenarios
Scenarios
created with
duplication in
stand-alone tools
Execute Test
Realize Test
Scenario
Mapping of scenario to real
components ia manual
mapping process
Log# ES-AVA-040912-0073
De-Configure
Range
Manual wiping
Approved for public release 0162; No Export Controlled Data
Physical
configuration and
hand crafting of
batch files for
initiation of
software
Plan is manually
implemented
Analyze Results
Bulk event feeds, uncorrelated
timing data
10
Automation of the test processes (2)
Define Test
Needs
“Human Modeler”
Supported by tools
that facilitate use of
machine-readable
materials
Plan Test
Built-in formalized
experiment
compliance,
flexible, support for
templates for ease
of use
Define Test
Scenarios
DSML support
intuitive look-andfeel, supports
“recipies” for
efficient reuse
Realize Test
Scenario
Can support full
automation while
always supporting
“human-on-theloo”
•
Using Domain Specific Modeling Languages (DSML) allows
model-based test tools to be tailored to specific user needs
•
Model representation supports re-use of test recipes,
enforcement of Information Assurance policies and the
application of scientific rigor
•
Transitions between steps become model transformations
that can be automated to increase R&D testing throughput
•
Test evolves from abstract concept to physical instantiation
Human intensive process
supported by automated data
reduction
Analyze Results
Configure Range
Execute Test
Always human involved process;
supported by modeling tools
that facilitate use of machinereadable materials
Automated execution of test
according to terms and
specification of driving models
Log# ES-AVA-040912-0073
De-Configure
Range
Precise resource mapping improves
assurance of clean-up
Approved for public release 0162; No Export Controlled Data
Almost exclusively
manual process
Human-driven, Modelassisted process
Model-driven, Humanassisted process
Almost exclusively
automated process
11
Cyber Experimentation & Test Process
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
12
Model-base Cyber Experimentation & Test Capability
DSML
Tool
•
•
Model-based Range
Operations concept
showing technical
aspect of the cyber
range operations
CYBER RANGE RESOURCES
AND ENVIRONMENT
Scenario
Model
Model
Interpreter
Deployment
Model
Physical
Hardware
Virtualization
Resources
Instrumentation
Model
Model
Interpreter
Model
Interpreter
Configuration
Files
Not shown are business
processes associated
with range operations or
tools associated with
inventory management,
resource control and
technology refresh
Simulation
Models
Emulators
Traffic Models
Configuration
Files
Analysis &
Visualization
Tools
Deployment
Engine
User Models
Instrumentation
(sensors)
Test Control
Tools
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
13
Summary
• The model-based automation framework provides a flexible infrastructure for
the development and testing of large-scale complex software systems
• The framework itself is an adaptive, model-drive system that is very flexible
• New DSML’s can be rapidly created to allow the framework to support domain
specific testing requirements
• We have demonstrated its use for cyber experimentation & test research
• This framework extends the existing paradigm of Live/Virtual/Constructive
simulation by applying the model-driven approach to the entire testing process
• Test planning, test deployment, test execution, range operations & testrecipes
• The process is realized as a set of model transformations and provides
enhanced support for scientific rigor and application of domain specific
constrains (e.g., IA)
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
14
Academic Research
• Vanderbilt University, Institute for Software Integrated Systems (ISIS) is a
world leader in the research and development of Model-based Engineering
and Model-Integrated Computing Tools
• University of Utah EMULAB is a leader in the development of large-scale
virtualization ranges for network and software research, especially for tools
that manage the deployment of experiments onto the range
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
QUESTIONS?
Log# ES-AVA-040912-0073
Approved for public release 0162; No Export Controlled Data
15