Transcript DEVS Agents to Support Conformance Testing of Emerging

DEVS Agents to Support Conformance
Testing of Emerging
Defense Information Standards
Bernard P. Zeigler,
Arizona Center for Integrative Modeling and Simulation
Tucson Arizona
www.acims.arizona.edu
and
Joint Interoperability Test Command
(JITC)
Fort Huachuca, Arizona
Outline
•
“Agent-supported simulation deals with the use of agents as a support
facility to enable computer assistance in problem solving or enhancing
cognitive capabilities.” (ADS’06 definition)
•
The agent-supported simulation metaphor applies to testing the
conformance of multi-agent systems to complex defense information
standards
•
Structure control agents induce structural change in themselves or others to
effectuate different behaviors under different circumstances
•
Structure control implemented in Dynamic Structure DEVS enables
automation of standards conformance testing
•
DEVS formalism is capable of capturing the information-processing
complexities, including the dynamics, underlying the MIL-STD-6016C
standard
•
DEVS modeling and simulation methodology is attaining core-technology for
automated testing of military tactical data link standards
Background: Simulation-based development implies the need for
simulation-based testing
Specified
as
abstract
model,
e.g. UML
Connecting middle
ware, e.g. HLA
System Under
Test (SUT)
send/receive
messages
Test Device
send/receive
messages
Network
-- distributed simulation allows testing a system that is first formulated as an
abstract model
-- both the System under Test (SUT) and the test device are coupled by an
common interface
Raises the question: How to develop the Test Device in an authoritative manner?
Problem: Conflicting Requirements
 To deal with the increasing complexity and advanced decision
capabilities of C4ISR systems
=>testing methodology has to become more
rigorous, in-depth and thorough
To keep up with the rapid change and short development life
cycles expected from the system builders
=> tests have to be ready to conduct in time
scales compatible with the agile development strategies of
new systems.
Solution: employ DEVS-based M&S
• to increase capabilities for simulation-based testing
and
•as a basis to increase the automation of testing processes.
Testing of interface standards is a focus area for automated
simulation-based testing.
Link-16 is required in all Joint and multi-national operations.
Theater
Warning
DSP/SBIRS
ABL
AWACS
JLENS
F-15
THAAD
TEL
PATRIOT
MEADS
SIS(MSCS)
SIS(MSCS)
AVENGER
ATACMS
AEGIS (CEP)
The Joint Interoperability Test Command (JITC) is developing an
automated test generation methodology as its core technology
for testing conformance of systems to Link-16
This methodology is fundamentally enabled by the DEVS
formalized modeling and simulation approach
Link-16
specification
DEVS Representation of Link-16
Transaction Level - example P.1.2 = Drop Track Transmit
1
Preparation
Constraints
(Exception)
Rules
2
Processing
Modify C2
Record for TN
3
System Theory Provides Levels
of Structure/Behavior
Transmit
Msg
Validity
checking
Track
Display
Rule
Processing
Level
Time
outs
Operator
decisions
Periodic
Msg
Other ConsequentProcessing
Stop
Stop, Do Nothing,
Alerts, Or jump to other
Transaction
3
Coupled
System
2
I/O
System
1
I/O
Function
0
I/O
Frame
Jumps (stimuli) to other
Transactions of specification
Output from
Input to
system
system
DEVS
t
1
t
2
t
3
t
4
Recent Successful Application
The JITC employed such testing for the initial major
milestone evaluation of the Integrated Architecture Behavior
Model (IABM) developed by the Joint Single Integrated Air
Picture (SIAP) System Engineering Organization (JSSEO) in
2005.
The test exercise produced significant results that
uncovered flaws in the model design and added
acknowledged value to the model development.
Types of Distributed Simulation Testing
Testing
Description
Example
standards conformance test whether system
conforms to
standard
supports
interoperability
Tactical Data Link
Standards,
Link-16, VMF,
USMTF,
interoperability
test whether systems
can interoperate (at
the syntactic,
higher, levels)
Joint Translator
Forwarder (JXF) – air
to/from land exchange
of tactical data
mission/capabilities
test whether system Joint Close Air
of systems have
Support (JCAS)
capabilities required
for mission
Multiplatform Distributed Simulation
– controlled testing
Platform
(System,
Component)
Platform
(System,
Component)
Platform
(System,
Component)
Test Driver
Test Driver controls the scenario
Multiplatform Distributed Simulation
- uncontrolled testing
Platform
(System,
Component)
Observer
Platform
(System,
Component)
Observer
Platform
(System,
Component)
Observer
Test
Coordinator
Distributed Observers look for opportunities to test
Test Driver for Controlled Testing
Coupled Test Model
Component
Test Model
1
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
Component
Test Model
2
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
Middleware
SUT
Component
Test Model
3
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
Test Model Generation for Controlled Testing
Mirroring (flipping) the transactions of a SUT model (system
model behavior selected as a test case) allows automated creation
of a test model
Test Model
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
t1
holdSend(Jx1,data1,t1)
holdSend (Jx2,data2,t2)
holdSend (Jx3,data3,t3)
waitReceive(Jx4,data4)
t2
t3
t4
time
SUT Model
receiveAndProcess(Jx1,data1)
receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3)
transmit(Jx4,data4)
Test Manager for Opportunistic Testing
•
•
•
•
Replace Test Models by Test Detectors
Deploy Test Detectors in parallel, fed by the Observer
Test Detector activates a test when its conditions are met
Test results are sent to a Collector for further processing
Test
Detector 1
SUO
Other
Federates
Observer
Test
Detector 2
Test
Detector 3
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
Test
Manager
Results
Collector
Test Detector Generation for Opportunistic Testing
The Test Detector watches for the arrival of the given subsequence of messages
to the SUO and then watches for the corresponding system output
•
•
Define a new primitive, processDetect, that replaces holdSend
Test Detector
–
–
Tries to match the initial subsequence of messages received by the SUO
When the initial subsequence is successfully matched, it enables waitReceive (or
waitNotReceive) to complete the test
Test
Detector
Jx1,data1
Jx2,data2
Jx3,data3
Jx4,data4
t1
processDetect(Jx1,data1,t1)
processDetect(Jx2,data2,t2)
processDetect(Jx3,data3,t3)
waitReceive(Jx4,data4)
t2
t3
t4
time
SUO
receiveAndProcess(Jx1,data1)
receiveAndProcess(Jx2,data2)
receiveAndProcess(Jx3,data3)
transmit(Jx4,data4)
Observer &
System Under Observation (SUO)
inports
inports System
outports
(e.g. DEVS)
outports
Tap into inputs and
outputs of SUO
Observer
For
System
Observer
outports
Gather input/output data
and forward for testing
Example: Joint Close Air Support (JCAS) Scenario
Natural Language Specification
JTAC works with ODA!
JTAC is supported by a Predator!
JTAC requests ImmediateCAS to AWACS !
AWACS passes requestImmediateCAS to CAOC!
CAOC assigns USMCAircraft to JTAC!
CAOC sends readyOrder to USMCAircraft !
USMCAircraft sends sitBriefRequest to AWACS !
AWACS sends sitBrief to USMCAircraft !
USMCAircraft sends requestForTAC to JTAC !
JTAC sends TACCommand to USMCAircraft !
USMCAircraft sends deconflictRequest to UAV!
USMCAircraft gets targetLocation from UAV!!
Observer of AWACS with JCAS
Observer is
connected to SUO
and
monitors its
I/O traffic
addObserver(USMCAircraft, JCASNUM1);
Data
gathered by
Observer
Test Detector Prototype:
Sequence Matcher
processDetect
(J2.2,data1,t1)
processDetect
(J3.2,data2,t2)
Sequential
triggering, same
as test models
waitReceive
(J7.0,data3,t3)
Example of Effect of State: AWACS
Rules
R1: if phase = “passive” & receive= "ImmediateCASIn“
then output = "CASResourcesSpec" & state = "doSurveillance“
R2: if state = "doSurveillance“ & receive= "sitBriefRequestIn“
then output = "sitBriefOut“ & phase = “passive”
i1
i2
o1
o2
state =
doSurveillance
state =
passive
matchsequence 1:
initial state = passive
processDetect(ImmediateCASIn,””,1)
waitReceive(CASResourcesSpec,””)
matchsequence 2:
initial state = doSurveillance
processDetect(sitBriefRequestIn,””,1)
waitReceive(sitBriefOut,””)
need to know the
state to enable
this sequence
Solution: make activation of matchsequence2
conditional on matchsequence1
matchsequence2 can only start when
matchsequence1 has successfully
been performed
Observation Test Of AWACS
Observer
of
AWACS
AWACS
Test
Manager
Problem with Fixed Set of Test Detectors
• after a test detector has been started up, a message may arrive
that requires it to be re-initialized
• Parallel search and processing required by fixed presence of
multiple test detectors under the test manager may limit the
processing and/or number of monitor points
• does not allow for changing from one test focus to another in realtime, e.g. going from format testing to correlation testing once
format the first has been satisfied
Solution
•
•
•
•
on-demand inclusion of test detector instances
remove detector when known to be “finished”
employ DEVS variable structure capabilities
requires intelligence to decide inclusion and removal
Dynamic Test Suite: Features
• Test Detectors are inserted into Test Suite by Test
Control
• Test Control selects Detectors based on incoming
message
• Test Control passes on just received message and starts
up Test Detector
• Each Detector stage starts up next stage and removes
itself from Test Suite as soon as the result of its test is
known
– If the outcome is a pass (test is successful) then next stage is
started up
Dynamic Inclusion/Removal of Test
Detectors
Test Manager
Active Test Suite
Test
Control
removeAncestorBroth
erOf(“TestControl");
message
arrives
addModel(‘test detector”);
addCoupling2(" Test
Manager ",“Jmessage",“test
detector", “Jmessage");
add induced
test detectors
into test set
test detector
subcomponent
removes its enclosing
test detector when
test case result is
known (either pass or
fail)
Adding Ports and Coupling by Relation
Methods can be issued by any atomic model:
• add my ports and coupling to ancestor relation of another model – E.g.,
find my ancestor that is a brother of the given model, and add my ports
and associated internal and external coupling to this ancestor
A
A
B
B
C
D
E
addMyPortsToAncestorBrotherOf(B)
C
D
E
A
C
B
D
E
Issued by E, adds E’s ports to C, which
is ancestor brother of B, and does
associated coupling
Adding Ports and Coupling (cont’d)
Methods can be issued by any atomic model:
• add output ports and coupling from source to destination – adds
outports of source as input ports of destination and couples them
• add input ports and coupling from source to destination – adds inports of
source as outputs of destination and couples them
A
A
B
B
In1
out1
addOutportsNcoupling(C,B))
addInportsNcoupling(C,B))
C
C
D
E
in1
Issued by E, allows E
to communicate with
B
D
E
out1
Removing Models by Name and by Relation
Methods can be issued by any atomic model:
•remove model by name – accepts the unique name of any atomic or
coupled model within hierarchy and removes it and its immediate coupling
• remove model by relation to another model – e.g., remove ancestor that is
a brother of the given model, can be used to avoid providing atomic
models with names of other models
A
A
B
removeModel2(C)
C
D
removeAncestorBrotherOf(B)
E
A
C
B
D
E
Issued by D, removes its parent which
in this case is its ancestor that is a
brother of B
B
AWACS Opportunistic Testing in JCAS
Test
Control
CAS Model with
AWACS
observation
Initially empty
Test Suite
AWACS Opportunistic Testing in JCAS
(cont’d)
Test Control
observes CAS
request
message to
AWACS
Test Control adds
appropriate Test
Detector and
connects it in to
interface,
AWACS Opportunistic Testing in JCAS
(cont’d)
Test Control
passes on start
signal and request
message
First stage detector verifies
request message receipt
and prepares to start up
second stage
AWACS Opportunistic Testing in JCAS
(cont’d)
First stage detector
removes self from test
suite
second stage waits for
expected response from
AWACS to request
AWACS Opportunistic Testing in JCAS
(cont’d)
Second stage
observes correct
AWACS response
and removes itself
and starts up second
part
AWACS Opportunistic Testing in JCAS
(cont’d)
At some later time,
second part of Test
Detector observes
situation brief request
message to AWACS
First stage removes
itself and starts up
second stage
AWACS Opportunistic Testing in JCAS
(cont’d)
Second stage observes
situation brief output from
AWACS thus passing test, It
removes itself and enclosing
Test Detector
Summary
•
DEVS formalism is capable of capturing the information-processing
complexities, including the dynamics, underlying the MIL-STD-6016C
standard
•
DEVS modeling and simulation methodology is attaining core-technology for
automated testing of military tactical data link standards
•
Structure control implemented in Dynamic Structure DEVS enables
automation of standards conformance testing
•
Structure control agents induce structural change in themselves or others to
effectuate different behaviors under different circumstances
•
The agent-supported simulation metaphor applies to testing the
conformance of multi-agent systems to complex defense information
standards