Transcript Document

Experimentation and
Operational Research
OR Society Defence Study Group
10 June 2004
George Pickburn
Information Management Department
Topics
• Operational Research / Analysis - what is it ?
• Experimentation
– Principles - why experiment ?
• ‘Applications’ and Experiment ‘types’
• Facilities
• Experiments in the programme
• Experimentation as a research Theme
• The UK Code of Best Practice
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
“There is no empirical method without
speculative concepts and systems;
and there is no speculative thinking
whose concepts do not reveal, on closer
investigation, the empirical material
from which they stem.”
Einstein (1953), in his introduction to an edition of
Galileo’s Dialogue Concerning the Two Chief
World Systems (1632)
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Operational Research / Analysis
• CSA’s Guidelines :
– “OA contributes to the basis on which MOD makes its decisions
by providing an open, explicit and objective assessment of the
military capability provided by a proposed course of action . . . “
– “in combination with . . . cost data . . . quantitative comparison .
. (of) . . new capabilities”
– “OA is based . . on objective data about equipment, its human
operators and their operational environment”
– “The idea of the model lies at the heart of OA. . . . a simplified
representation of reality . . to allow us to gain understanding of
the factors relevant to (a) decision”
• A scientific process of analysis . . . supporting decision
making
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
What’s new?
• New . . . . .
– op concepts
– information facilities and services
– ways of working - joint and combined
– doctrine . . . .
• How do people use information ?
• Which information do they use ?
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Analysis Techniques Spectrum
Experiments
High
Experiments
Realism
Low
High
Cost
Repeatability
Low
High
Low
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
What is experimentation?
• Controlled process of interaction with and observation of
the real world
• Twin modes of enquiry / experimentation : inductive, and
hypothetico–deductive
• Complementary nature of these modes –
– most new knowledge arises as (provisional) insight derived from
‘casual’ observation
– insights formally validated and tested through experimentation
– any one “experiment” can do both
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Principles - why experiment ?
• Standard part of the general purpose research enterprise
• Generation of new knowledge
• Consolidation of knowledge in revised and validated
theories and models
• This is a recursive, not a linear process : model-testmodel-test . . . . .
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Model - Test - Model Paradigm
New
Knowledge
(OA)
Studies
(Analytical)
Analytical
Models/Theories
Models
Real World
Experiments
Knowledge
Requirements
Knowledge
Knowledge
Experimentation
Knowledge
Environment of executive decision making,
studies, concept definition, op. planning,
procurement
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
“Couldn’t interface with a new paradigm,
whatever the hell that means “
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
‘Applications’ - Experiment ‘types’
• exploration / discovery / visualisation
• test & evaluation,
• formal verification and validation,
• demonstration
• systems design
• warfighting experiments - VCDS policy
• .......
• all are experiments to meet differing requirements for
confidence / burden of proof
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Experimentation typology
Burden of proof
Quality of knowledge
Confidence
#3
Expt
#2
Tests
Trials
Demos
Visualisations
Discovery
20 July 2015
© Dstl 2001
#1
No. of good ideas
No. of hypotheses
Quantity of knowledge
Dstl is part of the
Ministry of Defence
CCRP CoBP Experimentation
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
What changes between types? ( = design )
• Quantity (categories) of knowledge gained
• Quality (confidence) of knowledge gained (‘ burden ‘ of
proof)
• Both are controllable to meet the purpose of the
experiment - how will the knowledge gained be used?
• There is always a hypothesis
• Design entails trade-off of:
– effect size,
– confidence (in the truth of the research hypothesis),
– power to reject false hypotheses, and
– sample size
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Facilities
• Dstl Applied Research Technology Demonstrator (ARTD)
– C2 systems lab
• NITEworks
– an experiment on three levels
• Exercises as an experimentation environment of opportunity
- BATUS & Warfighting experiments
• Wargames / Training Systems
– Combined Arms Tactical Trainer
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Experiments in the programme
• Kill Chain - Responsive C2 of tactical air under different
sensor ‘treatments’
• Indirect Fire Integration - new munitions, role of
intelligence, targeting facilities
• Future HQ - organisation and design
• Effects Based Planning - MNE3
• ISTAR tasking and integration
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Experimentation as an R&D theme
• Coherence for efficient R&D
• Exploitation of exercises
• Standards across the programme
• Output sharing and re-use
• Theme Adviser on experimentation
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Codes of Best Practice
• NATO CoBP for C2 Modelling
• US Code of Best Practice
• US Marine Corps “Practice of Military Experimentation”
• Facility CoBPs - UK ARTD, US JFCOM
• TTCP CoBP for Experimentation - soon
• UK MoD Code of Best Practice
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Experimentation Best Practice I
• Identify the knowledge being sought
– what is weak / wrong in our current models
– use model to provide an expectation to be tested by an
experiment
• Identify the relevant variables
• Form hypotheses (testable ideas) about these variables
• Design experiments to test the hypotheses - in the light
of the expected uncontrollable variability - using
statistical methods
• Capture the design and its rationale in a design
document for peer review
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Experimentation Best Practice II
• Record the conclusions about the tested hypotheses
(for later re-use)
• Amend the encapsulating theory or model in the light of
the proven hypotheses
•
=
the model-test-model paradigm
New
Knowledge
(Analytical)
Analytical
Models/Theories
Models
Real World
Experiments
Knowledge
Requirements
Knowledge
Knowledge
Knowledge
Environment of executive decision making,
studies, concept definition, op. planning,
procurement
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence
Summary
Experimentation is . . . . .
• the best opportunity for a generation to explore
interactions of people and information to enable
– balanced exploration of the lines of development
– better military organisation design
– better procurement of information and communication systems
– effective military and government response to emergent
‘asymmetric’ threats
– extension and validation of the models for decision making
• the natural (and essential) complement of operational
research / analysis
20 July 2015
© Dstl 2001
Dstl is part of the
Ministry of Defence