Transcript No Slide Title

NATO Code of Best Practice (COBP)
for C2 Assessment
Prepare for
Success
3 Problem
Formulation
Sponsor
Problem
4
Solution
Strategy
6
Human &
Organisational
Issues
Data
Chapter 9
5 Measures of
Merit (MoM)
7 Scenarios
8 Methods
& Tools
11 Products
Stuart Starr
MITRE
9 Data
10 Assess
Risk
O
C
B
Data
P
Prepare for
Success
3 Problem
Formulation
Sponsor
Problem
4
Solution
Strategy
6
Human &
Organizational
Issues
5 Measures of
Merit (MoM
(MoM))
7 Scenarios
8 Methods
Models
& Tools
11 Products
Products
9 Data
10 Assess
Risk
2
O
C
B
P
Agenda
• Definitions, Taxonomy
• Nature of the Problem
• Key Needs, Best Practices
• Summary
3
O
C
B
P
Definitions
• “Data are factual information that are organized for analysis
and in a form suitable for machine processing”
• “Metadata are information about information; e.g.,
– Documentation of the attributes of data directly attached to the data
– Characterization about reliability of the data”
4
O
C
B
Data Taxonomy
P
Area
Examples
Sources
Official
Legacy Study Results
Open
Subject Matter Experts (SMEs)
Data Types
Raw
Processed
Aggregate
Statistical
Derived
Intermediate
Assessment Data Domains
Scenario
Human/Organizational
C2 Performance
Tool
Measures of Merit
DP - MOP - MOCE - MOFE - MOPE
5
O
C
B
P
Nature of the Data Problem (1 of 2)
“The government is very keen on amassing statistics.
They collect them, add them, raise them to the n-th
power, take the cube root and prepare wonderful
diagrams.
But you must never forget that every one of these figures
comes in the first instance from the village watchman,
who just puts down what he damn pleases.”
-- Comment of an English judge on the subject of Indian
statistics; Quoted in Sir Josiah Stamp in “Some Economic
Matters in Modern Life”
6
O
C
B
P
Nature of the Data Problem (2 of 2)
• “Data! Data! Data!” he cried impatiently. “I can’t make
bricks without clay.”
Sherlock Holmes
• “Metadata! Metadata! Metadata!”
Simone Youngblood, MORS Workshop
• “Theory without data = philosophy; data without theory =
noise”
Anonymous
7
O
B
P
C
Selected Data Problems
The data problems that the C2 assessment
community are facing are not unique to it -- many
other communities seem to have the same issues,
including, inter alia,
–
–
–
–
–
–
–
–
–
Data acquisition
Data conversion
Data sharing
Data reuse
Data purity
Metadata policy
Data shelf life
Data naming conventions
Data reconciliation
– Data maintenance
– Data protection
– Data provenance
– Data surrogation
– Data bloat
– Lack of knowledge of original purpose
– Lack of good data dictionaries
– Ontological development for
intelligent searches
8
O
C
B
P
Key Data Issue: Barriers to Reuse
• Lack of knowledge about existence of legacy data
• Security restrictions
• Quality of metadata (e.g., failure to document
conditions of collection)
• Varying definitions, language, measurement
instruments
• Form of accessible data
• Rapid change of technical data
• Fear (e.g., misuse, misunderstanding, adverse
consequences)
9
O
C
B
P
Why Do We Care About Data?
• Frequently drives solution strategy (e.g., study
schedule, costs)
• Determines surrogate vice ideal MoMs
• Constrains viable scenarios
• Drives treatment of key factors; e.g.,
– Treatment of human factors
– Addressing organizational issues
• Affects selection of tools
10
O
C
B
P
What the Team Needs to Know
About Data
• Data needs/data structure
– Preferred
– Necessary
– Available
• Data accessibility
– Ownership
– Security issues
– Costs (buy, collect, generate)
11
O
C
B
P
Comments on Selected
Data Challenges
• Overall goals
– Make data visible, accessible, and understandable across
military organizations and beyond
• Obtaining data
– Existing data: this entails finding, organizing, verifying,
processing, and converting data
– Non-available data (e.g., new concepts of operations)
• Tap SMEs, results of simulations,…
• Document via metadata
• Replace expert opinion with empirical data (e.g.,
experiments, operational data) ASAP
• Data conversion
– Initial: vague, uncertain, incomplete, contradictory, soft
– Desired: sharp, certain, complete, consistent, hard
– Be explicit about how you converted the data!
12
O
C
B
P
Selected Best Practices
• Near-term
– Employ emerging “analytical baselines” (DODD 8260)
(subsuming scenarios, concept of operations, integrated data)
– Find and reuse data to the extent feasible
– Use metadata to document key data actions (e.g., implement
DoD Discovery Metadata Standard (DDMS)) -- and update
appropriately!
• Longer-term
– Establish a C2 Assessment Community of Interest
– Align data processes and toolsets, as early as possible, with the
C2 systems community (e.g., use same Information Resource
Dictionary System)
– Employ data engineering to gather, organize, convert, and
VV&C available data
13
O
C
B
P
Summary
• The data problem is complex and enormous … and
increasing in both complexity and size!
• The community has taken significant initial steps to address
the problem; e.g.,
– Issued new directives, instructions (e.g., DoDD 8260)
– Created new organizations (e.g., Joint Analytic Data Management
Steering Committee (JADMSC))
– Formulated a framework based on the concept of enterprise,
community of interest, and private data
– Promulgated new tools, standards (e.g., DDMS)
• However, in order to make further substantive improvements,
we have to
– Transform the culture (e.g., by implementing incentives, overcoming
disincentives)
– Educate & train the users, providers of data -- and the decisionmaker!
– Implement new processes (e.g., work the metadata problem)
14