Transcript Measuring the Quality of Decisionmaking and Planning

Measuring the Quality of
Decisionmaking and Planning
Framed in the Context of IBC Experimentation
February 9, 2007
Evidence Based Research, Inc.
Today’s Ground Rules
• IBC provides the framework for the discussion on
measuring decisionmaking.
• This is not an IBC brief.
• The briefer does not have the answers!
• The purpose is to explore how one measures the quality
of decisionmaking in complex environments – this is a
discussion session.
2
Background Regarding IBC
Focus:
• Visualization, operational assessment, and planning
tools for EBO.
Multiphased effort:
• Phase 0: Proof of concept using ad hoc suite of tools.
• Phase I: Demonstrated improvement potential of
federated tool sets developed by two contractors.
• Phase II: Evolve tool set to expanded capabilities for
transition to JFCOM & COCOMS.
3
Objectives of Phase II Experimentation
• Establish the operational utility of the tools.
• Learn which tools can be helpful under what
circumstances and for what purposes.
• Explore novel approaches throughout the spiral
development process.
• Measure progress in increasing readiness of the
technology.
• Facilitate early transition to operational users.
4
Challenges
• Complex operational problem:
– Wide range of situations,
– Multiple knowledge domains, and
– Poorly understood relationships.
•
Major technical challenges:
– Variation in maturity of underlying models and tools,
– Limitations in availability of relevant data, and
– Composability of federated capability from continually
expanding set of tools.
• Transition hurdles:
– Operational risk due to high potential for limitations in
fidelity and possible misapplication.
5
Option Exploration Tool (1)
• Comprehensive family of models.
• Action-to-effect and effect-to-action.
• Generates the distribution of plausible outcomes.
• Visualization: assist leaders to understand and act.
6
Option Exploration Tool (2)
7
Modeling Paradigm
Concept Map
Proposed Action
Desired
Effect
Political/Economic Model
Social Network
Political/ Religious Model
Bayesian
Network
Agent Based Simulation
Spread
Sheet
Conceptual Models
Statistical Models
Influence
Diagram
Differential
Equations
Causal Models
Social/ Culture Model
Petri net
Economic/Infrastructure ModelObject Models
Event-based
Simulation
Undesired
Effect
Social Information Model
Military Model
Desired
Effect
Enables “What-if” Analysis, Answers “Why”
Distribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program
Campaign Planning Tools (1)
• Supports modular development of multiple lines of effort.
• Automatically detects and displays interdependencies,
assumptions, resources, actions, duration of effects,
metrics, and next state.
• Ability to modify plans, actions, interdependencies,
models, and next states based on measured
performance.
8
Campaign Planning Tools (2)
9
Political/ Strategy
Security Strategy
Interdependencies
Assumptions
Political-Economic Strategy
Actions
Next State
Resources
Measured Performance
Allows the leader to visualize complex plans
Distribution authorized only to US Government Agencies and their contractors involved in Phase 2 of the Integrated Battle Command Program
Network Centric Operations
Information
Sources
Value Added
Services
Quality of Organic
Information
Force
10
C2
Effectors
Quality of Networking
Degree of Networking
Net Readiness of Nodes
Degree of Information “Share-ability”
Quality of Individual Information
Quality of Individual Sensemaking
Individual Awareness
Individual Understanding
Degree of Shared Information
Quality
of
Interactions
Degree of Shared Sensemaking
Shared Awareness
Shared Understanding
Individual Decisions
Collaborative Decisions
Physical Domain
y
lit
i
g
Degree of Decision Synchronization
A
Ag
ili
ty
ce
Cognitive Domain
C2
Degree of Actions/ Entities Synchronized
Social Domain
Degree of Effectiveness
Fo
r
Information Domain
Decisionmaking Quality
• Two approaches:
– Quality of the process:
• How the decision was made, and
• Elements in the process.
– Quality of the decision:
• Was it “right” or at least “not wrong”?
• Did it have the desired effect?
11
Quality of the Process
•
•
•
•
•
Correct sequence of actions.
Complete set of choices.
Assumes an observable process.
Repeatability.
Outcome is irrelevant.
12
Quality of the Decision
•
•
•
•
Applicability to the solution space.
Compatibility with other SMEs.
Desired outcome/effects.
Outcome is relevant.
13
Measuring Quality
• The field needs a way to measure the
quality of a product:
– What does it mean to do good?
– What does it mean to make a good decision?
• A simple problem:
– Has a correct answer.
• A complex problem:
– Has no right answer,
– May have “not wrong” answers, and
– Will have multiple perspectives.
14
Discussion Time
Thoughts?
15