Transcript Introduction to Exercise Design & Evaluation

INTRODUCTION TO EXERCISE DESIGN
& EVALUATION
INTRODUCTIONS
 Name
 Agency
 Exercise Experience
 Course Goals
OBJECTIVES
 Familiarize participants with and/or review the exercise design and
evaluation process
 Introduce MTDES evaluation documentation
 Examine and analyze HSEEP Toolkit documents and systems
REASONS TO EXERCISE
 Establish a comprehensive exercise program
 Reveal planning weaknesses
 Reveal gaps in resources
 Satisfy regulatory requirements
 Clarify roles and responsibilities
 Test and evaluate plans, policies, and procedures
BENEFITS OF EXERCISING
 Individual Training
 System Improvement
REASONS TO USE HSEEP
 Common exercise policy and
program guidance
 Consistent terminology
 Provides tools for exercise design,
conduct, and evaluation
PREPAREDNESS CYCLE
CAPABILITIES-BASED PLANNING
 Reviewing current plan
 Conducting needs assessment
 Assessing jurisdiction’s
capability to conduct an
exercise
EXERCISE PROGRAM MANAGEMENT
 Project Management
 Multi-Year Planning
 Budgeting
 Grant Management
 Staff Hiring
 Funding Allocation
 Expenditure Tracking
MULTI-YEAR TRAINING & EXERCISE PLAN
 Goals
 Components
 Capabilities-Based
PROGRESSIVE EXERCISES
 Component of a
comprehensive exercise
program
 Goal:
 Large-scale functional or full-scale
exercise
 Small-scale discussion-based and
operations-based exercises in
preparation for large-scale
exercise
EXERCISE TYPES
DISCUSSION-BASED
EXERCISES
SEMINAR
 Overview/Introduction
 Purpose
 Leadership
 Participants
 Applications
WORKSHOP
 Purpose
 Characteristics
 Participants
 Applications
TABLETOP
 Purpose
 Characteristics
 Participants
 Applications
GAME
 Purpose
 Characteristics
 Applications
OPERATIONS-BASED
EXERCISES
DRILL
 Purpose
 Characteristics
 Applications
FUNCTIONAL EXERCISE
 Purpose
 Characteristics
 Applications
FULL-SCALE
 Purpose
 Characteristics
 Participants
 Applications
EXERCISE PROJECT MANAGEMENT
 Foundation
 Design and Development
 Conduct
 Evaluation
 Improvement Planning
FOUNDATION
FOUNDATION
 Base of Support
 Exercise Planning Timelines
 Exercise Planning Team
 Exercise Planning Conferences
BASE OF SUPPORT
 Senior Officials
 Buy-In
 Purpose
EXERCISE PLANNING TIMELINES
 Target timeframes
 Multi-Year Training and Exercise
Plan
 Origin & Composition
EXERCISE PLANNING TEAM
Exercise Planning
Team Leader
 Purpose
 Responsibilities
Safety (Ops
only)
 Composition
 ICS
Operations
Site Liaison
Resources
Planning
Exercise
Documentation
Evaluation
Logistics
Admin
Finance
Props
Reporting
Actors
Budgeting
EXERCISE PLANNING CONFERENCES
 Concept and Objectives Meeting
 Initial Planning Conference
 Mid-Term Planning Conference
 Master Scenario Events List
Conference
 Final Planning Conference
DESIGN & DEVELOPMENT
EXERCISE DESIGN STEPS
Injects
 Assess Needs
Expected Actions
 Define Scope
Major & Detailed Events
 Write a Statement of Purpose
 Define Objectives
Narrative
 Compose a Narrative
 Write Major and Detailed Events
 List Expected Actions
 Prepare Injects
Define Objectives
Statement of Purpose
Define Scope
Assess Needs
ASSESS NEEDS
 Purpose
 Needs Assessment Process
 Evaluations from Past Exercises
 Needs Assessment Product
TARGET CAPABILITIES
 Capabilities-based planning
 Capabilities-based exercises
 Tasks & Objectives
DEFINE THE SCOPE
 Purpose
 Determining Functions
 Key Elements
WRITE A STATEMENT OF PURPOSE
 Purpose
 Goal
 Content
DEFINE OBJECTIVES
 Purpose
 Origin
SMART System
Simple
Measurable
 Number
Achievable
 “Good” Objectives
Realistic
 SMART Objectives
Task Oriented
SCENARIO
 Purpose
 Characteristics
 Narrative Outline
WRITE MAJOR AND DETAILED EVENTS
Major Events
Sample
1.
Fuselage breaks apart and hits
buildings below
2.
Jet fuel ignites several homes
in the area
3.
About 60 survivors are
thought to be trapped in the
front section of the aircraft
4.
Estimates of passenger
casualties rise between 200 &
220 deaths and at least 70
severe burn victims
 Purpose
 Goal
 Strategy
 Origin
 Sample
WRITE MAJOR AND DETAILED EVENTS
(CONT.)
Detailed Events
Sample
1.
The mortuary is unable to
accept the large numbers of
remains resulting from the
crash.
2.
Local hospitals lack specialized
facilities and personnel to
treat large numbers of severe
burn victims.
3.
The American Red Cross has
agreed to set up a family
information center to link
victims and their families
 Purpose
 Goal
 Strategy
 Origin
 Sample
LIST EXPECTED ACTIONS
 Purpose
 Goals
 Types of Actions
 Guidelines
 Relationship to Objectives
 Example
DOCUMENTATION
 Situation Manual
 Exercise Plan
 Controller & Evaluator Handbook
 Master Scenario Events List
 Player Handout
 Exercise Evaluation Guides
 Exercise Policies
LOGISTICS
 Discussion-based Exercises
 Operations-based Exercises
EXERCISE CONDUCT
EXERCISE CONDUCT
 Setup
 Briefings
 Management of Facilitators,
Controllers, Evaluators, Players,
and Actors
 Wrap-Up Activities
SETUP
 Site Visits
 Pre-Exercise Check
 Discussion-Based Exercises
 Operations-Based Exercises
PRESENTATIONS/BRIEFINGS
 Discussion-Based Exercises
 Operations-Based Exercises
PERSONNEL
 Facilitators
 Controllers
 Evaluators
 Players
 Actors
 Simulators
EXERCISE EVALUATION
EVALUATION GOALS
 Identify successes and shortfalls
of:
 Response organizations
 Planning documents
 Facilities
 Emergency Management System
 Training/Staffing deficiencies
 Equipment
 Improve Emergency Management
Capabilities
EVALUATION STEPS
 Plan and Organize the Evaluation
 Observe the Exercise and Collect
Data
 Analyze Data
 Develop the Draft After Action
Report & Improvement Plan
PLAN & ORGANIZE THE EVALUATION
EVALUATION TEAM LEADER
 Qualifications
 Responsibilities
 Designation
EVALUATION REQUIREMENTS &
METHODOLOGY
 Purpose
 Defining the Team Structure
 Criteria
 Strategies
 Evaluation Plan
EVALUATION TEAM
 Team Structure
 Responsibilities
 Strategies
RECRUIT, ASSIGN, AND TRAIN
EVALUATORS
 Recruiting
 Experience & Expertise
 Time Commitment
 Establishing Base of Evaluators
 Assigning
 Strategy
 When to Assign
 Training
 Purpose
 Materials
 Instructions
FINALIZE EVALUATION PLAN
 Exercise-Specific Information
 Evaluator Team Organization,
Assignments, and Location
 Evaluator Instructions
 Evaluation Tools
CONTROLLER AND EVALUATOR BRIEFING
 Purpose
 Composition
OBSERVE THE EXERCISE AND COLLECT
DATA
EXERCISE EVALUATION GUIDES
 Purpose
 Use
 Composition
DISCUSSION-BASED EXERCISES
 Higher Level Capability Issues
 Types of Information Collected
 Debrief
OPERATIONS BASED EXERCISES
 Task & Activity Level Observations
 Types of Information Collected
 Use Events to Gauge Success
 Evaluator Guidelines
HOT WASH
 Purpose
 Roles and Responsibilities
 Guidelines
COLLECTING SUPPLEMENTAL DATA
 Delegation
 Purpose of Data
 Examples
ANALYZE DATA
IDENTIFYING ROOT CAUSE AND
DEVELOPING RECOMMENDATIONS
 What Happened
 Why It Happened
 Root Cause Analysis
DATA ANALYSIS
Discussion-Based Exercises
Operations-Based Exercises
 Controller & Evaluator Debrief
 Controller & Evaluator Debrief
 Draft AAR Content
 Draft AAR Content
 Preliminary Analysis Content
 Exercise Event Timeline
 Preliminary Analysis Content
DEVELOP THE DRAFT AFTER ACTION
REPORT & IMPROVEMENT PLAN
AFTER ACTION REPORT
 Purpose
 Who is responsible for it?
 Form
 Format
IMPROVEMENT PLANNING
CONDUCT AFTER ACTION CONFERENCE
 Purpose
 Content
 Format
IDENTIFY CORRECTIVE ACTION TO BE
IMPLEMENTED
 Corrective Actions
 Purpose
 Content
 Realistic Goals
IMPROVEMENT PLAN
 Purpose
 Components
 Capability
 Observation
 Recommendations
 Corrective Action Description
 Responsible Party/Agency
 Agency POC
 Start Date
 End Date
CAPABILITY ELEMENTS
 Planning
 Organization and Leadership
 Personnel
 Equipment and Systems
 Training
 Exercises, Evaluations, and Corrective Actions
FINALIZE AAR/IP
 Purpose
 Composition
 Goals
TRACK IMPLEMENTATION
 Event Points of Contact
 Participating Entity Points of
Contact
 Action Officers
 Continual Improvement
QUESTIONS?