Transcript EXERCISE EVALUATOR COURSE

HSEEP Exercise
Evaluation and Improvement
ODP’s Mission

Primary responsibility within the
executive branch to build and sustain
the preparedness of the US to reduce
vulnerabilities, prevent, respond to, and
recover from acts of terrorism (Homeland
Security Act).
ODP’s Responsibilities

Grant programs for planning,
equipment, training and exercises

National training program

National exercise program
Grant Programs

State Homeland Security Program

Law Enforcement Terrorism Prevention
Program

Citizen Corps Program

Urban Areas Security Initiative Program

Fire Fighter Assistance Program
State Homeland
Security Program
 Purpose: to enhance capacity of states
and locals to prevent, respond to, and
recover from terrorism
 Provides funds for





Homeland security and emergency operations
planning
The purchase of specialized equipment
CBRNE and cyber security training programs
CBRNE and cyber security exercises
State Homeland Security Assessments and
Strategies
Law Enforcement Terrorism
Prevention Program
 Provide law enforcement communities with
funds to support the following prevention
activities:






Information sharing to preempt terrorist attacks
Target hardening
Recognition of potential or developing threats
Interoperable communications
Intervention of terrorists before they can execute a
threat
Planning, organization, training, exercises, and
equipment
Citizen Corps Program
 Provides funds to support Citizen Corps
Councils with planning, outreach, and
management of Citizen Corps program
and activities





Form and sustain a Citizen Corp Council
Engage citizens in homeland security
Conduct public education and outreach
Develop and implement Citizen Corps programs
Coordinate Citizen Corps activities with other DHS
funded programs and other federal initiatives
Urban Areas Security
Initiative Program
 Address the unique needs of large urban
areas – 50 cities
 Conduct jurisdictional assessment and
develop Urban Area Homeland Security
Strategy.
 Funds for planning, equipment, training,
exercise, and administration and
operational activities related to heightened
threat levels
Fire Fighter Assistance

Protect public and fire fighters against
fire and fire-related hazards



Fire fighting Operations and Safety
Fire Prevention
Fire fighting Vehicles
Strategy Process Overview
Statewide Homeland
Security Strategy
Assessments
Conducted at the
local and state levels
Created at the regional
and state level
State Assistance
Plan
Created by ODP
State and Urban Area use strategy to
identify & allocate all HS resources
END RESULT = Capability Improvements
Strategy Participants

State and local jurisdictions
 All First Responder Disciplines
 Fire Service
 Public Health
 HazMat
 Health Care
 Emergency Medical Services
 Public Works
 Law Enforcement
 Government Administrative
 Emergency Management
 Private Sector
 Public Safety
Communications
 Non-Profit/Voluntary Sector
Assessment Overview
Statewide
Homeland
Security
Strategy
Risk Assessment
Threat
Assessment
Vulnerability
Assessment
Agricultural
Vulnerability
Assessment
Needs Assessment
Required
Capabilities
Current
Capabilities
Shortfalls or
“Gaps”
* CBRNE: Chemical, Biological, Radiological, Nuclear, Explosive
Threat Assessment
Who:
Risk Assessment
Local, state, and federal law
enforcement officials
What:
• Identify number of Potential
Threat Elements (PTEs)
Threat
Assessment
Vulnerability
Assessment
Agricultural
Vulnerability
Assessment
• Identify threat factors (existence,
violent history, intentions, WMD
capability, and targeting)
• Identify motivations (political,
religious, environmental, racial, or
special interest)
• Identify WMD capabilities
(CBRNE)
Vulnerability Assessment
Who:
Risk
Assessment
All response disciplines at local, state,
and federal levels
What:
Identify critical infrastructure/ potential
targets
Evaluate targets for:
• Level of visibility
Threat
Assessment
Vulnerability
Assessment
Agricultural
Vulnerability
Assessment
• Criticality of target site
• Impact outside of jurisdiction
• Access to target
• Target threat of hazard
• Target site population capacity
• Potential collateral mass casualties
Capabilities and Needs:
Planning

The results from the risk assessment
process (threat and vulnerability) provide
a link to the capabilities and needs
assessment process.
 Planning
 Organization
 Equipment
 Training
 Exercises
State Homeland Security
Strategy

Developed by State based on local
needs

Provides blueprint for planning of
homeland security efforts to enhance
preparedness and for use of resources
State
Assistance Plans

ODP uses the strategies and needs
assessment data to tailor and formulate a
State/Metro Assistance Plan (SAP/MAP) for
each state

A SAP/MAP is a blueprint for the delivery of
ODP training, exercise, technical assistance
and equipment services
National Training Program

Training for federal, state and local
homeland security professionals

Based on critical tasks to prevent,
respond to or recover from a terrorist
incident

Over 40 courses available
ODP Training Program

ODP offers more than 40 courses (Examples)





Live chemical agents training – Center for Domestic
Preparedness
Live explosives training – New Mexico Institute of
Mining and Technology
Radiological and nuclear agents training – Nevada Test
Site
Advanced emergency medical training using human
patient simulators – Texas A&M
Training on bioterrorism – Louisiana State University
National Exercise Program

Responsible for National Exercise Program

Threat and performance-based excises at
federal, state, local, and international levels

Strategy and Exercise Planning Workshops to
define exercise needs and plan for each state
Assess Program Success
Through Exercises

Performance measures for ODP’s grant,
training, and exercise programs are tied to
performance of critical tasks

Percent of jurisdictions that can perform
critical tasks as demonstrated through
exercises

500,000+ population

100,000+ population

50,000+ population
Overview of HSEEP

Threat- and Performance-based
Exercises

Cycle of exercises

Increasing complexity

To improve preparedness
HSEEP Manuals




Volume I:
Program Overview and
Doctrine
Volume II: Exercise Evaluation and
Improvement
Volume III: Exercise Development
Volume IV: Sample Exercise
Documents and Formats
Vol I: HSEEP Overview
and Doctrine




ODP’s exercise and evaluation doctrine
Uniform approach for exercise design,
development, conduct, and evaluation
Exercise design and implementation
process
Suite of common scenarios (TBD)
Vol II: Exercise Evaluation
and Improvement




Defines exercise evaluation and
improvement process
Provides uniform set of evaluation
guides
Defines data analysis process
Includes standardized After-Action
Report template
Vol III: Exercise
Development


Defines exercise planning and design
process
Provides guidance for the development
and conduct of various types of
exercises
Vol IV: Sample Documents


Provides sample letters, planning
documents, checklists, scenarios, etc.
Reduces development time for exercise
design team
Exercise Evaluation



Assess preparedness at federal, state and
local levels
Validate strengths and identify
improvement opportunities, resulting in
improved preparedness
Provide guide for resource allocations
Evaluation
Enhancements







Focus on performance of critical tasks and
mission outcomes
Use of uniform evaluation tools
Enhanced data analysis
Debriefing meeting with key officials
Improvement Plan
Track implementation of improvements
Suite of common scenarios (TBD)
Exercise Evaluation
Methodology Development

Exercise Evaluation Working Group

Builds on


Responder Guidelines

ODP exercise experience

CSEP and other programs
Will continue to enhance and improve
Exercise Evaluation and
Improvement Process
Exercise Evaluation and Improvement Process
Evaluation
Planning, Observation, and Analysis
Data Collection and Analysis
Step 1
Step 2
Step 3
Step 4
Plan & Organize
the Evaluation
Observe the
Exercise &
Collect Data
Analyze Data
Develop After
Action Report
Improving Preparedness
Step 5
Step 6
Step 7
Step 8
Conduct
Debriefing
Identify
Improvements
Finalize After
Action Report
Track
Implementation
Levels of Analysis

Performance is analyzed at three levels:



Task level
Agency/discipline/function level
Mission level (within and across
communities)
Levels of Analysis

Task Level Performance


Answers the question: did
the person or team do the
right thing the right way at
the right time?
Helps assess need for
training, equipment,
personnel, etc.
Task = work with measurable
output that has utility
Levels of Analysis

Agency/Discipline/Function Level
Performance — Multiple teams

Answers the question: did the larger team or
organization perform duties in accordance with
plans and policies?

Helps assess
communication,
coordination, planning
budgets, etc.
Levels of Analysis

Mission Level Performance


Answers the question: were the
mission level outcomes
achieved?
Addresses jurisdictional
preparedness
Outcomes = results
Mission Outcomes
Pre-Event
Emergency
Response

Prevention/Deterrence


Emergency Assessment
Emergency Management
Hazard Mitigation
Public Protection
Victim Care
Investigation/Apprehension

Recovery/Remediation




Post-Event
Evaluation Requirements





Determine what outcomes will be
evaluated, based on exercise objectives
Identify activities to be evaluated
Identify which functions should be
observed
Determine where observations will take
place
Identify the appropriate evaluation tools
Exercise
Evaluation Guides

ODP has developed Exercise
Evaluation Guides that:



Identify the activities that the evaluator
should be observing
Provide consistency in tasks across
exercises
Link individual tasks to disciplines and
outcomes
The EVALPLAN

Exercise-specific information

Plans, policies, procedures, and
agreements

Evaluator recruiting and assignments

Evaluator training and instructions
Recruiting and
Assigning Evaluators

Setting expectations – evaluators must
be available for:






pre-exercise training and briefing
pre-exercise site visit
the entire exercise (hours to days)
post-exercise hot-wash
post-exercise data analysis (1 day)
contribution to the draft AAR
Recording Observations

The emphasis is on Who? What?
When? Where? How? Why?

Record observations through:



use of Evaluator Guides
blank sheets of paper
Collect exercise documents
Record
Significant Activities

Initiating scenario events

Facility activities

Response actions

Key decisions made by Players

Deviations from plans and procedures

Completion time of events
Evaluator Summary

Compile observations into chronological
narrative of events

Describe outcomes achieved or not – use
questions below and evaluation guides:






What happened?
What was supposed to happen?
If there is a difference, why?
What is the impact of that difference?
What should be learned from this?
What improvements might be recommended?
Data Analysis

Conduct Hotwash

Develop timeline of significant events

Analyze performance:



Individuals
Teams/Functions
Outcomes
Hotwash

Player Hotwash:



Usually held immediately following exercise
play
Typically facilitated by the evaluator
Provides opportunity for:




Player self-assessment
An interactive discussion
Clarification of observed events
Assessment of exercise simulations
Timeline Development
Time
Make a team
timeline of
actions
Focus on
significant
actions
Identify the
appropriate
outcome for
each activity
Observations
Location
Team
Outcome
0852
Three staff members arrive at JIC –
PIO, Deputy, Admin. Ass’t; begin
setting up (computers, removing
displays from storage, job aids at
work stations, etc.)
JIC
Rushmore
Co.
EM
0905
First media call to JIC requesting info.
on event. PIO provides initial incident
info. & tells reporter to watch for news
release shortly re: JIC activation.
JIC
Rushmore
Co. PIO
EM
0906
Forest Co. PIO and Assistant arrive
at EOC; PIO immediately calls in to
EOC
JIC
Forest Co.
EM
0910
EAS message from EOC received by
Fax
JIC
EOC
EM
0912
EAS copied & distributed at JIC to all
staff work stations; additional copies
on tables in media room.
JIC
JIC staff
EM
Analysis of
Performance

Analysis of activities




What tasks were to be accomplished
How well were they performed
Root causes of differences between
expected and actual performance
Recommendations
Root Cause Analysis
1. Why did it happen?
2. Why did that happen?
3. Why was that?
4. And why was that?
5. And why was that?
6. And so on…
***Root Cause***
Each step must completely
explain the step above…
…down to the basic
underlying causal factor.
Integrated Analysis

Allows further identification of:





Successes and best practices
New gaps and problems
Root causes
Recommendations for improvement
Compares observations from different
locations and functions
Recommendations for
Improvement

Questions for identifying
recommendations for improvement:



What training and/or equipment is needed?
What changes need to be made to plans and
procedures, or organization structures?
What changes could be made to the management
processes?
The After-Action
Report (AAR)





Serves as feedback tool
Summarizes what happened
Identifies successes and
recommendations for improvement
May include lessons learned to share
with other jurisdictions
Help jurisdictions focus resources on
greatest needs
After-Action Report

Prepared in two stages:

Draft AAR – completed immediately after
the exercise for review


Community adds improvement
steps/corrective actions
Final AAR
AAR Format








Executive Summary
Part 1: Exercise Overview
Part 2: Exercise Goals and Objectives
Part 3: Exercise Events Synopsis
Part 4: Analysis of Mission Outcomes
Part 5: Analysis of Critical Task
Performance
Part 6: Conclusion
Appendix A: Improvement Plan Matrix
Improvement Process

Improving preparedness activities:




Conduct exercise debrief
Identify improvements
Finalize AAR
Track implementation
Step 5
Step 6
Step 7
Step 8
Conduct
Debriefing
Identify
Improvements
Finalize After
Action Report
Track
Implementation
Exercise Debrief

Provides a forum for
jurisdiction officials to:



Hear the results of the
analysis
Validate the findings and
recommendations in draft
AAR
Begin development of
Improvement Plan
Improvement Plan


Developed by local jurisdiction during
debrief
Identifies how recommendations will be
addressed:



What actions
Who is responsible
Timeline for completion
Finalize AAR
 Improvement
Plan is included in
final AAR
 Final AAR submitted to ODP
through State Administrative
Agency
Monitor Implementation
 ODP
Exercise Management
System (under development) will
provide:



Centralized calendar of exercises across
the country
Electronic submission of AAR/IPs to the
SAA and ODP
Monitoring of Improvement Plan
implementation
Sharing Lessons Learned

Ready-Net – Web-based, secure
information network



National repository for best practices
and lessons learned
Accessible to approved users within
the response community
Administered by the Memorial
Institute for the Prevention of
Terrorism
Benefits of
HSEEP Approach

Nationwide consistency

More useful after action reports and
improvement plans

Ability of jurisdictions to focus
resources on greatest needs

ENHANCED PREPAREDNESS
Exercise Evaluation
Training Course




2 ½ days - Exercise Evaluation
methodology
6 sessions to train ODP staff and
contractors as change agents (225
people)
Training for SAAs Feb-May 2004
ODP Exercise Design Course being
revised to deliver consistent message
Goal for Working Group

Review and modify Exercise Evaluation
Guides for Radiological and Biological
attacks




Are the right tasks identified?
Do other tasks need to added?
Are the conditions and typical steps logical
and complete?
Are the followup analysis questions the
right questions to assess performance?