Transcript Kirkpatrick Four Level Evaluation Program

Kirkpatrick Four Level Evaluation
Certification Program
Presented Jim Kirkpatrick, PhD
Hosted by Booz Allen Hamilton
Herndon, VA
November 23-24, 2009
These slides are provided as a courtesy to those who have
attended one of our classes or presentations.
These are for internal use within your organization only; all
rights reserved.
We thank you in advance for respecting our intellectual
property and refraining from duplicating, distributing,
selling, or otherwise using this presentation in whole or
part for your own profit or promotion.
The following marks are the property of Kirkpatrick
Partners, LLC : The Kirkpatrick Business Partnership
ModelSM, KBPMSM, Return on ExpectationsSM, ROESM,
and Chain of EvidenceSM
Kirkpatrick Foundational
Principles
1. The end is the beginning.
2. ROESM is the ultimate indicator of value.
3. Business partnership is necessary to
bring about positive ROE.
4. Value must be created before it can be
demonstrated.
5. A compelling Chain of EvidenceSM
demonstrates your bottom line value.
©2009, all rights reserved.
©2009, all rights reserved.
©2009, all rights reserved.
©2009, all rights reserved.
Kirkpatrick Four Levels
Level 4: RESULTS
The final outcomes that occur as a result of the
training and reinforcement
©2009, all rights reserved.
Kirkpatrick Four Levels
Level 3: BEHAVIOR
The extent to which participants apply what they
learn on the job
©2009, all rights reserved.
Kirkpatrick Four Levels
Level 2: LEARNING
The extent to which participants acquire intended
knowledge, skills, and attitudes
©2009, all rights reserved.
Kirkpatrick Four Levels
Level 1: REACTION
How training participants react to the training
©2009, all rights reserved.
Why Evaluate?
• How can the program be improved?
• How can we maximize training
effectiveness by reinforcing knowledge,
skills, and attitudes?
• How can we demonstrate the value of
training?
©2009, all rights reserved.
“What is your job here at the hotel?”
“I am a window washer.”
©2009, all rights reserved.
“What is your job here at the resort?”
“I am part of a team that creates great experiences for our
guests!”
©2009, all rights reserved.
The Kirkpatrick Business Partnership ModelSM
Identify
NECESSITIES For
Success
Results
PLEDGE to
work together
ADDRESS jury
issues
REFINE
expectations to
define outcomes
Learning
TARGET critical
behaviors and
required drivers
Determine
required KSAs,
Learning
Objectives
Reaction
Consider
necessary learning
environment
Design and build learning program and evaluation tools
Deliver learning program
Analyze findings, adjust,
repeat steps as
necessary
Business
need
identified
Behavior
Measure L2
Learning
Measure L1
Reaction
Initiate ongoing
reinforcement and
monitoring
Measure L3
Behavior
Measure L4
Results
Prepare Chain of EvidenceSM to demonstrate ROE
ROESM
Present L4
Results findings
© 2009. All rights reserved.
Present L3
Behavior
findings
Present L2
Learning
findings
Present L1
Reaction
findings
Drivers – encouragers or
discouragers
Coaching
Mentoring
Refreshers
Business
Critical
Behaviors
Results
Executive
Modeling
Accountability
Level 3
Evaluation
©2009, all rights reserved.
Recognition
DOT Example
1. Jury – safety managers, district engineers, traffic
control, risk managers, maintenance supervisors.
2. Expectations – smooth traffic flow; increased safety for
workers and motorists in highway work zones; better
planning.
3. Success Outcomes – less than 15 minute delays;
reduction in injuries and fatalities; work zone projects
completed on time and within budget.
4a. Key New Behaviors – written work zone plans; plan
communicated to public; proper work zone set-up;
flaggers following procedures.
4b. Success Drivers – observation, feedback and coaching
by supervisors; ongoing compliance tracking; ongoing
execution of formal and informal recognition programs.
©2009, all rights reserved.
Methods, Tools, and Tips
Evaluation Methods – the process that is used to
gather evaluation data
Evaluation Tools – the actual document that is
utilized to gather the evaluation data and/or
information
Evaluation Tips – what you will do to gain elicit the
desired candor and response rates
©2009, all rights reserved.
Guidelines for Evaluating
Reaction
• Determine first what you want to find out
• Consider a blend of objective and
subjective questions
• Consider your rating scale
• If indicated, get delayed reactions
• Develop with the learner in mind
• Set the table for your learners
©2009, all rights reserved.
Level 1
• Reaction Sheets
• Interviews
• Focus Groups
WHAT?
WHO
WHO?
HOW?
©2009, all rights reserved.
WHEN?
WHY?
Guidelines for Evaluating
Learning
• Start with the issues of usefulness and
credibility
• Measure before and after attitudes,
knowledge, and/or skills if indicated
• Beware of an overemphasis on assessing
knowledge
• Evaluate during the course as well as after
• Use a performance test for skills
©2009, all rights reserved.
Level 2
•
•
•
•
Knowledge Checks
Knowledge Tests
Skills Observation
Refreshers
WHAT?
WHO
WHO?
• Teach Backs
• Developing Action
Plans
• Case Studies
HOW?
©2009, all rights reserved.
WHEN?
WHY?
Guidelines for Evaluating
Behavior
•
•
•
•
•
•
•
•
Start with need and importance
Be sure to set the table for this
Don’t overwhelm – more is not better
Measure on a before/after basis if indicated
Consider immediate implementation
Repeat if indicated
Consider more than one source
Answer the ?s, “If so, why?” and “If not, why not?”
©2009, all rights reserved.
Level 3
• Behavior Survey
• Behavior Checklist
• Work Review
WHAT?
WHO
WHO?
• Focus Groups
• Monitor Action Plans
• Action Learning
HOW?
©2009, all rights reserved.
WHEN?
WHY?
Guidelines for Evaluating
Results
• Determine based on responses from
stakeholders – ROESM
• Borrow metrics when possible
• Gather data and information when indicated
• Consider a tiered approach (short and long term
measures)
• Consider evidence versus proof
• Don’t make it harder than necessary
©2009, all rights reserved.
Level 4
• Borrowing Metrics
– Business
– HR
• Surveys
• Focus Groups
WHAT?
WHO
WHO?
HOW?
©2009, all rights reserved.
WHEN?
WHY?
Evaluation Methods
Evaluation Levels
1
Reaction
2
Learning
3
Behavior
4
Results
Survey
●
●
●
●
Questionnaire / Interview
●
●
●
●
Focus Group
●
●
●
●
Methods
Knowledge Test / Check
●
Case Studies
●
Work Review
●
●
Skills / Behavior Observation
●
●
Presentation / Teach Back
●
Action Planning
●
●
●
●
Action Learning
●
Key Business HR Metrics
©2009, all rights reserved.
Kirkpatrick Four Level Evaluation
Feedback Loop to Eliminate ‘Snags’
1
Delivery of Learning Event
L1 & L2 Formative Evaluation
Level 1
Improve Session
4
Post Session L1
2
3
Was session delivered effectively?
NO
YES
Post Session L2
10
Did targeted learning occur?
6
NO
Improve
Reinforcement
and/or Coaching
YES
Post Session Reinfcmt,
Coaching & L3
YES
NO
Level 2
5
Was failure due
to session?
Level 3
7
9
NO
Was there successful transfer
of learning to behavior?
8
YES
Level 4
Expected results will occur.
Build your chain of evidence & showcase
©2009, all rights reserved.
11
Percentage of learning transfer
1975: Percentage of formal learning that is
actually applied to the job:
15%
2005: Percentage of formal learning that is
actually applied to the job
15%
Dana Robinson, ASTD ICE, 2008
©2009, all rights reserved.
Brinkerhoff Study
Training Application
Did not try new skills: 15%
Tried new skills and failed: 70%
Achieved sustained new behaviors: 15%
Josh Bersin and Associates, 2008
©2009, all rights reserved.
Causes of “Training Failure”
Preparation and Readiness: 20%
Learning Intervention: 10%
Application Environment: 70%
2006 ASTD Study
©2009, all rights reserved.
Typical Learning Investment
P re-W ork 10%
L earning E vent 85%
F ollow-U p 5%
Dr. Brent Peterson, University of Phoenix, 2004
©2009, all rights reserved.
The increase in “informal
learning”
Where Learning Takes Place
Prior to being "trained": 20%
During "training": 10%
On the job: 70%
Josh Bersin and Associates, 2008
©2009, all rights reserved.
Transferring Learning to Behavior
Balancing two major forces:
SUPPORT
ACCOUNTABILITY
©2009, all rights reserved.
Data Collection Plan
Level
Methods
Tools
Timing
1
Surveying
Focus Group #1
Reaction Sheet
F.G. Questionnaire
Immediately after course
1 week after course
2
Knowledge test
Skills Observation
Teach backs
Focus Group #1 (see above)
Objective test
Checklist
Instructions
F.G. Questionnaire
Pre and post course
During course
During course
1 week after course
3
On-the-job Observation
Surveying – 180
Focus Group #2
Action learning
Checklist
Survey (partic / mrg.)
F.G. Questionnaire
Instructions
1 week + monthly
4 months after course
4 months after course
6 months after course
4
Survey – Customer
Focus Group #2 (see above)
Review Bus. & HR Metrics
Survey
F.G. Questionnaire
Business & HR Metrics
4 & 8 months after course
4 months after course
6 &12 months after course
©2009, all rights reserved.
Chain of EvidenceSM
Level 1
Reaction
Level 2
Learning
Level 3
Behavior
Level 4
Results
Gather data at all four levels and show that your
training delivers true value to your organization
©2009, all rights reserved.
Four Practical Ways to Make a
Good Business Case
1. Show them the data
2. Conduct an impact study
and showcase it
3. Review your training
curriculum and redeploy
your resources
4. The Brunei Window
Washer
©2009, all rights reserved.
More Information
Feb. 2010
kirkpatrickpartners.com
[email protected]