Transcript Slide 1

Exercise Unit 1
Jennifer Maddrell
R561: Evaluation & Change
in the
Instructional Development Process
Indiana University
Professor Knuth
May 29, 2006
Branching Out
To a new Training Evaluation Plan
Evaluation is a means to:
Brinkerhoff & Dressler:
–
Increase the value of the training investment
Kaufman & Watkins:
–
Compare results with intentions
Morrison, Kemp & Ross:
–
Make judgments about the worth or success of the
training program
Phillips:
–
–
Capture the contribution of human resource
development
Determine customer satisfaction
Van Tiem, Moseley & Dessinger:
–
–
Judge the results of performance
Trigger or support a decision
An evaluation plan will provide:
• Feedback
– As a quality control measure
• Control
– To assess value and worth to organization
• Research
– For knowledge to improve
• Intervention
– To affect how training is viewed, used and
shared
• Power
– To fairly represent results to stakeholders
Source: Knuth, 2006
Questions we must answer …
• How do participants feel about our
training program?
• Are participants learning?
• Is their learning transferring to the
job?
• Does the organization benefit
from our training efforts?
Finding answers …
Kirkpatrick’s 4 Levels of Evaluation
• Level 1: Reaction
– How do participants feel about our training
program?
• Level 2: Learning
– Are participants learning?
• Level 3: Behavior
– Is their learning transferring to the job?
• Level 4: Results
– Does the organization benefit?
Source: Kirkpatrick,1998
Kirkpatrick’s Model Measures:
• Level 1: Reaction
– Customer satisfaction
– Learners motivation to learn
• Level 2: Learning
– Attitude change
– Knowledge improvement and skill gain
• Level 3: Behavior
– Change in behavior
– Transfer of skills
• Level 4: Results
– Financial Impact: On costs, production, etc.
– Non-financial Impact: On morale, motivation, etc.
Source: Kirkpatrick,1998
Appeal of Kirkpatrick’s Model:
• Assesses important areas
• Widely known
• Simple framework
• Easy to explain and understand
However . . .
• Widely Know ≠ Widely Used
– Level 1: Often (over 90%)
– Level 2: Sometimes (less than 35%)
– Level 3 & 4: Rarely (less than 15%)
• Why is this a problem?
– Level 3 and 4 often perceived as:
• Difficult to measure
• Time consuming
• Beyond the realm of most trainers
– Level 1 result does not always mean
similar Learning / Transfer / ROI results
Source: Pershing & Gilmore, 2004
Other problems …
• Undermines Management Partnership
– Training ≠ “Silver Bullet”
– Training is only one strategy within entire
Performance System
– Level 3 & 4 should include evaluations of entire
Performance System - not just training
• Lacks Performance System Focus
– What about rest of Performance Environment?
– What factors impede / enable usage of training?
• Feedback Goes to Wrong People
– Feedback to training function only is incomplete
– Must include Performance Environment owners
Source: Brinkerhoff & Dressler, 2002
Alternatives to Kirkpatrick:
• Numerous alternative evaluation
options exist
• Recommend implementation of
Brinkerhoff & Dressler’s
Success Case Evaluation Model
• This alternative focuses on
training’s business impact as part
of entire performance system.
Success Case Model Answers:
• What is the business impact of
instructional program?
• What is the organization doing
that is facilitating performance
improvement?
• What is the organization doing
that is impeding performance
improvement?
Source: Brinkerhoff & Dressler, 2002
Success Case Model Approach:
• Brief survey to large sample to assess:
“To what extent have you used your recent training in a
way that you believe has made a significant difference to
the business?”
• In depth small sample review of both:
– successful groups:
• nature and business value of their application of
learning, and
• performance context factors (support)
– unsuccessful groups:
• performance context factors (obstacles)
• other factors preventing use of learning
Source: Brinkerhoff & Dressler, 2002
In Summary:
• An evaluation tool must be integrated into the
training programs at BIG.
• Kirkpatrick’s Four Levels of Evaluation is well
known, but has limitations:
– Unlikely completion of all 4 Levels
– Lacks performance system focus
• Brinkerhoff & Dressler’s Success Case
Evaluation Model is the recommended next
step approach.
– Rapid evaluation / feedback process
– Addresses key business impact issues
– Contemplates entire performance environment
References
Brinkerhoff, R. O. & Dressler, D. (in press). Using evaluation to build organizational performance and learning
capability: A strategy and a method. Performance Improvement.
Kaufman, R., Keller, J., & Watkins, R. (1995). What works and what doesn't: Evaluation beyond Kirkpatrick.
Performance & Instruction, 35, (2). 205-209.
Kirkpatrick, D. L. (1998). The four levels: An overview. Ch. 3 in Evaluating Training Programs: The Four Levels, 2nd
ed. San Francisco: Berrett-Koehler.
Knuth, R. (2006). Unit 1: Introduction to Evaluation. Retrieved May 18, 2006, from Indiana University R526 Syllabus
http://www.indiana.edu/~istr561/knuth06sum/unit1print.html
Morrison, Gary R., Kemp, Jerrold E., & Ross, Steven M. (2001). Chapter 10, The Many Faces of Evaluation. In
Designing Effective Instruction (3rd edition). New York: John Wiley & Sons.
Pershing, J., Gilmore, E. (2004). Evaluating Training Programs Kirkpatrick’s Four Levels. [PowerPoint Presentation]
Retrieved May 18, 2006, from Indiana University R526 Syllabus
http://www.indiana.edu/~istr561/knuth06sum/PPTs/r561d1_kirk_perspective.ppt
Phillips, J. (1997). Handbook of Training Evaluation and Measurement Methods (Improving Human Performance
Series) 3rd Edition. Butterworth-Heinemann. Chapters 1 - 3.
Van Tiem, Darlene M., Moseley, James L., Dessinger, Joan Conway (2004). Chapter 7, Evaluation. In Fundamentals
of Performance Technology: A Guide to Improving People, Process, and Performance, 2nd Edition.
Washington, DC: International Society for Performance Improvement.