Transcript Document 7886400

Effective Training: Strategies,
Systems and Practices, 2nd Edition
Chapter Eight
Evaluation of Training
Session Overview




Define evaluation
Describe why many organizations don’t
evaluate training
Explain who is interested in evaluation
data
Describe the various levels of training
evaluation
Chapter 8
2
Evaluation Defined

Evaluation:
judgment of training programs
worth or value

Formative Evaluation
(process)
examines how the training was designed,
developed, and carried out

Summative Evaluation
(outcome)
determines if training accomplished its objectives
Chapter 8
3
Resistance to Training Evaluation
A few reasons why training managers
don’t evaluate training:



Nothing to evaluate
No one really cares
Evaluation is a threat to my job
Chapter 8
4
Types of Evaluation Data
The following are types of evaluation
Data that should be collected:




Reaction Questionnaire
Learning
Job Behavior
Organizational Results
Chapter 8
5
Who Is Interested
Outcome Data
Reaction
Learning
Behavior
Results
Trainer
Yes
Yes
Yes
No
Other Trainers
Perhaps
Perhaps
Perhaps
No
Training Manager
Yes
Yes
Yes
Yes
Trainees
Yes
Yes
Yes
Perhaps
Trainees’ Supervisor
Not really
Only if no
transfer
Yes
Yes
Upper Management
No
No
Perhaps
Yes
Training Department
Customers
Chapter 8
6
Kirkpatrick Level 1:
Reaction Questionnaire
Reaction Questionnaires are:
“Used to determine what the trainees
thought about the training.” (p. 353)
Affective Questionnaire:
- measures general feelings about training
Utility Questionnaire:
- measures the trainee's belief about the value
of the training
Chapter 8
7
Reaction Questionnaire: Types of
Questions
The following is obtained from a reaction
questionnaire:

Facilities issues

Trainer issues

Perceived value of training

Relevance of training
(example: pp. 355-356)
Chapter 8
8
Timing of Reaction Questionnaire
The following times can be used to
administer a reaction questionnaire?

During training (after each segment)

Immediately following training

One or two months after training
Chapter 8
9
Steps in Developing a Reaction
Questionnaire (1 of 2)
1.
Determine what you want to find out (consider training objectives).
2.
Develop a written set of questions to obtain the information.
3.
Develop a scale to quantify respondents’ data.
4.
Make forms anonymous so participants will feel free to respond honestly.
5.
Ask for information useful in determining differences in reactions by
subgroups (e.g., young vs. old; minority vs. non-minority). This could be
valuable in determining effectiveness of training for different cultures, for
example, which may be lost in an overall assessment.
Chapter 8
10
Steps in Developing a Reaction
Questionnaire (2 of 2)
6.
Allow space for “Additional Comments” in order to allow participants the
opportunity to mention things you might not have considered.
7. Decide the best time to give the questionnaire to get the information you
want.
a. If right after training, have someone other than the instructor
administer and pick up the information.
b. If some time later, develop a mechanism for obtaining a high response
rate (e.g., encourage the supervisor to allow trainees to complete the
questionnaire on company time).
Chapter 8
11
Kirkpatrick Level 2: Learning
Assessing Learning:

Declarative knowledge – paper/pencil multiple
choice test

Procedural knowledge – paired comparison
method (p. 359; 8-4)
1.

Determining relationships among a number of concepts
Strategic knowledge – Probed Protocol Analysis
1.
2.
Define a problem
Explain step-by-step how to solve problem
Chapter 8
12
Kirkpatrick Level 3: Job Behavior
Job Behavior:
“Measuring if training actually
transferred to the job”
Chapter 8
13
Kirkpatrick Level 3: Job Behavior
Data collection approaches:

Behavior data



Interviews
Direct observation
Archival records of performance

Scripted situation - a process for assessing

Attitudes – attitudinal surveys
transfer of training to the job by providing rater with
a number of responses to rate trainee (p. 363)
Chapter 8
14
Kirkpatrick Level 4:
Organizational Results
Organizational Results:
“This measurement would be used to
determine any change after training
was complete” (p. 364)


Did grievances go down?
Did scrap rates diminish?
Chapter 8
15
Kirkpatrick Level 4:
Organizational Results
Data collection:
 Consistent tracking of organizational
performance should take place at
intervals before, throughout, & after
training.
 Scrap rate after training – scrap rate
before training = results
Chapter 8
16
Evaluation: Kirkpatrick
Organizational
Results
External environment of the organization:
Economy, regulations, suppliers, etc
Internal environment of the organization:
Policies, procedures, systems
Employee performance, KSAs, and needs
Job Behavior
Transfer of training
Motivational forces in the job setting
Opportunity to applying training on the job
KSAs
Learning
Reactions
Trainee readiness for the course
Trainee motivation to learn
Design, materials, and content
Trainer(s) behaviors
Perceived match between trainee
Exceptions and what training provided
Chapter 8
17