Transcript PSI Metrics Presentation

Sharing Lessons on Learning Metrics
LINGOs Annual Meeting
Sept. 23-24, 2009
Steven Honeyman and Marie-Laure Curie
Global Capacity Building Department
Let’s start with small group
discussions
• Briefly describe your learning metrics (overview only)
• Discuss advantages and disadvantages of different
components
• Summarize what the group thinks are some new or
interesting ideas the whole group should know about
LINGOs Annual Meeting - Sept. 23-24, 2009
Kirkpatrick’s model
Includes four levels/steps of outcome evaluation:
• Level 1—Reaction – satisfaction survey
• Level 2—Learning – tests
• Level 3—Behavior – on-the job behavior change
survey
• Level 4—Results – business outcomes
LINGOs Annual Meeting - Sept. 23-24, 2009
Measuring Training Impact at PSI
SMILE - Social Marketing Initiative for Learning and Excellence
Introduction to the PSI
Behavior Change Framework
• Behavior change framework drives PSI
marketing activities
– Fundamental to social marketing
• Applicable to all behaviors promoted by PSI
• Based on behavior change theory
• Determinants measured through TRaC
(Tracking Results Continuously) survey
LINGOs Annual Meeting - Sept. 23-24, 2009
The overall framework
IMPROVED HEALTH STATUS/QUALITY OF LIFE
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
AT RISK POPULATIONS
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
SOCIAL MARKETING INTERVENTION
PRODUCT PRICE
PLACE
PROMOTION
LINGOs Annual Meeting - Sept. 23-24, 2009
What PSI does…
IMPROVED HEALTH STATUS/QUALITY OF LIFE
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
AT RISK POPULATIONS
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
SOCIAL MARKETING INTERVENTION
PRODUCT PRICE
PLACE
PROMOTION
LINGOs Annual Meeting - Sept. 23-24, 2009
Our sphere of
influence
What we try to influence…
IMPROVED HEALTH STATUS/QUALITY OF LIFE
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
AT RISK POPULATIONS
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
SOCIAL MARKETING INTERVENTION
PRODUCT PRICE
PLACE
PROMOTION
LINGOs Annual Meeting - Sept. 23-24, 2009
To change a behavior...
IMPROVED HEALTH STATUS/QUALITY OF LIFE
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
AT RISK POPULATIONS
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
SOCIAL MARKETING INTERVENTION
PRODUCT PRICE
PLACE
PROMOTION
LINGOs Annual Meeting - Sept. 23-24, 2009
That improves health
IMPROVED HEALTH STATUS/QUALITY OF LIFE
RISK REDUCING BEHAVIOR, PRODUCT OR SERVICE USE
AT RISK POPULATIONS
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
SOCIAL MARKETING INTERVENTION
PRODUCT PRICE
PLACE
PROMOTION
LINGOs Annual Meeting - Sept. 23-24, 2009
OAM
• Opportunity: institutional or structural factors that
influence an individual’s chance to perform a
promoted behavior.
• Ability: an individual’s skills or proficiencies needed to
perform a promoted behavior.
• Motivation: an individual’s desire to perform a
promoted behavior.
LINGOs Annual Meeting - Sept. 23-24, 2009
TRaC (Tracking Results Continuously))
• What is it?
– Repetitive quantitative survey that measures behavior
change of a specific behavior of a specific target group
– Measure the effectiveness of project interventions and
activities
• Why do we use it?
– To better design programs, to monitor program progress
(ensure that the target population is adopting the promoted
behavior and if needed, make adjustments) and evaluate
how we did (target group and behavior)
– Informs decision-maker where to invest more/fewer
resources
• Who does it?
– All PSI programs
• When does it occur?
- Prior, during (sometimes often) and at the end of an
intervention
LINGOs Annual Meeting - Sept. 23-24, 2009
The overall Capacity Building framework
IMPROVED JOB PERFORMANCE AND OVERALL IMPACT
OF PSI PROGRAMS
USE OF SKILLS ACQUIRED AT THE TRAINING
BY PSI STAFF
OPPORTUNITY
Availability
Brand
Attributes
Social
Norm
ABILITY
MOTIVATION
Brand
Appeal
Knowledge
Attitudes
Quality of
Care
Social
Support
Intentions
Locus of
control
Outcome
expectation
Self
Efficacy
Subjective
norm
Threat
Willingness
to pay
Belief
POPULATION CHARACTERISTICS
CAPACITY BUILDING INTERVENTION
LINGOs Annual Meeting - Sept. 23-24, 2009
TRaC-T (Tracking Results Continuously for Training))
• What is it?
– Short, quantitative survey
– Measures application of training on-the-job
• Why do we use it?
– Programmatic check to ensure that the training had the
intended impact of transferring needed skills/behaviors to
staff and to identify gaps in training content/delivery
– Informs CB decision-maker where to invest more/fewer
resources
• Who does it?
– Global Capacity Building department (for virtual training as
well)
• When does it occur?
– Approx. 6 to 8 weeks after the training ended
LINGOs Annual Meeting - Sept. 23-24, 2009
Kirkpatrick’s model
Includes four levels/steps of outcome evaluation:
• Level 1—Reaction – satisfaction survey
• Level 2—Learning – tests
• Level 3—Behavior – TRaC-T survey
• Level 4—Results – business outcomes
LINGOs Annual Meeting - Sept. 23-24, 2009
TRaC-T survey (example)
LINGOs Annual Meeting - Sept. 23-24, 2009
LINGOs Annual Meeting - Sept. 23-24, 2009
Some TRaC-T results
TRaC-T Budget Review training - August 5 & 6 2009
At the training, you learned how to review
agreements under $100,000. Since the training
have you had an opportunity to use this skill?
Please rate how confident you are that you can
now apply the new skills you acquired at this
training:
60.0%
100.0%
80.0%
40.0%
60.0%
20.0%
40.0%
20.0%
0.0%
0.0%
Yes
No
Confident
Somewhat confident
Not confident
LINGOs Annual Meeting - Sept. 23-24, 2009
Some TRaC-T results
Who suggested that you take this training ?
After the training, did your supervisor follow-up
with you regarding the training?
60.0%
50.0%
40.0%
100.0%
30.0%
80.0%
20.0%
60.0%
10.0%
0.0%
40.0%
20.0%
0.0%
Yes
LINGOs Annual Meeting - Sept. 23-24, 2009
No
TRaC-T might have some limitations:
• Self reported behavior,
• Needs to be built into the training program,
• Needs resources to develop and manage,
• May not apply learning right away.
But it has many potential benefits:
• Demonstrates training effectiveness to senior managers,
• Convinces supervisor of the value of a particular training,
• Allows targeting of training resources to highly effective training,
• Provides feedback to those leading training on whether they are
being effective, and
• Can be automatically set-up for online training !
LINGOs Annual Meeting - Sept. 23-24, 2009
For more information:
• www.psi.org/research/
• Marie-Laure Curie, Manager, GCB Dept.,
[email protected]
• Steven Honeyman, Director, GCB Dept.
[email protected]
LINGOs Annual Meeting - Sept. 23-24, 2009