Transcript Financial Fitness for Life Teacher Training Workshop

Some Practical Tips
for Measuring Financial Success
Dr. Angela Lyons
University of Illinois
Program Evaluation II:
Creating Your Evaluation Toolkit
Presented by
Dr. Angela Lyons
University of Illinois
October 2009
Identifying the “ideal” approach
to program evaluation

Evaluation methods and measures vary widely
across programs and academic disciplines.

Wide variation in financial outcomes across
programs.

Significant differences in financial needs
across consumers.

Some participants unable to implement certain
financial behaviors.
Then you need to decide….
“To
“To evaluate
evaluate or
or not
not to
to evaluate,
evaluate,
that
is
the
question?”
that is the question?”
Questions to ask yourself ….

At the end of the day, what are you trying to show?

What is the purpose of the evaluation?

Who will use the information – and how?

What information do you want to collect?

Who is your target audience?

What is your primary delivery method?

What are your available resources (i.e., time, money,
and staff)?

What is your timeline?

What is your expertise and evaluation capacity?

Who are your partners, funders, and stakeholders?
Common survey methods
used to collect impact data
• Post evaluation only
• Retrospective pre-test (RPT)
• Pre and post evaluation
• Follow-up
• Stages to Change (TTM)
• Control groups and longitudinal studies
Key question to ask:
What is the length of your program?
Post evaluation only
When to use: Short programs that are less than 2 hours
Advantages:
 Only need to survey group once.
 Good for limited-resource audiences and groups that are
transient.
 Relatively inexpensive and less time intensive.
 Can document participants’ levels of knowledge, skills, and
planned behaviors at the end of the program.
Disadvantages:
 With no pre assessment, it’s difficult to document potential and
actual changes in knowledge, attitudes, and behavior.
Retrospective pre-tests (RPTs); The Post-Then-Pre Evaluation
Retrospective pre-test (RPTs)
When to use: Any program, but typically 2 hours or less
Advantages:
 Only need to survey group once.
 Good for limited-resource audiences and groups that are
transient.
 Controls for “response shift bias.”
 Can document “relative” change.
Disadvantages:
 Potential for respondent bias (social desirability factor).
 Self-assessment measures are subjective.
7
Pre and post evaluations
When to use: Programs that are 2 hours or longer
Advantages:
 Can compare pre and post responses and document changes in
knowledge, attitudes, and behavior.
 Can be used to document immediate changes in knowledge, skills
and planned behaviors following the program.
Disadvantages:
 More time intensive.
 Identification numbers are needed to match pre and post surveys.
 May be difficult to show actual behavior change.
 May be difficult to show that the intervention caused the change.
 Doesn’t account for other possible reasons for change.
 Transient populations may lead to low unmatched evaluations.
Follow-ups
When to use:
•
•
•
•
Program is comprehensive
enough to potentially result in
intermediate and long-term
impact.
Must have adequate resources
and evaluation capacity.
Usually administered three to six
months after the program.
Can document changes in
actual financial behaviors, ability
to achieve financial goals, and
overall financial position.
Stages to Change (TTM)
When to use: Programs that have multiple sessions
Advantages:
 Can document intermediate and long-term change.
 Easier to measure actual behavior change and to control for other
factors that may lead to change over time.
 Can identify stage at which individual is ready and able to change
behavior.
 Behaviors can be recorded at the beginning, middle, and end of the
program so that changes in actual behavior can be observed.
Disadvantages:
 Time and resource intensive.
 May require additional progress reporting and long-term follow-up.
 Can only be used with multi-session programs.
Train-the-trainer evaluations

Similar to pre and post evaluation,
but more content specific.

Covers subject material in more
detail to ensure that trainers have
an adequate level of knowledge to
teach the program to others.

Can be used to document
changes in both the instructors’
teaching skills and personal
financial behaviors.

Follow-ups can document how the
curriculum materials are being
used and identify additional
programming needs.
Designing the evaluation instrument:
Key survey content
1. General reactions to the session
2. Changes in knowledge
3. Changes in motivation, confidence, and abilities
4. Intended changes in behavior
5. Actual changes in behavior
6. Future programming needs and preferences
7. Demographics
8. Qualitative / open-ended responses
General reactions to the session
Please rate the instructor(s), materials, and the overall program
by checking the box that best applies.
Not
Helpful
Somewhat
Helpful
Helpful
Very
Helpful
Instructor(s)




Educational materials




Overall program




Measuring changes in knowledge
Testing Knowledge
Please circle your answer to each of the following statements.
1. Fixed expenses are expenses that typically change from
month to month such as food, clothing, and utilities.
True False
Not
Sure
2. Financial experts recommend having an emergency fund
that is equal to 3-6 months worth of living expenses.
True False
Not
Sure
3. Goals should only be made for long-term plans such as
homeownership, college tuition, or retirement.
True False
Not
Sure
4. Gross income is defined as income after taxes and other
withholdings have been subtracted from net income.
True False
Not
Sure
5. Credit reports can affect an individual’s ability to get a job,
purchase a home, and obtain home and auto insurance.
True False
Not
Sure
Measuring changes in knowledge (conti.)

Format can be True/False or multiple choice.

True/False is reliable indicator for low literacy audiences and
youth.

The more questions you ask, the greater the reliability
measure.

May include a “don’t know” option to control for guessing.

Post-test: 10 questions (established standard)

Pre- and post-test: 10-20 questions

Train-the-trainer: 10-25 questions
Changes in motivation, confidence,
and abilities
Building Skills/Confidence Indicators
Please check the box that best describes your confidence to do
the following:
Your confidence to:
Not
Confident
A little
confident
Somewhat
confident
Confident
Very
confident
Set short and longterm financial goals.





Save money
regularly.





Use a spending plan
to track income and
expenses.





Pay bills on time
each month.





Changes in motivation, confidence,
and abilities (conti.)
Recording Participants’ Attitudes
Please check the box that best describes how much you agree
with the following statements.
Statement
Strongly
Disagree
Disagree
Undecided
Agree
Strongly
Agree
Saving money
regularly is important
to me.





Keeping track of
spending is a good
habit.





Planning my
personal budget is a
priority.





Intended changes in behavior
Please indicate how often you plan to do each of the following
financial practices. There is no “right” or “wrong” answer. (Choose only one)
Financial Practice
Never
Rarely
Set
and of
long-term
As short
a result
this program,
financial goals.
No


you plan to:
Save money regularly.
Set short and long-term financial

Use a spending plan to
goals.
track income and
expenses.
Use
a spending plan to track income
and
expenses.
Pay bills
on time each

month.
Pay bills on time each month.
Sometimes
Maybe
Always

Already
doing

this
 not
Does
apply













Yes








Usually




Actual changes in behavior
Please indicate how often you are currently doing each of the following
financial practices. There is no “right” or “wrong” answer. (Choose only one)
Financial Practice
Never
Rarely
Sometimes
Usually
Always
Setting short and longterm financial goals.





Saving money regularly.





Using a spending plan
to track income and
expenses.





Paying bills on time
each month.





Capturing behavior change in follow-ups
and “stages”
Financial Progress Indicators
Please check the box that best describes how your financial
position has changed since completing the program. Then,
indicate specifically how your financial position has changed.
Decreased
Stayed
about the
same
Increased
Monthly income



Monthly expenses



Amount saved monthly



Current credit card debt



Amount of
change,
if any
(% or $)
Capturing behavior change in follow-ups
and “stages”
Progress Reporting
Please record your financial position based on your current progress in
the program.
Financial Position
How much credit card debt do you owe? ($)
How many credit cards do you have? (#)
What is the highest interest rate on your
credit cards? (%)
How much do you have in savings? ($)
At the
beginning of
the program
In the middle
of the program
At the end of
the program
A few words about
train-the-trainer programs….






Testing knowledge
Building teaching skills
Shaping personal skills
Taking action for teaching
Taking action for personal financial success
Follow-ups
Qualitative / Open-Ended Questions
(common examples)
“Post Evaluation Only” and “Pre and Post Evaluation”
 What did you like the most about this program?
 What did you like the least about this program?
 How could this program be improved?
 Would you recommend this program to others?
“Stages to Change Evaluation”
 What has made it easier for you to improve your financial practices?
 What has prevented you from improving your financial practices?
 With respect to the overall program, what did you like the most?
 What did you like the least?
 How could this program be improved?
 Have you shared what you learned with others?
 Would you recommend this program to others?
Qualitative / Open-Ended Questions
(conti.)
“Train-the-Trainer Evaluation”
 What was the most helpful information you received during this
training program?
 How could this training program be improved?
 How do you plan to share this information with your target
audience(s)?
 What information and materials from this training do you plan to
share with your target audience(s)?
 Will you share what you learned with other instructors and
colleagues?
 Would you recommend this training program to other instructors
and colleagues?
Demographic Questions










Age
Gender
Race, Ethnicity, and Language
Marital Status
Education
Employment
Family Structure
Health Status
Income, Assets, and Debts
Region/Location



Financial Experience
Students/Youth
Instructors/Educators
Rigor vs. Reality
Longitudinal data?
Control groups?
Randomized experiments?
NEFE Financial Education
Evaluation Toolkit ®
http://www2.nefe.org/eval/intro.html
NEFE Financial Education Evaluation Toolkit®
 Database
•
•
•
•
Post evaluation only with option for follow-up
Pre and post evaluation with option for follow-up
Stages to Change Evaluation
Train-the-Trainer
• Testing Knowledge
• Building Skills
• Taking Charge
 Manual
• How-to-guide for grass-roots level organizations
• Examples (survey instruments, executive summary, reports)
• Guidance on how to organize and present impact data
Manual
http://www2.nefe.org/eval/manual.html
Manual
http://www2.nefe.org/eval/manual.html
Part I: Financial Education Overview
Part II: Understanding Program Evaluation
Part III: The Evaluation Planning Process
Part IV: Using the Evaluation Database
Part V: Reporting Program Impact
Part I:
Financial Education Overview
Part II:
Understanding Program Evaluation
Part III:
The Evaluation Planning Process
Part IV:
Using the Evaluation Database
Part V:
Reporting Program Impact
Appendix:
Sample Evaluation Instruments
Database
http://www2.nefe.org/eval/index.php
Step 1:
Program Info and Follow-up
Step 2a:
Knowledge Indicators
Step 2b:
Customizing Questions
Step 3:
Confidence and Behavior Indicators
Step 4a:
Selecting Statements
Step 4b:
Customizing Statements
Step 5:
Qualitative Info
Step 6:
Demographics
Step 7:
Follow-Up: Financial Progress Indicators
Step 8:
Follow-Up: Personal Achievements
Step 9:
Follow-Up: Demographics
Step 10:
Finalizing Evaluation
ACTIVITY:
Putting It All Together
(Evaluation Action Plan – Part B)
Think about your signature program,
what is the most appropriate evaluation method?

Post-test only

Interviews

Retrospective pre-test

Case studies

Pre and post-test

Observations

Follow-up survey

Stories/anecdotal evidence

Stages-to-change

Tests of ability

Focus groups

Other
What types of questions will
your evaluation seek to answer?
I would really like to know….
1.
2.
3.
What types of indicators will you use
to document this impact?









Changes in satisfaction levels
Changes in knowledge
Changes in skills and confidence levels
Changes in attitudes
Changes in aspirations
Anticipated or intended changes in behavior
Actual changes in behavior
Socio-economic changes
Other
List some specific indicators.
1.
2.
3.
4.
U of I Center for Economic and Financial Education
http://www.cefe.illinois.edu/
Educational Tools for Evaluation
http://www.cefe.illinois.edu/tools/evaluation.html
Checklist of evaluation resources
Program Planning Guide
Evaluation Action Plan
Evaluation Road Map
Evaluation Reading List
Sample Evaluations
NEFE® Financial Education Evaluation Toolkit
http://www.cefe.illinois.edu/tools/evaluation.html
University of Wisconsin-Extension
http://www.uwex.edu/ces/pdande/evaluation/index.html
Cornell University Extension
http://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm
Penn State Extension
http://www.extension.psu.edu/evaluation/
Contact Information
Dr. Angela Lyons
Associate Professor
Director, U of I Center for Economic and Financial Education
University of Illinois
Phone: 217-244-2612
E-mail: [email protected]
Questions