Transcript Financial Fitness for Life Teacher Training Workshop

Some Practical Tips
for Measuring Financial Success
Dr. Angela
Lyons
Program
Evaluation
I:
University of Illinois
Setting the Stage
for Outcomes-Based Success
Presented by
Dr. Angela Lyons
University of Illinois
October 2009
Motivation:
Recent Financial Challenges
•
•
•
•
•
Rising debt
Low savings rates
Home foreclosures
Bankruptcies
Unemployment
Consumers are in financial trouble!!!
Reality Check!
How much trouble are we in?
2009 U.S. deficit = $1.4 trillion
The Million Dollar Question
At the end of the day,
does financial education
make a difference?
What have we learned so far?

What does current research tell us?

Why is evaluation important?

What’s in it for you and your organization?

What’s in it for your clients?
On the front lines
with the evaluation experts….
“What do we mean by evaluation?
“What are we really trying to measure?”
“How do we define program success?”
“How do we know if participants are improving?”
“What financial outcomes and indicators should we be
using?”
“What constitutes a successful, or even acceptable,
evaluation?”
Getting Started: Thinking like an evaluator….
(Program Planning Guide)

Take stock of who you are – What is “your vision?”

Conduct a needs assessment.

Collect baseline information from your target audience.

Identify your “signature program(s).”

Identify your program objectives. Be realistic!

Create an evaluation action plan.

What do you want to accomplish? At the end of the day,
what do you want to show?
How will you define “program success?”

Setting realistic expectations for program participants.

Choosing appropriate outcomes and indicators based
on participants’ financial situation or other external
constraints.

Identifying participants’ individual financial needs and
applying appropriate educational interventions.

Finding “the teachable moment.”
What is an outcome-based evaluation?

Outcomes are benefits to clients from participating in
the program.

What do you want your participants to know or be able
to do when they have finished the program?

Outcomes are usually in terms of enhanced learning
and improved behaviors.

Outcomes are often confused with program outputs or
units of service (e.g., number of clients who went
through the program).
The Logic Model
• A picture of the program.
• Simple representation of the program
“theory” or “action” which explains the
program and what it is to accomplish.
• Shows relationships between inputs,
outputs, and outcomes.
The Logic Model (conti.)
INPUTS
Resources used to
develop the program
are called inputs.
Time and money are
the most common
inputs needed to
implement
educational
programs.
OUTPUTS
OUTCOMES
If inputs are invested
into the financial
education program, then
learning opportunities
will be created for the
target audience. The
created educational
materials, services, and
opportunities are called
the program outputs.
Changes in participants’
perceptions, knowledge,
and behavior that
represent real impact in
their lives. The benefits
derived by the
participants from the
program are called
outcomes.
University of Wisconsin - Extension
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
Impact Hierarchy of Outcomes
Planned End-Result or
Improved Socio-Economic Condition
Changes in
Behavior/Practices
Changes in Knowledge, Attitudes,
Skills, and Aspirations (KASA)
Changes in Perceptions
and Levels of Satisfaction
Another useful framework….
Transtheoretical Model of Behavior Change (TTM)
 TTM integrates major psychological theories
into a theory of behavior change.
 Used to identify the state at which individuals
are ready and able to change their financial
behaviors.
 Appropriate educational interventions are
then tailored to meet individual’s specific
needs at that particular stage.
5 Stages of Change
Precontemplation
•
•
Individual not ready to take action and change
behavior in the immediate future.
Rarely seeks help and rarely uses information.
Contemplation
•
•
Individual is getting ready to take action and intends
to change behavior in next 6 months.
Open to education.
Preparation
•
•
•
Individual is ready to take action and intends to change
behavior in next 30 days.
Practices behavior by taking small steps towards the goal.
Seeks information and support, but often concerned that
changing behavior may be too difficult and they may not
succeed.
5 Stages of Change (conti.)
Action
•
•
•
•
Individual changes behavior and maintains behavior for at
least 6 months.
Believes they can change.
Can control “triggers” that cause them to relapse into old
behaviors.
Has a support system to get them through challenging times.
Maintenance
•
•
•
•
Individual has changed behavior and it has lasted for more
than 6 months.
May relapse into old behaviors, but can overcome temptations
so that behavior becomes permanent.
Can assess the conditions under which relapse might occur.
Can establish successful coping strategies.
Example:
Have
been
doing for
more
than 6
months
Do not
plan to
do
Plan to
do in the
next
month
Plan to
do in the
next 2-6
months
Have
been
doing for
1-6
months
Set short and long-term financial goals.





Save money regularly.





Use a spending plan to track income and
expenses.





Maintain sufficient balances in bank
account(s).





Pay bills on time each month.





Review bills each month for accuracy.





Comparison shop before making
purchases.





Pay off credit card balances in full each
month.





Financial Practice
Identifying Program Objectives
Objectives should be:
Specific
Measurable
Achievable and observable
Reasonable
Time specific
S.M.A.R.T. objective statements should clearly
define what you want to achieve with your program.
They should list the end outcomes the program
intends to affect or change.
Writing objective statements
First-time home buyer education program
The objectives of this program are to:
 Develop first-time home buyers’ ability to shop for the lowest mortgage
interest rate.
 Teach first-time home buyers how to save money for closing costs.
 Teach first-time home buyers how to assess affordable housing.
Debt reduction education program
The objectives of this program are to:
 Develop participants’ ability to identify needs and wants separately.
 Develop participants’ ability to control “wants” to reduce expenditures.
 Develop participants’ ability to avoid impulse and emotional spending.
Achieving your objectives:
Selecting appropriate indicators
General Indicators (objective and subjective):





Number of programs, participants, etc.
Knowledge gains
Changes in attitudes and satisfaction
Changes in skills and confidence
Changes in intended and actual behaviors
Specific Indicators (objective):




Actual dollar changes (reduce debt, increase savings)
Development of financial plans
Changes in spending habits
Building or rebuilding credit reports and credit scores
Putting it all together….
Creating Your
“Evaluation Road Map”
Signature Program
Program
Objective 1
Program
Objective 2
Program
Objective 3
Indicator 1a
Indicator 2a
Indicator 3a
Indicator 1b
Indicator 2b
Indicator 3b
Indicator 1c
Indicator 2c
Indicator 3c
Final program outcomes
A few words of caution
when selecting indicators….

Measurement error and validity of indicators.




Financial knowledge
Confidence level
Financial behavior
EXAMPLE: Which of the following are valid indicators
of behavior change?
 Participant opened a bank account.
 Participant increased savings.
 Participant avoided bankruptcy.
 Participant did not default on mortgage payments.
One non-profit administrator commented….
“What is driving this financial education movement? Why is it so
important? What are we ultimately trying to address? Is it
reducing the poverty gap in this country? Between those that have
and those that don’t have. And it’s widening. And those at the
bottom end of the spectrum….what we’re asking them is to build
wealth. And at the same time, what we’re asking people in this
country who make $20,000 or less is: ‘Absent us raising your
wages in this country, we’re asking you to build wealth, to
participate in IDA programs. We’re asking you to save with the
little amount of money you’re making. We’re asking you to reduce
your debt burden, learn how to manage your money, and clean up
your credit history with the little amount of money you’re working
with. And we want you to get from point A to point B with all those
constraints.”
Source: Lyons, A. C., Palmer, L., Jayaratne, K.S.U., and Scherpf, E. (2006). "Are We
Making the Grade? A National Overview of Financial Education and Program Evaluation.”
The Journal of Consumer Affairs, 40(2), 208-235.
A few additional “words of wisdom”

Avoid indicators that may be subject to biases.





Social desirability
Norms and “rules of thumb”
Misperceptions and over-optimism
Memory distortion and recall bias
Avoid indicators that may ask sensitive or personal info.
 Non-response bias
 Program attrition
 Self-selection
 Low response rates (e.g., follow-ups)

Environmental factors can affect outcomes.




Unexpected life events
Program incentives (e.g., rewards, special benefits,
enrollment programs)
Individualized financial advice or “coaching”
Psychological factors.
 Inherent motivation
 Ability
 Attitudes
There is no “magic set” of outcomes or indicators.
The list is endless….







Increases in savings.
Decreases in debt.
Maintaining a regular budget.
Comparison shopping.
Increases in new accounts opened.
Improved credit scores.
Improved communication with spouse/partner/parents
about finances.
Other common indicators?
How do these indicators
change for various
target populations?
Youth?
Underserved?
Adults?
Members of your
organization?
ACTIVITY:
Your Evaluation Action Plan – Part A
 What is your signature program (e.g., course, workshop,
educational materials, initiative, campaign)?
 How are you going to define program success? At the
end of the day, what do you want to show?
 Who is your target audience(s)?
 What is your delivery method(s) (e.g., in-person,
telephone, Internet)?
 What financial and non-financial resources are available to
you for evaluation?
 Who will use the evaluation and how will it be used?
Summary: 4 Steps to Creating a Successful
Evaluation Action Plan
Step 1: Define objectives of the program.
At the end of the day, what do you want to show?
Who will be the target audience?
Who will use the evaluation and how?
Step 2: Select appropriate outcomes, indicators, and
methods.
What is the most appropriate evaluation format?
What types of questions will the evaluation seek to answer?
What types of indicators will be used to show
impact?
Step 3: Identify resources and constraints.
What financial and non-financial resources are available?
Are there others who can help?
What is the program timeline?
Given constraints, what can you realistically do?
Step 4: Analyze and disseminate findings.
How will data be analyzed?
What do you hope to learn from findings?
What are the potential impacts?
How will the results be used and disseminated?
Where do we go from here?
Evaluation resources at your fingertips
U of I Center for Economic and Financial Education
http://www.cefe.illinois.edu/
Educational Tools for Evaluation
http://www.cefe.illinois.edu/tools/evaluation.html
Checklist of evaluation resources
Program Planning Guide
Evaluation Action Plan
Evaluation Road Map
Evaluation Reading List
Sample Evaluations
NEFE® Financial Education Evaluation Toolkit
http://www.cefe.illinois.edu/tools/evaluation.html
University of Wisconsin-Extension
http://www.uwex.edu/ces/pdande/evaluation/index.html
Cornell University Extension
http://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm
Penn State Extension
http://www.extension.psu.edu/evaluation/
Program Evaluation II:
Creating Your Evaluation Toolkit
More to come soon….
Contact Information
Dr. Angela Lyons
Associate Professor
Director, U of I Center for Economic and Financial Education
University of Illinois
Phone: 217-244-2612
E-mail: [email protected]
Questions