Transcript Hispanic Needs Educational Assessment Workshop

University of Wisconsin-Extension
Cooperative Extension
Evaluating Financial Education Programs:
A framework for measuring results
Ellen Taylor-Powell, Ph.D.
Evaluation Specialist
American Savings Education Council
April 18, 2007
Washington, D.C.
The state of affairs …
• Evidence tells of the situation we face: low
saving rates; high debt; high bankruptcy
rates, shift in policies, sophisticated financial
markets and products…
• Partners have responded in force to build
financial literacy with a myriad of investments
and activities
• The scope and size of the financial
educational effort is impressive
Our purpose today…
What difference is all this effort making?
Does financial education work?
How do we know?
• Examine the logic model as a framework for
planning and evaluation
• Explore evaluation challenges and options
• Discuss how we can make evaluation
meaningful
Accountability age
No longer good enough to report
numbers of services delivered; number
of participants reached
Need documentation of changes:
improvements for individuals, families,
communities
Need to be able to answer:
So what?
“I think you should be more explicit here in Step Two.”
What is a Logic Model?
A graphic depiction of a program that shows the
relationship among activities and intended results
– Linkage between and among components: Program “theory”
or “program action”
– Intended results: what the program is to accomplish
Serves as a tool for integrating planning and evaluation
– Helps with both planning and evaluation
Is not an evaluation model, method or reality
Is widely used: nonprofits, public sector, foundations,
international agencies, evaluation community
Other names: theory of change, program action, program
theory, causal road map
A Family Vacation
Family Members
Drive to state park
Budget
Set up camp
Car
Camping
Equipment
Cook, play, talk,
laugh, hike
Family members
learn about each
other; family
bonds; family has
a good time
A youth financial literacy program
Teens establish sound financial
habits
Teens make better decisions about
the use of money
Teens gain knowledge and skills in
money management
A high school financial planning program – 7
unit curriculum - is developed and delivered
in high schools
Partners invest resources
Comprehensive Cancer Control
IF
THEN
Build
Partnerships
Enhance
Infrastructure
Utilize
Research &
Data
Integrated &
Coordinated
Plans with
Effective
Interventions
System,
Environmental,
& Policy
Changes
Individual
Behavior
Change
New Cases
Prevented
Decrease
Morbidity
Cancers
Detected
Earlier
Decrease
Mortality
State-of-the-Art
Treatment
Reduce
Disparities
Appropriate
Rehabilitation
and Support
Increase
Quality of
Life
Provide
Education
Effective
Palliation
Common graphic of logic model
the “four box approach
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
A bit more detail
INPUTS
Program
investments
What we
invest
OUTPUTS
Activities
Participation
What we
do
Who we
reach
OUTCOMES
Short
Medium
What results
SO WHAT??
Longterm
Fully detailed
OUTPUTS
What we do
ACTIVITIES
• Assess needs and assets
• Design and deliver
curriculum
• Conduct workshops; teach
• Provide one-on-one
counseling
• Produce products
- Exhibits
- Printed materials
- Electronic materials
•Populate/maintain web sites
• Hold Conferences
• Work with the media
• Facilitate policy dialogue
• Partner – collaborate
• Advocate
Who we reach
PARTICIPATION
•Underserved populations
Low income consumers
•Youth: elementary, high
school, post grad; college
students
•Homebuyers/homeowners
•Military personnel
•Teachers/educators
•Immigrant populations
•Mandated audiences
•Financial professionals
Reactions – Satisfaction
OUTCOMES
What results for individuals, families, communities..…
SHORT
MEDIUM
LONG-TERM
Learning
Action
Conditions
Changes in
Changes in
Changes in
• Awareness
•Behavior
• Human
•pay bills on time
•well-being
• Knowledge
+ savings
•stability
• Attitudes
+ assets
•security
• Skills
–debt
• Economic
• Opinion
•achieve
• Community
• Aspirations
personal goals
–civic activity
• Motivation
•spend wisely
• Behavioral intent
•track spending
and income
•Decision making
•Policies
•Social action
CHAIN
OF
OUTCOMES
Social-economicenvironmental improvements
Hierarchy of
effects
Source: Bennett and Rockwell, 1995,
Targeting Outcomes of Programs
Actions
Changes in behaviors
and practices
Learning
Changes in knowledge, attitudes,
skills, aspirations
Reactions
Degree of satisfaction with program; level of
interest; feelings toward activities, educational
methods
Participation
Number and characteristics of people reached;
frequency and intensity of contact
Simple logic model
SITUATION: During a county needs assessment, majority of parents reported that they were
having difficulty parenting; were often uncertain about what to do and felt stressed as a result.
INPUTS
Staff
Money
Partners
Research
OUTPUTS
Parents
increase
knowledge of
child dev
Develop
parent ed
curriculum
Deliver
series of 8
interactive
sessions
Facilitate
support
groups
OUTCOMES
Parents
of 3-10
year olds
Parents better
understanding
their own
parenting style
Parents gain
skills in effective
parenting
practices
Parents
increase
knowledge of
community
resources
Parents
identify
appropriate
actions to
take
Parents use
effective
parenting
practices
Improved
childparent
relations
Elaborated logic model - Parent support initiative
Situation
OUTCOMES
Develop, deliver
parent ed
curriculum
Staff
Facilitate parent
support groups
Time
Money
Volunteer
Partners
Materials
Supplies
Researchbest
practices
Provide
individual
services;
counseling
Provide
parent/child
Activities:
Play group; Toy
lending
Parents/ caregivers:
Single parent
Couples
Divorced
Teen parent
Fathers
Low-income
Homeless
Mandated
Children
Ages
Low-income
Special needs
Changes in
knowledge, skills
intent re. child dev.;
parenting style
Demonstrate
positive parenting
practices;
improved family
communications
Increased knowledge
of community
resources
Use community
supports/resources
Increased ability to
advocate for self
and families
Increased abilities to
socialize and
communicate
Advocate for self
and family
Demonstrated
language,
communication and
social skills
Do home
visits;Outreach
Train
Support
Do assessments
Childcare
providers:
directors, staff
Increased
knowledge of quality
improvement
strategies:
developmentally
appropriate practice,
child assessments,
learning environments
Implement quality
improvement
strategies
Early detection of
delays and referrals
Improved
child-family
relati’ships
Inclusive,
appropriate
services
available
Children
enter school
ready to
learn
Families
connected
to
community
What does a logic model look like?
•
•
•
•
Graphic display of boxes and
arrows; vertical or horizontal
– Relationships, linkages
Any shape possible
– Circular, dynamic
– Cultural adaptations; storyboards
Level of detail
– Simple
– Complex
Multiple models
– Multi-level programs
– Multi-component programs
PLANNING: start with the end in mind
What do you want to know?
How will you know it?
EVALUATION: check and verify
Plan backwards
PLANNING: start with the end in mind
What do you want to know?
How will you know it?
EVALUATION: check and verify
Logic Model and Planning
• Apply to any level: national plan,
statewide plan, individual plan of work,
specific project/activity plan
• Model vs. a more detailed narrative plan
or a management plan
• Focus on outcomes: “start with end in
mind”
Integrate evaluation into program
planning
Think about:
• What do you (and other key stakeholders) want to
know?
• What data will you want to have?
• When will it be appropriate to collect data?
• What evidence will be credible; believable by
various consumers? If you told me that people are
benefiting, what would I see if I came to visit?
• Is it your program or something else that is making
this difference?
EVALUATION:
check and verify
What do you
want to know?
How will you know it?
Match evaluation questions to program
INPUTS
Program
investments
OUTPUTS
Activities
Participation
OUTCOMES
Short
Medium
Longterm
Evaluation questions: What questions do you want to answer?
e.g., accomplishments at each step; expected causal links; unintended
consequences or chains of events set into motion
Indicators: What evidence do you need to answer your
questions?
Example: Evaluation questions and indicators
Staff
Money
Partners
Research
Parents
increase
knowledge of
child dev
Develop
parent ed
curriculum
Deliver series
of 8
interactive
sessions
Parents
of 3-10
year olds
Parents gain
skills in
effective
parenting
practices
Facilitate
support
groups
EVALUATION QUESTIONS
What amount
of $ and time
were
invested?
How many
sessions were
held? Quality of
implementation?
#, quality of
support groups?
Parents better
understand
their own
parenting style
Who/how many
attended/did not
attend? Did they
attend all sessions?
Supports groups?
Were they satisfied
– why/why not?
To what extent
did knowledge
and skills
increase? For
whom? Why?
What else
happened?
Parents
identify
appropriate
actions to
take
Parents use
effective
parenting
practices
To what extent
did behaviors
change? For
whom? Why?
What else
happened?
Improved
childparent
relations
To what
extent are
relations
improved?
INDICATORS
# Staff
$ used
# partners
# Sessions
held
Quality criteria
#,% attended
per session
Certificate of
completion
#,%
demonstrating
increased
knowledge/skills
Additional
outcomes
#,%
demonstrating
changes
#,%
demonstrating
improvements
Types of
changes
Types of
improvements
Logic model and evaluation
Needs/asset assessment:
What are the
characteristics, needs,
priorities of target
population?
What are potential
barriers/facilitators?
What is most appropriate
to do?
Process evaluation:
How is program
implemented?
Are activities delivered
as intended? Fidelity of
implementation?
Are participants being
reached as intended?
What are participant
reactions?
Outcome evaluation:
To what extent are desired
changes occurring?
Goals met?
Who is benefiting/not
benefiting? How?
What seems to work? Not
work?
What are unintended
outcomes?
Impact evaluation:
To what extent can
changes be attributed to
the program?
What are the net
effects?
What are final
consequences?
Is program worth
resources it costs?
Key considerations
• Match evaluation to stage of program’s
development
• Define evaluation purpose
• Identify specific evaluation questions
• Select measurable indicators
• Match design to purpose, audience and
resources
• Attend to cultural relevance
Evaluation challenges
Internal
• Commitment
• Training
• Common language
and understanding
• Value
• Leadership
• Resources
Community context
• Complex issues
• Lagged outcomes
• Attrition
• Non-response
• “Noisy” systems
• Implementation fidelity
• Varied programming:
contexts, content,
intensity, audience and
implementation
So, what can we do?
Internal
• Build capacity
• Provide leadership
• Showcase use
• Institute operational
and structural
changes
• Provide resources
• Practice, practice,
practice
Community context
• Use logic model to
show connections
• Match evaluation to
purpose, program,
questions
• Use multiple sources
of information
• Mix methods
• Conduct formative
research
Summing up
Logic model links activities to results
 integrates planning and evaluation
 helps match evaluation to the program
effort
 identifies points for appropriate data
collection
Evaluation challenges are many but not
insurmountable!
Remember that evaluation equals learning
Resources
•
•
•
•
•
www.uwex.edu/ces/lmcourse
www.uwex.edu/ces/pdande
http://www.cdc.gov/eval/index.htm
http://citnews.unl.edu/TOP/english/
Weiss, Carol. 1998, Evaluation,
2nd Ed; Prentice-Hall, Inc.
“We build the road and the road builds us.”
-Sri Lankan saying