Transcript Smart Investing @ your library Orientation

OBE: Outcomes-Based
Evaluation
ALA Midwinter Meeting
January 24th, 2014
Presented by Kit Keller
Workshop goals

Provide an understanding of outcomes-based
planning and evaluation so grantees may refine
Smart investing@your library® evaluation and
marketing plans.

Set the stage for effective program
implementation.
2
Workshop schedule
Part 1 – What is outcomes-based evaluation?
Part 2 – What data should I collect? When?
3
What is outcomes-based evaluation?

What are outcomes?

Will I know it when I see it?

How can I track them?
4
What are outcomes?
Outcomes are benefits to people.
Outcomes are changes in skills, knowledge,
attitude, behavior, condition, or life status.
5
Like what?

Learned how to read a credit report. Knowledge

Learned how to compare mortgage loan offers.
Knowledge and Skill

Can balance my checkbook. Skill

Know how to safely pay bills online. Skill

Feel in control of finances. Attitude
6
What else?
BEST!

Refinanced mortgage based on content of
presentation. Behavior

Established a personal savings account. Behavior

Paid off a credit card balance in full. Behavior

Deposited money in a retirement account.
Behavior
7
Behavior changes …
…lead to changes in life status.
8
Why measure outcomes?

See if programs really make a difference in the
lives of people.

Improve programs.

Improve planning.

Improve accountability.

Ensure best use of funds.

Demonstrate impact.

Satisfy funders.
9
What can OBE do?

Improve program effectiveness.

Demonstrate success.

Facilitate program adjustment.

Inform future project planners.
10
Process and impact

Evaluating the process: allows you to see how
efficient your program is.

Evaluating the impact: allows you to know if your
program is making a difference.
11
Grant program elements
Inputs
Activities
Outputs
Outcomes
Resources used by
the program
Actions of the
program
Counts of activities
Measures of
participant change
12
Examples of inputs, activities,
services, and outputs
Inputs:
Resources
dedicated to or
consumed by the
program.
Activities:
Program actions
that are
management
related.
Services:
Program actions
that directly
involve end
users.
Outputs:
Numbers of
direct program
products
Staff, time,
computers,
facilities,
materials, money,
consultants,
website,
software,
Internet,
instructors
Recruiting,
coordinating,
promoting,
purchasing,
scheduling, and
evaluating
activities.
Conducting
workshops,
mentoring, online
offerings,
following up with
customers.
Number of
participants
served;
materials
developed and
used;
workshops
offered; website
usage counts.
13
Outputs and outcomes

Outputs are not outcomes.

Outputs tell us how much we’ve done.

Outcomes tell us how much difference we’ve
made.

Outcomes do not replace outputs; they
complement them.

Both are important to provide a full picture of a
program’s results.
14
Programs and services

Workshops

Classes

Marketing

Collection development

Partnership development
15
Smart investing@your® library
program goals
Community members will:

view the library as a reliable place for
unbiased financial and investment
information. [Attitude]

make increased use of library programs and
resources. [Behavior]

be more knowledgeable about key financial
and investment issues. [Knowledge]
16
Individual project goals – examples

Increase access to financial literacy materials.

Improve reference skills of staff in areas of finance
and financial literacy.

Increase investment knowledge of target
audience.
17
Project outputs – examples
o 250 participants attended 5 workshops
o 45 participants attended counseling sessions
o 1,200 children participated in kick-off program
o Purchased 38 eBooks and 41 audiobooks
o In 4 months each new book was borrowed an average
of 8.78 times
o 2,600 brochures were distributed
18
Outputs vs. outcomes

Three programs held

Participants know about
several types of investment
vehicles.

Website developed

Patrons regularly use
investment website to help
with personal finance
decisions.

Print, electronic materials
increased 10%

Patrons use materials from
investment collection to
inform decision-making.

100 PSAs run

Participants report PSA
motivated attendance.
19
Outcomes defined
Outcome:
A target audience condition changed or improved – a change in skills, attitudes,
knowledge, behaviors, status, or life condition brought about (partly or wholly) by
experiencing a program.
Type of Outcome
Definition
Example
Knowledge
What someone knows
Participants will increase their understanding of credit
scores.
Skill
What someone can do
Participants will create a household budget.
Attitude
What someone feels or thinks about
something
Workshop attendees are interested in learning more about
controlling their own finances.
Behavior
How someone acts
High school students research options for college
scholarships and other funding sources.
Status
Someone’s social or professional condition
More high school seniors will apply for scholarships.
Life condition
Someone’s financial condition
Rate of foreclosures will drop as a result of more
financially informed community members.
20
Get ready
Smart investing goals at the library level:

Increased requests for investment materials

Increased visits to library website

Increased awareness of library resources

Increased staff competencies
21
Sample outcome statements

All reference staff can use key financial and
investment resources.

Staff coordinate financial literacy training for
patrons.

Patrons know where to find credible, unbiased
financial information online.
22
Outcomes categorized

Immediate (short term)
 Likely to be changes in attitudes, skills, and
knowledge
 Occur during program cycle

Intermediate (medium term)
 Likely to be changes in behavior or decision making
 Can occur a few months into program cycle and a
few months after program completion

Permanent (long term)
 Likely to be changes in life status or condition
 Occurs sometime after program cycle
23
Get ready
Smart investing program elements

Classes/programs/training/exhibits

Staff training

Partnerships

Collection development and positioning in
physical/virtual library

Web presence

Marketing/outreach
24
Building outcome statements

Focus on audience

Staff learn about key
financial and investment
resources

Identify the anticipated
change

Staff provide financial
literacy training to
patrons

Keep it simple


Check that they are
SMART
Patrons know where to
find credible, unbiased
financial information
online
25
S–M–A–R–T?

Specific

Measurable

Achievable

Relevant

Time-specific
26
Example…
Five members of reference team will increase their
knowledge of Morningstar database by 20% after
completing three training workshops offered in the
spring.
27
How will you know?
Five members of reference team will increase their
knowledge of Morningstar database by 20% after
completing three training workshops offered in the
spring.

In order to demonstrate change, you have to
establish a starting point.
28
Learn from the best!
Most Effective Evaluation Techniques
29
Challenges to Evaluating Effectiveness
Non-response by participants
20
Hard-to-collect information
14
Time constraints
11
Difficulty developing surveys
10
Other
7
Difficulty hosting focus groups
3
0
5
10
15
20
25
30
“Other”?

Older adults had difficulty with lots of paper. They seemed to complete
front pages only or skipped many questions.

Getting those pesky longer term outcomes that you know are out there.

Once the activity is complete, so is their obligation to participate. You can
tell a program is effective while it's happening. Are people showing up to
each meeting? Are they engaged, asking questions? That's how we know.

Even though we prepare participants to let them know we will follow up
after a program to capture outcomes, we don't get good responses on
follow up email surveys or phone calls. Bringing teens together at the end
of a series of program, in an informal focus group, helped to provide us
with some really good information.
31
Bringing it home…





Involve staff, early & often.
Embrace the project.
Involve the community, early &
often.
Make friends with local media.
Ensure privacy – continually.
32
In their own words…
Key components of effective financial
literacy instruction & programming:
•
•
•
•
Knowledge
Objectivity
Privacy
Empathy
33
Hands-on

During the break: Write a goal for your target
audience.

Identify outcomes, indicators and potential data
sources for at least one outcome.
34
Workshop schedule
Part 2 –What data should I collect? When?
35
Choose the outcomes
you want to measure
Smart investing outcomes at the library level:

Users demonstrate increased skills and/or
knowledge

Users take action with new skills (i.e. start
investing, reduce debt)

Program partners report a positive experience
working with the library

Users participate in programs as a result of
PR/marketing activities
36
How do you know?
Outcomes
Indicators
Source/Method
Participants know how to
use financial databases
(knowledge/skill)
Data-usage statistics show Establish baseline data
20% increase in four
use; measure use over four
months
months following training
Participants establish
regular savings activities
(behavior)
Increased number of
participants with savings
accounts
Pre/post-survey responses
37
Staff example

GOAL: Increase staff competency in providing
financial/investor education information

Outcome: Reference staff are comfortable
providing financial/investor information.

Outcome: Reference staff know sources of
accurate and unbiased information on
investing.
38
Identify indicators for your outcomes


Indicators: measurable conditions or behaviors
that show an outcome was achieved:

What you hoped or intended to see or know

Observable evidence of accomplishment,
changes, gains
For each outcome generate a list of possible
indicators and then narrow to at most three that
best show the outcome was achieved
39
Specify indicators for your outcomes

Be SMART with indicators

Use the formula:


Number and/or percent of a specific target population who report,
demonstrate, exhibit an attitude, skill, knowledge, behavior, status, or
life condition in a specified quantity in a specified timeframe and/or
circumstance
Examples:

Outcome: Reference staff will know sources of accurate and unbiased
information on investing


25 or 50% of staff will be able to name 3 online resources that provide
financial/investor education information after attending a training
Outcome: Users take action to better their personal finance

The # and % of users who report they made one or more life-style changes
from a list of 10 key personal finance factors in the last six months
40
Prepare to collect data
on your indicators

Data Sources: Tools, documents, and locations
for information showing what happened to target
audience.

Data source options:








Feedback forms/short surveys
Point-of-use inquiry by staff
Focus groups
Interviews
Skills assessments
Observation
Instructor assessments
Library use statistics
41
Prepare to collect data
on your indicators

Pre-survey and Post-survey

You can’t measure success without a baseline.

What is the “current state of affairs” — what do
people know, perceive and do before the
program… and how does the program move
the audience forward?

Include all stakeholders.

Retrospective surveying

Post-observation interview to clarify learning
that occurred during project activities.
42
Prepare to collect data
on your indicators

Other considerations:

When will you collect data?

How often will I collect data?

Include all participants or a sample?

Who will collect data?

Who will record/compile data?

How will confidentiality be protected?

How will participants be informed about the
data collection process?
43
Test your measurement system

Pilot or “beta test” your surveys or
questionnaires.

Clarity of questions

Ease of use

Are you measuring what you intended to
measure? Are you asking the most appropriate
questions?

REVISE instrument as needed!
44
Analyze and report findings
o Review feedback from participants
o Collect, input data at regular intervals
o Get familiar with the data
o Look for and note oddities in reporting
o Peruse the data and identify patterns
o Substantiate patterns — do data sources
corroborate each other?
45
Analyze and report findings

Organize data logically (tables/charts)

Analyze and interpret data to develop narrative for
final report

Document findings

Maintain files or database of outcomes and
activities

Determine outcomes you want to continue
monitoring
46
Use your findings

Tell your story!

Marketing

Accountability and long-term assessment

Improved services and/or programs

Resource (re)allocation

Include data and anecdotes
47
Additional resources
IMLS
www.imls.gov/applicants/outcome_based_evaluations.aspx
Shaping Outcomes
www.shapingoutcomes.org
United Way
www.slideshare.net/lynnereed/united-way-logic-model-presentation
48
Questions?
Kit Keller
[email protected]
402-450-4639
49