Transcript Helping our Clients to Measure Outcomes

Helping Our Clients
Identify and Measure
Outcomes
ESC Brown Bag # 2, April 17 2013
Edward M. Haugh Jr.
ESC Consultant
Introduction
By the end of this session:
• Define outcomes
• Differentiate outcomes from outputs
• Define the logic model
• Describe several uses of a logic model
• Develop a logic model
• Define “indicators”
• Develop a logical framework
Impact
• You will love or at least tolerate outcomebased planning
• You will be able to discuss outcomes based
planning, monitoring and evaluation with your
ESC clients
What is an outcome
Outcomes are benefits or
changes for individuals or
populations during or after
participating in a program:
Outcomes
• Often expressed as verbs in past tense
–Improved (nutrition, morale)
–Increased (in employment)
–Reduced (HIV transmission)
–Enhanced (knowledge)
–Maintained (vaccination coverage)
Outcomes
Reduced smoking among teenagers
Changed attitudes towards smoking
(by non-smoking teenagers
School readiness among children
attending preschool program
Other examples
Improved board performance after an ESC
engagement
Improved strategic focus after an ESC
engagement
Employment after attending a dress-forsuccess program
Improving the Delivery of Prenatal Care and Outcomes of
Pregnancy: A Randomized Trial of Nurse Home Visitation
Outcomes of Nurse Visit Program
• Became aware of more community services
• Attended birth classes more frequently
• Made more extensive use of the nutrition
supplementation program for women, infants
and children
• Made greater dietary improvements,
Outcomes of Nurse Visit Program
• Reported that their babies’ fathers became
more interested in their pregnancies
• Were accompanied to the hospital by a
support person during labor more frequently
• Reported talking more frequently to family
members, friends and service providers about
their pregnancies and personal problems
• Had fewer kidney problems
Outcome Indicators
• Outcomes sometimes confused with outcome
indicators
– Outcome - girls participating in a training program
obtain formal employment
– Outcome indicators – the # and % of participants
employed within 3 months of completing the
program
Importance of Focusing on Outcomes
• Outcomes are what matter to stakeholders
• Focusing on outcomes promotes
accountability
• Government, and funders demanding real
tangible results on the ground
• Measuring outcomes helps us know if we are
we really getting value for our money
Outcomes help us
•
•
•
•
•
•
Strengthen existing services
Target effective services for expansion
Identify staff and training needs
Develop and justify budgets
Prepare long-range plans
Focus board on programmatic issues
Funder Asked ESC the Following:
• What are your measurable outcomes, i.e.
number of communities impacted, number of
children served?
• Mark each category that applies to your grant
Increased
visibility/awareness of issue
measures.
Effected policy change
Engaged constituents & beneficiaries and participation
Increased resources for issue
Educated constituents
Improved condition/ quality of life in community
Influenced stakeholders
Outcomes vs. Outputs
• Outcomes are the results
• Outputs are the tangible products
produced by an intervention
–Usually expressed as nouns
–Can be counted
Outputs
•
•
•
•
•
•
Classes taught
Brochures distributed
Clients served
Contracts completed
Board retreats completed
Strategic plans prepared
Many Nonprofits Report Outputs
Rather than Outcomes
• Main Street Homeless Shelter provided 2,903
nights of safe shelter for 173 men.
• 23,661 hot, nutritious meals were served at
the Maple Avenue Kitchen.
• 100 residents received free health care and
168 residents received free mental health
care.
What Wrong with Using Outputs
• Mosquito bed nets might be used for
wedding dresses or fishing nets
• Loans repaid might not mean income
generated
• Job training might not lead to jobs
So Why Don’t We Focus on Outcomes
• Outputs are tangible
• Outcomes are intangible
• Its easier to count tangibles than to describe
intangibles
• Outcomes take longer to produce
• Measuring outcomes is challenging
• Measuring outcomes can be expensive and
time consuming
But Tracking Outcomes will Not……
• Explain why an outcome is or is not being achieved
– Still need to track inputs, activities, outputs
• Prove that the outcome is the result of your program
Outcome Model
Logic Model
Inputs
Processes
Outputs
Outcomes
Impact
Logic Model Definitions
• Inputs are the resources required to
undertake activities
• Activities (or processes) are what the agency,
program or strategy does to fulfill its mission
• Outputs – tangible products
• Outcomes – changes, benefits
• Impact - a longer-term outcome
A logic model for a program to distribute HPV
Vaccine
Processes
Inputs
•
•
•
•
•
•
•
Money
Staff time
Facilities
Equipment
Laws
Regulations
Funders
requirements
•
•
•
•
•
•
•
•
•
•
Brochures
Vaccine supplies
Doses of
vaccines
delivered
Data bases
established
Providers
monitoring
•
•
•
•
Parents aware
of vaccine
benefits
Provider
awareness
Changed
attitudes
Children
immunized
Mediumterm
Outcome
•
Reduced
incidence
of HPV
Reduced
number of
girls who are
infected with
HPM
Long-term
•
Reduced
incidence
of cervical
cancer
Reduced
mortality for
cervical cancer
Impact
•
Educate
consumers
Educate
providers
Distribute
vaccines
Establish
data base
Establish
reminder
systems
Monitor
Coverage
Short-term
Outcomes
Outputs
Reduced premature death
It is important to note the broad environmental context of the intervention:
Policy context
Macroeconomic picture
Inputs
Other “players” in field
Activities
Outputs
Environment
Political environment
Outcomes
Impact
Public attitudes
Source: Imas and Rist (2009)
How can we use the logic model
•
•
•
•
ESC Project planning
Strategic Planning
Program Monitoring
Program Evaluation
Logic model for an ESC Engagement
Processes
Inputs
•
•
•
•
•
•
•
•
•
Staff time
CRM Time
Lead time
Consultant
time
Materials
Car and
gasoline
Knowledge
Experience
Place to meet
•
•
•
•
•
•
Contracting
Analysis
Study
Meetings w
client
Meetings with
stakeholders
Consultant
meetings
Outputs
•
•
•
•
Progress
reports
Draft report
Final report
Board
presentation
Deliverables
Mediumterm
Outcome
Short-term
Outcomes
?
Impact
?
•?
Example of a logic model for an ESC Strategic
Planning Engagement
Inputs
Short-term
Outcomes
Strategic Plan
Approved
Outputs
Process
Medium-term
Outcome
Strategic Plan in
regular Use
Longerterm
Outcome
Strategic Plan
improves agency
performance
Impact
Community
benefits from
agency’s
improved
performance
Non-profit Logic Model
Inputs
Programs
Outputs
•
•
•
•
•
•
•
Money
Staff time
Facilities
Equipment
Laws
Regulations
Funders
requirements
Program A
Program B
Short-term
Outcomes
Output A
Output B
Output C
Outcome 1
Impact
Outcome 2
Program C
Program D
Output D
Outcome 3
Improved Ability to Fulfill Mission
Outcome 1:
Improved
Financial
Position
Outcome 2:
New Facility
Developed
Outcome 3:
Strengthened
Volunteer
force
Outcome 4:
Improved
Public
Recognition
Outcome 5:
New Training
Program
Developed
Improved Ability to Fulfill Mission
Outcome 1:
Improved Financial Position
Strategy 1:
Capital Campaign
Completed
Strategy 2:
Increased donor
base
Strategy 3: Client
Fees Increased
Indicators
• Measure the quantity or quality of the
outcome
Indicators
Outcome: Participants in a job training program
obtain a job in a timely manner
• # (%) of participants employed within 6 weeks
• # (%) of participants still employed 3 months
after completing the program
Indicators
Outcome: Homeless people obtain secure
housing
• # (%) of homeless moved into temporary
quarters and stayed for 6 weeks
• # (%) of homeless who obtained a job and
were still employed after 3 months
Indicators Targets
• # (%) of participants
employed within 6
weeks
• # (%) of participants still
employed 3 months
after completing the
program
• 60% of participants will
gain employment within
6 weeks
• 95% of participants who
obtained employment
will still be employed in
3 months
Logical Framework
Component
Outcome
Output
Program/
Activity/
Process
Input
Indicator
Baseline
Target
What Can Our Clients Do to Measure Outcomes
7. Use the Findings
6. Improve the System
5. Analyze and Report Findings
4. Try out the Measurement System
3. Prepare to collect data on the Indicators
2. Specify the Indicators for the Outcomes
1. Choose
the Outcomes to Measure
Two Great Resources
• Measuring Outcomes, A Practical
Approach, The United Way
Foundation
• Logic Model Development Guide,
WK Kellogg Foundation
Thank you for participating in ESC
Brownbag No. 2.
Look for more Brownbags to come