Transcript Evaluation 101 – Evaluation for the small community based

Evaluation 101 –
Evaluation for the Small Community
Based Organization
Leticia Peguero
Ayorkor Gaba
The Robert Wood Johnson Foundation
To know how well you’re doing…you
must have some place you’re trying to
get to….
• “If you don’t know where you’re going,
you’ll end up somewhere else….
•ASSUMPTIONS
•VALUES
•BELIEFS
Pathway Mapping
How does it work?
•We all hold beliefs and theories that guide our
thinking about problems and the way we respond
to them.
•Clarity about the problems and about how to
work with these problems is critical because
these ideas and beliefs guide our work…
Pathway Mapping
How does it work?
• Define key problems/issues challenging our communities and
clients
• What are our UNDERLYING ASSUMPTIONS?
• Describe WHAT we are hoping to change (our desired outcome)
in this community through our intervention
• Describe HOW (strategies, actions, interventions) we hope to
accomplish these outcomes
Pathway Mapping
How does it work?
•Examine assumptions about what works….
• While assumptions can be based on experiences from the field,
sometimes they are built on intuition. Challenging these assumptions
allows us to look more closely at their theory and reflect on the choices
we are making.
Pathway Mapping
How does it work?
•Think about the assumptions we are making concerning WHY an
intervention might work...
•Discuss differences in assumptions among stakeholders in your
organizations
•CHALLENGE THOSE ASSUMPTIONS!!
•What and whose experiences are they based on?
•Think about the assumptions that we are
making concerning WHY a particular
intervention might work
•We don’t want our theory to fall apart
so…lets identify our assumptions…
Strategies
Actions
Outcomes
• Closely examine the links between our
strategies, actions and outcomes and ensure they
are logical and achievable
•Identify and address gaps in our strategies –
paths that might not get us where we want to go
Context:
Identify the major
conditions that serve as the
backdrop to your
work…what are the
opportunities and
challenges surrounding
your work
Goals/Strategy of the Program
Positive changes that you hope to
achieve in the long run
The overall approach to
conducting your work. If you
cannot implement all of the
goals…prioritize them
Actions/Program Activities
Activities are the individual services
your organizations uses to
implement strategies. Identify the
services you want to provide
Long Term Outcomes
Short Term Outcomes
Assumption
s
Identify the short term changes in
your target population after they have
participated in your program/activity
Theory of Change…..
Theory of Change…
• What is it ?
• A theory of change explains your underlying
understanding of the issue you are addressing
•It helps to clarify
WHY you are doing
– WHAT your doing
Developing a
theory of
change…
• Convene stakeholders for a planning session
•Designate a facilitator
•Address each element of the theory of change
•Share with stakeholders and revise based on
feedback
Using a theory of change
This should be a living document
Should be kept and
revised and
compared with goals
and outcomes of the
program/s
Can be used as a
reference tool
Should be
transparent
Using a Theory of Change
As a planning and
communication
tool it can:
Can provide
opportunities to
correct faulty
underlying
assumptions about
the program’s
design
Can help inform
decisions about
changes to the
program/s as it
evolves
Involve
stakeholders in
planning and
helping them get
clarity and
consensus around
program activities
and outcomes
Using a Theory of Change
As an
evaluation
tool:
• It can lay the foundation
for the evaluation by:
• Helping program staff
identify outcomes that
should be measured
• Provide a map to inform
your understanding of
program outcomes
OUTCOMES
Outcomes ???
Outcomes are:
•
•
•
•
•
•
• Changes in:
behavior
skills
knowledge
attitudes
condition
status
Outcomes should be:
• Related to the
business of the
project
• REALISTIC and
ATTAINABLE
• RELEVANT to the
project
• Within the program’s
sphere of influence
OUTCOMES….
are what a program
is accountable for…
•
Grantee Outcomes…
The group,
condition, process
targeted for change:
And what we hope
to change about it
Maya Language
speakers in South
Florida
Improve
communication and
coordination of mental
health services for the
Mayan community
Refugee DV offenders,
Increase community
awareness about
Domestic Violence in
the refugee community
Restaurant workers in
NYC
Develop restaurants
workers knowledge
about health resources
Outcomes
Are logical and reasonable if there
• Is a logical connection between our activities
and outcome
• Activity: English Language Learners in
Lincoln Nebraska will be given instruction on
English language terms for anatomy,
symptoms, interacting with health care
providers
• Potential Outcome that we can measure….
Outcomes…
Example:
• Center for New Community will Improve
services by health care providers to the
Spanish speaking population of the
twelve target communities
• Possible measurable outcome:
Outcomes…
Are logical and reasonable if it is…
• Reasonable to believe that the outcome can
be accomplished within the timeframe that a
program has.
• Based on:
• Program previous experience
• Context
• Resources
What do we mean by Outcomes?
Access
HealthColumbus
• Target Group:
• Latino Immigrants
• Hope to Change:
• Overall Health
• What are you striving for – Short Term?
• Identify strategies to improve health
• Develop models
• Select models
What do we mean by Outcomes?
Long Term:
• Create best practice in
the field for working
with this population
• Replicate program in
other counties in Ohio,
In the Midwest…in the
USA
• Publish your findings
Questions to think about?
• What is meant by improving strategies to improve
health?
• How will you know (measure) that it is the strategy that
has improved health outcomes?
• What are the health outcomes that you are looking at?
• Baseline measures – Will you collect baseline
information?
What do we mean by Outcomes?
Georgia
Department of
Human Resources
• Target Group:
• Refugees in need of services in Georgia
• Hope to Change:
• Community Awareness and understanding of
Domestic Violence
• Give community greater access to services
• What are you striving for – Short Term?
• Support services to 105 partners and their families
• Intervention for 80 refugee men
• Increase awareness
What do we mean by Outcomes?
Create best practice
for working with
Refugee men of this
community
Create best practice
for working with
domestic violent
offenders (or those
thought to be at risk)
Build a movement of
men involved in
preventing domestic
violence
Long
Term:
Create a network of
local, regional
and/or national
organizations
addressing this
issue
Questions to think about…
•What are the goals of the intervention?
•How is the program defining community
awareness?
•How is the program measuring community
awareness?
The 5 Questions….
What is
our
mission?
What are
our
results?
What is
our plan?
Who is our
client/customer/
patient?
What does
our client
value?
Final Thoughts…
•There are no correct number of outcomes
•Outcomes can go beyond the project’s formal
conclusion
•PLEASE be realistic about when to expect
outcomes
• Short, Intermediate, Longer term
Context (Contextual
Analysis)
Long Term Outcomes
Goals/Strategy of the Program
Actions/Program Activities
Short Term Outcomes
Assumptions
Good Sources on Evaluation
• Kellogg Foundation - www.wkkf.org
• “Level Best – How Small and Grassroots
Nonprofit can Tackle Evaluation and Talk
Results” By Marcia Felsten and Marianne
Philbin, 2007
• The Drucker Foundation Self-Assessment
Tool Process Guide and Workbook,
– www. pfdf.org
• Leticia Peguero, [email protected]
• Ayorkor Gaba, [email protected]