Transcript Workshop Outcomes

Impact Assessment Training
10/11th September 2013
Oslo
Workshop Outcomes
By the end of these two days you will have:
• Developed a common understanding of what is
meant by impact assessment; and how it differs from
and complements the processes of monitoring and
evaluation.
• Identified key challenges that you face in conducting
impact assessments
• Worked with pre-selected case studies to design an
impact assessment process
• Considered how the results of impact assessments
might be used in learning, reporting and being more
accountable to both donors and stakeholders
• Identified ways in which you can take forward
learning from this workshop.
The vicious circle
Impact
So what do we mean when we talk about
Impact?
Impact
Think about an event or a person who
made a significant impact on your life (a
birth or death of a family member or
friend, a relationship, a marriage, an
accident)
– What does impact mean in relation to
this?
– What concepts or ideas does it
include?
How do you “assess” this
impact?
• ... You try to understand the nature of the
change that has taken place in you and to
determine its significance in your life.
Impact Assessment
“The systematic analysis of significant
and/or lasting change – positive or
negative, intended or not – in the lives of
target groups, brought about by a given
action or a series of actions”
Its not that easy... consider Danny..
Why assess impact?
• To understand the implications of our work
• To become more accountable to those we
work with (stakeholders)
• To support institutional learning and decision
making and improve future work
• To contribute to policy development and
effective advocacy
• To help demonstrate organisational
performance
Key areas of enquiry for impact
Five essential questions:
•
•
•
•
•
What has changed?
For whom?
How significant was it?
Will it last?
In what ways did we contribute to
these changes?
The challenge for this workshop
To be able to develop Impact Assessment
processes which are:
• Simple and user-friendly
• Build on existing structures and systems
• Are useful for accountability both
upwards and downwards
• Are useful for your organisational
learning
Session 2
Relationship between M&E and
Impact Assessment
M&E or Impact Assessment?
• How is the assessment of impact different
from the processes of monitoring and
evaluation?
• What to do?
• When to do it?
Example
In a project to build social housing for a local
community, for example:
• Monitoring would relate to the purchase of
materials, and building the houses according
to plans that have been drawn up.
• Evaluation would assess the results of these
efforts: how good was the plan? How well
were the houses built? Was the project cost
effective etc
But...
It’s possible to have well-built cost effective housing
schemes which are no use to those for whom they
were intended (e.g. aborigine “settlements”).
Thousands of “successful projects and programmes”
which fail to make a positive impact on the lives of
people they aim to serve. Some projects result in
negative impacts.
Impact Assessment addresses the “So what?”
question: as a result of our efforts, what’s changed
for whom; and how significant is this for them?
Impact – so what?!
Thousands of “successful” projects and
programmes make no lasting difference to
people’s lives.
Impact Assessment addresses the
“So What?” Question
As a result of our efforts, what has changed for
whom; and how significant is this for them?
Differences in Brief
Log Frame Approach
e.g. Skills training programme - a results chain...
Does it always work like this???
Inputs
Funds,
expertise etc
Activities
Training
Outputs
Improved
skills
Outcomes
Improved
livelihoods
Impact
Less poverty
Small groups’ task
Small UK based NGO, working through partners
in Ethiopia in Education
Goal: All children have access to free basic
education
1. What high level outcomes would ensure that
this goal is achieved?
Small groups’ task
• Select one outcome
• Discuss what activities/outputs would lead
to the achievement of this outcome
• Develop this into a mini project
• Develop indicators at each level
• Use different colour post its!
Small groups’ task
Level
Detail
Impact
All children have access to
free basic education
Outcome 1
Outputs
•....
•....
•.....
Indicators
Analysing the results
To what extent are you able to assess the
key questions for impact assessment?
– What has changed?
– For whom?
– How significant/lasting are these changes for
different target groups?
– What -if anything - did our programme
(project) contribute?
Measuring impact based on
results chain
Tends to be limited to assessing what you
expect/hope will change...
Good for demonstrating results to donors..
Positive impact Negative
impact
Expected
xxxx
?
Unexpected
?
?
So...
• So how else to assess impact?
• What are the challenges?
• What different approaches could we use?
Answers this afternoon!
Session 3
Focus on change and how to
assess it
Thinking about Change
• How does change happen?
• How do/can development organisations
conceptualise this?
• What is their realistic sphere of influence on
changes that do take place?
Change is..
Complex in that many different changes can take
place simultaneously in people’s lives
• Continuous in that nothing in society or the
environment is ever static
• Variable in pace, scale and/or over the course
of any intervention
• Not necessarily lasting or sustainable.
What do we need to know to assess
impact?
•Who or what was involved in the change? (e.g.
individual actors or state institutions)
•What strategies were used to bring about the change?
(e.g. reform, mass mobilization)
•What were the contexts that affected how the change
happened? (e.g. urbanization, power relationships)
•What was the process or pathway of change? (e.g.
demonstration effects, cumulative progress)
•How were our efforts connected to this?
Organisational response to this?
• Many organisations are developing Theories
of Change to better support their
understanding of how change happens and
their role in the process
Four Key Elements
Understand how change
happens and your role
in this
Critically reflect on your
pathway and how you
thought change
happened
Develop
Organisation/Programme
change pathway
Assess the impact of
your efforts
How do they complement other
planning and M&E processes?
• Strategic plans?
• Log frames?
• M&E systems?
• Learning loops?
3. ToC doesn’t replace results frameworks they make them stronger
How change
happens
Organisation
or Programme
change
pathway
Reflection and
adaptation of
ToC
Impact
assessment
Summary of some of the differences
Logic Models
Theories of Change
•Situate
programme efforts
•Describe component parts in bigger picture of change
of a projects
•Surface and articulate
• Make linear links between assumptions
activities and results
•Explain causal pathways
•Are
used as a
management tool
of Change
Are used as a critical
analysis process
34
Examples of Impact Assessment
Frameworks:
MRDF
Contributes Contributes to:
To change in: Power holders at
Capacity
Building to
partners
Funds for
Capacity
Building
Increased
partner
capacity
Awareness
of rights
Funds for
partners
Access to
information
Activities
proposed
by
partners
Access to
support
Access to credit
local and regional
level ensure that all
women access rights,
opportunities and
services
Marginalised women
have the confidence,
knowledge and skills to
access rights,
opportunities and
services
Marginalised women
are organised, active
and influential and play
an active role in society
Access to resources
Access to training
Marginalised women
are supported and
empowered by their
families, society and
culture
Theory of Change – Women's Empowerment
(MRDF as a stone making ripples in the water)
Contributes to:
Marginalised
women are
empowered and
access rights,
opportunities and
services
including:
• Equal access to
property and
resources
•Reduction in
harmful traditional
practises
•Increased
representation in
local, regional and
national leadership
positions
•Increased economic
empowerment
•Reduction in
violence and
exploitation against
women?
Session 4
Approaches and strategies
• Three approaches:
– Post programme: Testing logic of log frame ( impact “evaluation”)
– Participatory ToC approach which is used to design monitor and
assess efforts
– Research: looking back sometime later and assessing changes and
their relation to programme efforts
• Four strategies:
–
–
–
–
Build into existing M&E
Tracer and tracker studies
Ensure key moments of critical reflection
Commission a retrospective study
Plans for tomorrow
Case Study Task
Day 2
Session 1
Case Study Task 1
Reflection on yesterday
Main steps in designing an
Impact Assessment
1. Define purpose, approach and scope of the
assessment
2. Develop/confirm theory of change and/or
dimensions of change
3. Develop a list of areas of enquiry which will help
you to explore impact
4. Select tools and methods for gathering and
analysing relevant information
5. Decide on reporting strategies
6. Make concrete plans and timelines (who, when,
how....)
The challenges
To develop a process which is:
• Simple and user-friendly
• Build on existing structures and systems
• Are useful for accountability both upwards
and downwards
• Are useful for your organisational learning
And the challenges that we identified yesteday
ACCOUNTABILITY TO THE POOR
Case Study Task 1
In small groups:
• Share the details of the programme you are
working on (you need to be clear about
context, overall goal and top level outcomes)
Case Study Task 1
• What will be the main purpose of doing this
impact assessment? Organisational learning?
To meet donor demands? Accountability to
stakeholders? For advocacy
• Which approach (or combination of
approaches?) to Impact Assessment would be
most appropriate in this Why?
• What will be the scope and scale of this
assessment?
Session 2
Case Study Task 2
Case Study Task 2:
• What is your realistic “scope of influence”? Which
what areas of impact will you realistically be able
to “assess”, and which areas of change will you be
able to “illustrate contributions to change”?
• Based on this, what “Dimensions of Change” will
you be looking to assess?
• Develop a menu of areas of enquiry which will
enable you to set baselines and track progress in
relation to impact
•
Being clear about your scope of
influence
Sphere of direct
influence
(you are working directly
with target groups on
specific changes)
Need to assess impact
rigorously
Sphere of indirect influence
(there are other factors/actors
which influence changes
you want to see)
Assessment process less
rigorous –
illustrations of impact
Dimension
Task 1
Areas of Enquiry
Task 2
Direct scope of influence
e.g.
Capacity of partners
•Ability to plan and
deliver
•Levels of technical
capacity
•Approaches to
networking with others
• Level of sustainability
•Shifts in confidence
Indirect scope of influence
Tools/ Sources of
Information
Task 3
Areas of Enquiry and indicators
Areas of enquiry
indicators
Written neutrally to encourage
expected/ unexpected
Designed to test logic of project and
expected changes
Can track levels of xx, shifts in xxx,
trends, perceptions,
Requires open-ended questions
Session 3
Case Study Task 3
Case Study Task 3
• Propose a range of appropriate methods that
you could use to gather relevant data for both
monitoring and assessing impact (including
building on or adapting existing tools and
mechanisms).
Making the process manageable
and robust
•
•
•
•
Baselines
Sample size
Triangulation
Selecting the right tools and ways
collecting useful information
• Ensuring that you have asked the right
questions in the right way
Baselines
• The problems?
• The solutions?
– Dedicated desk research and ask others
– Plan from the start and use rolling baselines
– Reconstructing them
Sample size
A good sample is one that is sufficiently large
and is unbiased.
How to select?
– Mathematical?
– Random?
– Pragmatic?
Triangulation
• You need three perspectives to ensure validity
of information gathered. Could include:
– Three ways of checking same info
– Asking the same question of three target groups
A word about tools and
collecting information
Less is more..
• Be aware of time resources and capacity
• Stick to the “old faithfuls” as the basis
• Mix of methods is good
• Be led by areas of enquiry
• And perhaps the hardest thing of all.. Try
to enable people to tell the real story...
Checklist – use for areas of enquiry and
selection of tools
In order to assess impact effectively, your areas of enquiry
combined with data gathering methods needs to capture this
information
1. Have there been any changes?
2. How many people were affected (which target groups)?
3. How were they affected (and were they affected
differently)?
4. Were these changes intended?
5. How do they compare to baselines (have you got
evidence)?
6. What can be attributed to your organisational efforts?
7. How confident are you in reporting these findings?
Session 4
Wrap up and the way forward