Transcript Outcome Mapping

Outcome Mapping
Bringing learning into
development programmes
15-18 September 2009
Capetown South Africa
Robert Chipimbi
Simon Hearn
Acknowledgements

This presentation makes use of various
materials that were shared by various
OM community members on the OM
learning community website. Without
being exhaustive special thanks goes to
Terry Smutylo, Enrique Mendizabal, Steff
Deprez, Jan Van Ongevalle, Daniel
Roduner, Kaia Ambrose and many
others.
´´The only real voyage of discovery exists,
not in seeing new landscapes, but in
having new eyes´´
Marcel Proust
Workshop objectives
1.
2.
3.
Understand basic OM concepts &
principles
Decide if & how OM applies to your work
Apply some OM tools
Before we start ...
Outcome Mapping is different from conventional PM&E models




It requires time to absorb the concepts and implications for
your work
It might seem very complex at first (new concepts, new
language, …)
Requires a change in thinking (new paradigm)
Generates different reactions
 Interest
 Confusion / doubt
 Disagreement
 Defense
 Let’s go for it!
 How can we ever do this?
 …
DON’T WORRY - ASK QUESTIONS!
origins of outcome mapping
Developed between 1998 and 2001 by the International
Development Research Centre (IDRC) as an answer to the
difficulties faced in evaluation processes of development
programs:

Demonstrating results

Proving cause/effect relationships

Measuring & attributing impact

What about unintended results?

…
Some core principles of outcome
mapping
1. Embracing complexity
2. Sphere of influence
3. Boundary partners
4. Outcomes as behavioural changes
1. Embracing Complexity
Looking at the bigger
picture
See yourself as a part of
an interconnected web
of relationships and
systems
Constant change
“It’s not possible
to see the same
river twice”
Recognizing that change is…
 Continuous
 Complex
 Non-linear
 Not controllable
 Multidirectional
Keep your eyes wide open…
Being attentive
along the journey
is as important as
the destination
opportunities for P,M,E and
learning
beginning
during
end
results,
impact
objectives,
inputs,
Activities
outputs
Life cycle of the program
Often Dealing with social systems
Non-linear process
Complex relationships & interactions
Unpredictable nature of human behaviour
Unknown factors & actors
different approaches
Logical
Frame
Analysis
Results
based Mgt
changes in behaviour
Life cycle of the program
Objectives
(activities, products)
Impact
(changes in
environmental
or social conditions)
2. Sphere of influence
Sphere of interest
Sphere of influence
Sphere of control
Inputs
Activities
Outputs
Outcomes
Impact
Relative Influence Along the Results Chain
Sphere of control
Inputs
Activities
Sphere of influence
Outputs
Sphere of interest
Outcomes
Impact
Behavioural Changes
3. Focus on boundary partners

Key concept is
« boundary partners »

The individuals, groups,
and organizations you
work with directly and
anticipate opportunities
for influence
An actor-centered approach to development
planning and M&E

“Development is essentially about people (groups and
organisations) relating to each other and their
environment.”

Although a programme can influence the achievement of
outcomes, it cannot control them because ultimate
responsibility rests with the people affected.

“Outcome Mapping centred around programme actors
and their relationship in development programmes.”
A programme can not control change, it can only
influence and contribute to changes at the level of its
boundary partners
Beneficiary 2
Beneficiary 1
Sphere of
interest
Beneficiary 3
Boundary
partner 1
Boundary
partner 2
Sphere of
influence
Boundary
partner 3
Implementing
team
Sphere of
Control
4. Outcomes as behavioural changes
Outcome mapping focuses on outcomes as
changes in behaviour of those with whom
the program or project is working directly
(boundary partners)
Why behavioural change?

For each change in state (e.g. security of land for
marginalized groups) there are always correlating
changes in behaviour of certain people and groups.

Assessing changes in state – as often in logframes – not
necessarily provide the kind of information that
programmes need to improve their performance and
relevance.

Development is done by and for people.
Principles of use

Flexibility



Participatory



OM needs to be adapted to use in your specific context.
Not a fixed route but a guide for the journey we take.
OM implies dialogue and collaboration with partners.
We co-create the ´map´ with our partners.
Evaluative thinking

Fosters a reflective practice, organisational & social
learning.
principles of use
Flexible: Can be adapted
to use & context
(Not a fixed
route but a guide for the journey we take)
Complementary: combine
with other methods
principles of use
Participatory: seeks dialogue and
collaboration with partners (We cocreate the ´map´ with our partners)
Evaluative: promotes culture of
reflection, results oriented thinking,
and social & organizational learning
where is the map?
✓ OM is a guide to the journey
we take with our partners. We
co-create the map.
✓ It focuses on the intention and
what happens along the way
✓ The map is not the territory, it
shows the route taken
The outcome mapping
framework
stage 1
Intentional Design
Step 1: Vision
I have a dream!
Martin Luther King, Jr.
August 28, 1963
The vision

Description of the universe in terms of
human, social, economic or environmental
conditions

Large-scale development-related changes

Describes the ideal world

Beyond program's capability

Functions as a lighthouse – something to
guide movement

…and a star – something big to aim for

Written in present tense
Example vision statement

Education in Zimbabwe is reoriented towards
critical thinking, action competence and
responsible behaviour by individuals and groups
to achieve sustainable living in a healthy
environment. In this way, the Zimbabwean
community is empowered to make informed
individual and collaborative decisions, which
ensures continual effective environmental
management.
vision in graphic form, Nagaland (India)
vision
facilitation questions
Imagine that, 5-10 years from now, the program has
been extremely successful. Things have improved
beyond your most ambitious dreams.
• What changes have occurred?
• What (& how) are your intended beneficiaries doing?
• What are your partners doing?
• Describe the better world you are seeking.
stage 1
Intentional Design
Step 2: Mission
The Mission
The mission is that “bite” of the
vision statement on which the
project is going to focus.
Mission statement
Describes how the program will contribute to
the Vision, including its main working areas
and the partners with whom it will implement
them.
Written in future tense as something the program will do
Example Mission Statement
In support of this vision, St2eep will create an enabling
environment for sustained EE implementation in Secondary
Teachers’ Colleges through the encouragement of active
learning processes which promote participation, critical
thinking, informed decision-making, action competence and
responsible citizenry. St2eep will conduct on-going pre- and
in-service capacity enhancement, curriculum review and
implementation and strive for the reorientation of assessment
of teaching and learning in line with principles of
Environmental Education. St2eep will lobby for motivational
strategies and policies to ensure that EE is institutionalized in
the education system and encourage outreach programs
through networking with schools, communities and
environmental interest groups. St2eep will ensure that
graduates of the Secondary Teachers’ Colleges are able to
implement EE in their teaching.
Mission
facilitation questions
• What areas do you need to work in?
• What do you need to do in these areas?
• Who can you work with?
• How will you stay effective, efficient, and
relevant?
Summary
Vision
✓ about the future
✓ observable
✓ idealistic
✓ not about the program
Mission
✓ feasible
✓ identifies activities
and relationships
✓ about the program
Exercise: Vision and Mission
stage 1
Intentional Design
step 3: Boundary Partners
boundary partners
Those individuals, groups, and
organizations with whom the program
✓ interacts directly to effect change
✓ anticipates opportunities for
influence
✓ engages in mutual learning
= boundary partners
Program
a programme…..
• has direct control over its inputs, activities,... in working
with the boundary partners, but
• can not control change at level of its boundary partners
and beneficiairies. The ultimate responsibility rest with
the people affected.
• hopes to have direct influence / contribute to changes
at the level of its boundary partners
• can only indirectly influence change at the level of
ultimate beneficiaries (impact)
boundary partners have boundary partners
program
program’s bp
bp’s bp
strategic partners
selected on the basis of their contribution
to the mission
a person or group with whom the program
works directly to achieve the mission,
without necessarily wanting to change the
partner’s behavior
Examples
Other donors
Other organisations doing
similar work
Media
St2eep project 2005-2008
Vision
Mission
Beneficiaries: Colleges, Lecturers & staff, Student Teachers, Pilot schools
Communities, ….
BP1
BP2
BP3
BP4
Ministry of
Education, S., C.
Ministry of Higher &
Tertiary Education
College
administrations
EE Steering teams
( EE Coordinator)
Dept of Teacher
Education (UZ)
Outcome
Challenge 5
Outcome
Challenge 1
Outcome
Challenge 2
Outcome
Challenge 3
Outcome
Challenge 4
BP5
St2eep impl team
EE coordinators and
VVOB facilitators
Strategic partners
Min. Of Education
Min. Of Environment
SADC REEP
ZWEECF
Universities
NGO’s
...
Funding
Organisations
VVOB
SADC REEP
...
boundary partners
facilitation questions
✓ In which individuals, groups, or organizations is
your program trying to encourage change as a
contribution to the vision?
✓ With whom will you work directly?
✓ Are you choosing boundary partners because
you want to influence the ways they help or
influence others?
Boundary partner exercise

Worksheet to be supplied
stage 1
Intentional Design
Step 4: Outcome Challenges
what indicators do you use for:
Greater awareness…
Empowered women…
Reduced conflict…
Increased collaboration…
Gender sensitivity…
Equal access…
Active participation…
Poverty alleviation…
Strengthened capacity…
?
In OM you describe what the BP is DOING
DIFFERENTLY and those are your ‘results’.
outcome challenge
✓
describes the ideal behavioural changes,
relationships, actions (professional
practices) of a boundary partner for it to
contribute to the vision
outcome challenge
characteristics
✓ one for each boundary partner
✓ describes the boundary partners’
ideal (but possible) contribution to
the vision
✓ written like this: “The program
intends to see [boundary partner]
who [description of behaviors in
the active present tense]”
outcome challenge
facilitation questions
✓ Ideally, in order to contribute to the vision,
how would the boundary partner be
behaving?
✓ With whom would they be interacting?
✓ What would they be doing to contribute
maximally to the vision?
Example outcome challenge statement
St2eep intends to see college administrators who
continually support EE activities within the colleges by
appointing full–time EE coordinators and providing
office space, transport, equipment, finances,
communication and other resources for successful EE
implementation. These college administrators are
cooperating with St2eep by incorporating EE in the
college strategic plan and encouraging lecturers for EE
orientation, workshops and programs. They are
attending EE organised functions. They are actively
supporting policy development and implementation
through the recognition that EE is part and parcel of the
curriculum and college functions.
Outcome challenge exercise

Worksheet to be supplied
Program Framework
Vision:
Mission:
Boundary
Partner:
Outcome Challenge:
Boundary
Partner:
Outcome Challenge:
Boundary
Partner:
Boundary
Partner:
Outcome Challenge:
Outcome Challenge:
stage 1
Intentional Design
Step 5: Progress Markers
progress markers
✓
Describe changes in actions, activities and
relationships leading to the ideal outcome
✓
Articulate the complexity of the change
process
✓
Can be monitored & observed
✓
Enable on-going assessment of partner’s
progress (including unintended results)
progress markers (ladder of change)
Love to see
Expanding influence,
helping others, sharing
expertise
Like to see
Actively engaged,
learning, commitment
Expect to see
Early encouraging response to
program, initial engagement
progress markers are graduated
✓move from initial to more profound
changes in behaviour
✓show transformation in a single
boundary partner
✓more informative than a
single indicator
Example Progress Markers
The Project Expects to See College Administrators:


Authorising lecturers to participate in EE activities.
Providing office space and equipment for the EE coordinators.
The program would Like to See College Administrators :


Providing transport, finances and other resources for EE activities.
Including and positioning EE, high on the agenda of staff meetings.
The program would Love to See College Administrators :


Incorporating EE in the college strategic plan.
Appointing full time EE coordinators.
some advantages of
progress markers

Do not require consultant to write
quantifiable indicators



Show directionality of change

Permit on-going assessment of partner’s
progress (including unintended results)

Provide basis for dialogue with partners
Support learning function
Can be monitored & observed from the
start of the program
How many progress markers?
It depends.
Suggestion: total of 15, with most occurring in
the “like to see” range.
Expect
4
Like
8
Life of Program
Love
3
Exercise Progress Markers
Identify 3 strips that are not progress markers
 Divide remaining strips between




Expect to see
Like to see
Love to see
Boundary
Partners
Programme
Team
...
Outcome
challenge and
progress markers
Interventions/strategies??
stage 1
Intentional Design
Step 6: Strategy Maps
Strategy Maps
-
Which intervention strategies are we developing in
order to support/influence the boundary partners
towards the desired changes?
-
Outcome Mapping provide a framework to develop a
mixed set of strategies to assure greater potential for
success:
1.
2.
-
Strategies directly aimed at individuals, teams, organisation of
the boundary partner
Strategies aimed at the environment in which the boundary
partner is working
Programme interventions most likely change during the
course of an initiative
causal
I
aimed at
individual
boundary
partner
E
aimed at
boundary
partner’s
environment
persuasive
supportive
6 types of support strategies
causal
I
aimed at
individual
boundary
partner
E
aimed at
boundary
partner’s
environment
direct
Influence
(e.g. funding, prepare
a report, …)
alter the
physical or
regulatory
environment
(e.g. technical
transfer, policy
change, internet
access, terms of
reference, rules,
guidelines)
persuasive
arouse new
thinking;
build skills,
capacity
supportive
on-going
support
supporter/mentor who
guides change over
time,involvement is more
frequent and sustained,
nurturing for self(e.g. skill
sufficiency
enhancement,
(e.g. program member
methodological
who provides regular
workshops, training)
guidance and input,
coaching, expert, …)
broad
create /
information
strengthen
dissemination/ peer networks
access to new partners working
together and
info
(e.g. Radio, TV,
website, publications,
conferences,
workshops)
collectively
supporting each
other)
(e.g. research
network, …)
strategies: facilitation questions
causal
I
aimed at
individual
boundary
partner
E
aimed at
boundary
partner’s
environment
persuasive supportive
I-1
I-2
How to
produce an
“immediate
effect in the
BP”?
How to build
capacity, to
influence new
ways of
thinking or
doing things?
E-1
I-3
How will
sustained
support,
guidance or
mentoring be
provided?
E-2
E-3
How to influence
What networks
the quality,
How to alter
or relationships
availability,
the physical or
will be
accessibility of
policy
established,
information?
environment? How to influence strengthen or
the media?
utilized?
Example strategy map
causal
I
aimed at
•Fund research
projects
individual
boundary
partner
E
aimed at
boundary
partner’s
environment
•Provide computers
and internet access
•Include work with
women and youth as
a condition for the
grant
persuasive supportive
•Run workshops on
quantitative and
qualitative methods
•Offer internet
reesarch courses
•Coordinate training
on participatory
methods
•Offer gender
sensitivity training to
those working with
HIV-infected women
•Hire a professional
writer to work on
dissemination
strategies with
researchers
•Hire a fundraiser to
help identify donors
and develop a
fundraising strategy
•Provide ongoing
technical assistance
•Organise regional
conferences for
HIV/Aids research
community
•Develop internet site
with tools and
methods
•Publish ‘special
paper’ series
•Establish fromal
mentorship program
that partners
experienced and
junior researchers.
•Facilitate the
development of an
electronic HIV/AIDS
policy research
network
value of strategy maps
✓ Clarify what the project is doing, with whom
and why
✓ Articulate the range, mix and logic of the
strategies
✓ Detect strategy gaps and over extension of
resources
✓ Encourage multiple interventions to increase
effectiveness
Boundary
Partners
Outcome
challenge and
progress markers
Programme
Team
Strategies
&
activities
Organisational practices?
stage 1
Intentional Design
Step 7: Organisational Practices
Organisational practices

Supporting change in boundary partners
requires that the program/organisation
be able to change and adapt as well

Move beyond being efficient & effective
(operational capacities) towards being
relevant (adaptive capacities)

Therefore OM incorporates an ongoing
assessment and reflection of the own
practices & capacities into the design of
the programme
what you do as an organization to
✓ keep learning
✓ foster creativity & innovation
✓ seek better ways to assist your partners
✓ maintain your niche
✓ maintain high level support
✓ build relationships
organizational practices
1.
Prospecting for new ideas, opportunities,
and resources
2.
Seeking feedback from key informants
3.
Obtaining the support of your next highest
power
4.
Assessing and (re)designing products,
services, systems, and procedures
…organizational practices
5. Checking up on those already served to add
value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
Exercise – organisational practices

Worksheet to be supplied
vision
mission
Banks
Women’s Self Help
Groups
BAIF
State
NGOs
Girls & Women
Police
Families
Community Leaders
Public Health
Clinics
Strategies
85
Strategic
Partners
Boundary
Partners
Project’s
Outcome
s
BP’s outcomes
Overview of Stage 1
Interest
Influence
Organizational Practices
Control
project team
stage 2
Outcome and Performance Monitoring
4. Monitoring & Evaluation in OM
Monitoring and evaluation
« Not everything that counts can be
counted.
and
not everything that can be counted,
counts. »
Albert Einstein
An outcome mapping
based M&E system looks
both at the processes that
are going on within the
programme and how the
programme (through its
processes) is contributing
to certain outcomes (no
causal relationship)
M&E – what’s the difference?
Monitoring
 Ongoing, continuous, more regular
 Internal activity
 Responsibility of project staff and management
 Continuous feedback to improve programme & report on
performance
Evaluation
 Periodic and time bound
 Can be internal, often external
 Responsibility of evaluator with staff and management
 Periodic feedback
 A more in depth study and assessment of an issue
Planning and monitoring opportunities in OM
contextual
information
Boundary
Partner
Program
Change in
behaviour
Internal
performance
Outcome
challenge and
progress markers
(outcome journal)
Strategies
& activities
(Strategy
Journal)
Organisational
practices
(Performance
Journal)
Accountability & Learning: A Balancing Act
Changing the M&E perspective
Old M&E:
 M&E is mainly for external
funding body
 Concentrates on activities
and financial reporting
 Lots of data and little
analysis (big reports!)
 Little learning takes place
 Little stakeholder
participation
 M&E is boring and not very
useful for project staff and
beneficiaries
New M&E
 M&E is for everybody
involved in project
 Also look at effects and
outcomes and why there is
success and failure
 Learning is a key function
of M&E
 Good analysis on how to
improve project
 Lots of stakeholder
participation
 M&E is active, interesting
and useful
-> Evaluative thinking throughout the working cycle
Accountability & Learning: A Balancing Act
Exercise Monitoring tools

Monitoring Journals
Monitoring Plan
Monitoring
Priority
Strategies
Org’l Practices
Boundary
Partner
Outcomes
Who
will
use
Purpose
When is
it needed
Who
will
collect
& how
How
often
Proposed Tool
Strategy Journal
Performance
Journal
Outcome Journal
Three additional questions:
1. How will we learn from the monitoring information, how do we make sure
lessons learned feed into future planning?
2. How is the reporting going to be done?
3. Who gets the report?
Exercise – Monitoring Planning

See worksheet
Useful M&E Depends On
Learning & Reporting
(Info to users/events/audiences)
M&E Implementation
(Data collection & interpretation)
Clarifying Intent
(Both interventions & outcomes)
M&E Planning
(Choosing what to track)
reviewing the intentional design
1. Read the vision
statement
Does this still reflect the program's dream?
2. Read the mission Is this the greatest contribution our program can make?
statement
Have we been doing this? Why? Why not? Should we add
anything or take anything away?
3. Review boundary Is this who we are working with directly? Do we need to add
partners
or drop any boundary partners?
4. Review outcomes Do these accurately reflect transformations in our boundary
partners as they strengthen their contributions to the
vision?
5. Review progress
markers
Was the change process we set out accurate and useful?
What now needs to be added or taken out?
6. Review strategies What did we plan to do? Have we implemented these
activities? Do we need to add, remove any?
7. Review
organizational
practices
Are we doing everything we can to maintain & enhance our
capacity to support our partners?
104
stage 3
Evaluation Planning
« Evaluative thinking is a learned
behaviour that you have to practice
and improve at on an on-going
basis! »
Michael Quinn Patton
Step 12: Evaluation Plan

A short description of the main planning
elements of an evaluation study

For Whom? What? Where? How? By
Whom? When? How Much?
Elements of an Evaluation Plan
Evaluation Issue:
Date Cost
Info.
Evaluation Who Will
Who Will Use Questions
Sources Methods Conduct & (Start &
to be
Evaluation,
Manage the Finish)
How, & When? Answered
Evaluation?
Setting Evaluation Priorities
Criteria to select evaluation issue include:
Learning needs
 Accountability or Reporting requirements
 Partners’ information needs

Guiding Principles for Evaluation

Balancing learning & accountability

Action oriented

Each study is unique

Process not only product

Build capacity at all levels through
participation
Points to Remember:
1.
Stay in touch with reality.
2.
Celebrate the contributions of your
team and your partners.
3.
Be “idealistic realists.”
OM’s underlying message:
Social change has multiple, interacting causes.
Trying to determine the “impact” achieved by a
single intervention may be a waste of time and
resources.
Seek, instead, to deepen understanding of how
the actors and factors influence each other in
relation to the program’s vision.
ongoing OM applications
http://www.outcomemapping.ca
Ghana
Korea
Switzerland
Kenya
Ecuador
United Kingdom
Madagascar
Mexico
Brussels
Mali
Egypt
Netherlands
Namibia
Bhutan
Australia
Uganda
Honduras
Sri Lanka
Zimbabwe
Guatemala
India
examples of use:
http://web.idrc.ca/en/ev-27705-201-1-DO_TOPIC.html
for more information
and to share your experiences
www.outcomemapping.ca
www.idrc.ca/evaluation
www.mapeodealcances.net
Thank You Very Much