Transcript Monitoring of PRS Processes and Promises

Assessing Contribution and
Enhancing Social Learning:
An Introduction to
Outcome Mapping (OM)
16 December 2004
Kaia Ambrose
Presentation Outline

OM in Carchi, Ecuador.

Underlying logic of OM.

Brief outline of stages of OM.

Some key concepts - boundary partners, their changes
and how we measure that change.

OM and learning organizations.

The OM toolbook.

Conclusions and ideas for the afternoon session.
First, a little context…

Ecopar
• Ecuadorian NGO focused on research, training and
capacity building in tropical ecosystems/life zones.

The Ceja Andina Project
• IDRC-funded.
• Sustainable use of agriculture and forest biodiversity
in the ceja andina (cloud forest) region of the
northern Ecuadorian Andes.
• Research-oriented (traditional and participatory),
social learning processes, policy development and
strengthening of local government.
• Developing OM since March 2002 – “walking it
through and making it make sense in our context”.
M&E concerns in the Ceja Andina Project





As a team and as an NGO, how can we learn
from the project in order to improve the way we
“do” development (in a research context)? How
can we become a learning organization while
still being accountable?
How can we learn (iterative processes) together
with our local partners?
How can we bring creativity and real interest
into the M&E process? (create culture of M&E)
What is the balance between rigor and utility in
our M&E objectives?
Assess need for OM (complementary, bits of).
What is Outcome Mapping?
an integrated and participatory M&E approach;
also for planning.
 an approach that views outcomes as changes in
the behaviour, relationships, or actions of
partners. Development is accomplished through changes in the behaviour

of people.
a methodology that characterizes and assesses
the program or project’s contributions and
influence to the achievement of outcomes.


an approach for designing M&E in relation to the
broader development context but assessment is within
your sphere of influence.
Program / project’s Sphere of Influence
PROGRAM
(performance and strategies)
= Program’s partners (behavioural change)
Outcome Mapping does not:





Focus on impact (we’re not in the impact business,
we’re in the business of change).
Look for attribution (we look for contribution).
Isolate contributions of program / project in
achievement of results.
Promote linear, cause-effect thinking in a
“sterile” context.
Focus solely on actions promoted by program /
project.
Influencing Outcomes
Partners
Assessing Changes in
Behaviour
Program
Assessing Program
Influence
Assessing Internal
Performance
Outcome Mapping
OM can help us:

Besides designing or clarifying programme logic, helps
record and assess monitoring data:
• How far have our boundary partners progressed towards
achieving outcomes?
• What are we doing to support the achievement of outcomes?




Indicate cases of positive performance and areas of
improvement; front stage and back stage.
Evaluate intended and unexpected results; positive
deviance.
Gather data on the contribution that a program made
to bringing about changes in its partners (strategies new tools, aptitudes, knowledge).
Establish evaluation priorities and evaluation plan.
Boundary Partners – Core ideas



Boundary partners – individuals, groups,
organizations with whom the program interacts
directly and with whom the program anticipates
opportunities for influence.
Outcome challenges – description of the ideal
changes in the behaviour, relationships, activities
and/or actions of a BP that contribute to vision.
Progress markers – a set of graduated indicators
of changed behaviours for a BP that focus on
the depth and quality of change.
Boundary Partners




Social change is about relationships.
We’re all interconnected.
Social learning – create consensus, among
multiple cognitive beings, regarding direction of
program in terms of influencing outcomes, in
order to arrive at concerted action.
Boundary partners participate in process of OM
(in M&E).
Boundary Partners (have their own BPs)
Ceja Andina Project
Municipality
Town population
Programme
Rural Agriculture Association
Pesticide salesmen
Programme’s
Boundary Partners
Boundary Partners’
Boundary Partners
Ask yourselves …
« In which individuals, groups, or organisations is our
programme trying to encourage change so that they can
contribute to the vision? With whom will we work
directly? »
« Are we choosing X boundary partner because we want
to influence their behaviour and actions, or because
they will influence others? Or both?»
« What behavioural changes do we (collectively, between
project and boundary partners) want to see in BP that
will contribute to the vision (Outcome Challenges)?.»
Progress Markers



Describes progression of changed behaviours in
the boundary partner (is a set, as opposed to a
single indicator in order to demonstrate the
depth and complexity of change process).
Changes in actions, activities, & relationships
leading up to the ideal outcome challenge
statement.
Expect to see, like to see, love to see.
Progress Markers = Change Ladder
Love to see PMs
Truly transformative.
Set quite high.
Like to see PMs
More active learning,
engagement.
Expect to see PMs
Early response to program’s
basic activities.
Example Progress Markers
The project Expects to See local communities:
1. Participating in field day demonstrations.
2. Establishing a structure for cooperation in the partnership.
The project would Like to see local communities:
3. Articulating a vision for the ceja andina forest that is locally
relevant.
4. Requesting new opportunities for training.
The project would Love to see local communities:
5. Influencing national policy and debates on CBNRM.
Progress Marker Checklist
Each Progress Marker:
 Describes a changed behaviour by the boundary partner
 Can be monitored & observed
 Limit the number of PMs.
As a set, Progress Markers:
 Are graduated from easier to more difficult to achieve
changes in behaviour
 Describe the change process of a single boundary partner
Remember: progress markers are changeable!!
6 Types of Strategies
The program facilitates new tools, techniques, resources in order to contribute to
outcomes and development process.
Strategy
Aimed at
the
Boundary
Partner
Causal
Persuasive
Supportive
I-1
I-2
I-3
• Direct Output
• Arouse New
Skills/ Thinking
•Supporter who
guides change
over time
E-2
E-3
•Modify the
information
system
•Create /
Strengthen a
Peer Network
E-1
Aimed at the
•Alter physical or
Boundary
regulatory
Partner`s
environment
Environment
Creating your strategy map
I-1
I-2
I-3
What will be
done to produce
an “immediate”
output?
What will be
done to build
capacity?
How will
sustained
support,
guidance, or
mentoring be
provided?
E-1
E-2
What will be
done to change
the physical or
policy
environment?
How will you use
the media or
publications?
E-3
What networks/
relationships will
be established
or utilized?
Organizational Practices




Help us to define our role as an NGO according to
changing contexts and shifting paradigms.
Important to how the program is going to function
to effectively fulfill its mission.
Supporting change in its boundary partners
requires that the program be able to change and
adapt as well.
The things that we do as an organization to:
• foster creativity & innovation
• seek the best ways to assist partners
• maintain our niche
Eight Organizational Practices
1.
2.
3.
4.
5.
6.
7.
8.
Prospecting for new ideas, opportunities, &
resources.
Seeking feedback from key informants.
Obtaining the support of your next highest power
Assessing & (re)designing products, services,
systems, and procedures.
Checking up on those already served to add value.
Sharing your best wisdom with the world.
Experimenting to remain innovative.
Engaging in organizational reflection.
Journals
Set overall
intentions and
strategies; design
and articulate
program’s logic (how
it will contribute to
change within a
complex system).
Record internal and external data:
How we are progressing towards outcomes.
How we are contributing to change.
Indicate cases of positive performance /improveme
Assess intended and unexpected results.
Outcome and Performance
Monitoring
Boundary
Partners
Outcome Challenges and
Progress Markers
Outcome Journal
Program
Strategies and
Activities
Strategy Journal
Organizational Practices
Performance Journal
Conclusions



OM can help a program to be more
strategic about the actors it targets, the
changes it expects to see, and the means it
employs to support and facilitate those
changes.
OM is not based on a cause-effect
framework; it recognizes that multiple,
nonlinear events lead to change. It sees
interrelationships, not a snapshot.
Attribution becomes contribution.
Conclusions



Focus is on behavioural change – monitors
and evaluates whether a program has
contributed to changes in behaviours in a way
that would be logically consistent with
supporting development changes in the
future.
Program must also learn and change,
reconsider and adjust its goals, methods,
interventions.
Self-assessment, systematically collected.
Conclusions

Improving rather than proving.

Understanding rather than simply reporting.

Creativity and knowledge, rather than just
taking credit.
For more information, case studies,
materials, articles and presentations on
Outcome Mapping:
www.idrc.ca/evaluation
Afternoon session
(what I propose)


Fun dynamic! Identify your boundary partner!
Continued, in-depth presentation on:
• Boundary partners.
• Outcome challenges.
• Progress markers.




Practical exercise on progress markers.
Challenges / successes of the methodology.
Open discussion.
Postcard exercise (key messages).
Not everything that counts can be
counted. And not everything that can
be counted, counts.
Albert Einstein
OM challenges






Workload / paperwork– we made too much
work for ourselves and didn’t prioritize (too
many progress markers, tried to monitor
everything).
Trial-and-error reporting system.
Scaling-up within NGO.
New boundary partners.
Different boundary partners have different
experiences with the projects (example:
municipalities).
Project team had no formal training on the
methodology.
OM challenges



Weak monitoring plan (no one to lead OM from
the beginning).
How to combine with more quantitative data
collection.
Encouraging on-going boundary partner
participation in OM M&E activities.
OM successes






Build cohesion with team and with partners.
Identify and develop interconnectedness among
boundary partners through dialogue, consensus and
collaborative action; social learning space.
How to connect with people becomes part of the
methodology.
Highly participatory.
Donor is flexible – encourages learning, creativity,
adjustments. Makes space for dialogue and interaction
(and really listens!)
Self-assessment “all around” (of team, of boundary
partners).
OM successes





Innovation of tools and application of tools.
Build culture of M&E within NGO / team and
boundary partners.
Concrete data for developing evaluation plan.
Meetings for discussion / self-assessment =
learning organization (analyze our learning).
Permanent system of reflection.
Interest to learn more about OM by other
NGOs in Ecuador (request for information,
presentations, etc.) = good promotion of
methodology (and project).