Transcript Introduction to Outcome Mapping slides

outcomemapping.ca
Introduction to
Outcome Mapping
Simon Hearn, Julius Nyangaga, Ziad Moussa
Brussels, Belgium
March 25-26, 2013
outcomemapping.ca
Workshop Objectives
Clarify what Outcome Mapping is and does
Introduce and apply Outcome Mapping
concepts and tools
Consider if and how Outcome Mapping could
be useful in our work
Agenda Overview
This morning:
• Key concepts
• Vision
• Mission
• Boundary
Partners
This afternoon:
• Outcome
Challenge
• Progress Markers
• Strategy Map
Tomorrow:
• Organisational
Practices
• M&E planning
and journals
Proposed process guidelines:
 Everyone can participate
 One speaks, all listen
 Value each other’s ideas
 No smoking or mobile devises
 Respect each other by being on time
 Questions any time (answers may come later)
 Enjoy... Network... Laugh... Sing..!
...Other?
If your M&E processes were
practical and useful, what
would they do, what
characteristics would they
have?
Key concepts in
Outcome Mapping
Acknowledgements
This presentation makes use of various materials
that were shared by members of the global OM
community. Without being exhaustive, special
thanks goes to Terry Smutylo, Steff Deprez, Jan
Van Ongevalle, Robert Chipimbi, Daniel Roduner,
Kaia Ambrose, Ricardo Wilson-Grau and many
others.
Source: A guide for project M&E: IFAD
Conventional thinking…
Time
Source: Ricardo Wilson-Grau (inspired by Jeff Conklin)
… clashes with relationships of cause and
effect that are unknown
Time
Source: Ricardo Wilson-Grau (inspired by Jeff Conklin)
Social change can be…
• Complex:
• Unstable:
involve a confluence of actors and factors
independent of project duration
• Non-linear:
• Two-way:
unexpected, emergent, discontinuous
intervention may change
• Beyond control:
but subject to influence
• Incremental, cumulative:
watersheds & tipping points
Source: Terry Smutylo
Time
Source: Ricardo Wilson-Grau
Brief definition of OM
• A participatory method for
planning, monitoring and
evaluation
• Focused on changes in
behaviour of those with
whom the project or
program works
• Oriented towards social &
organizational learning
OM’s answer
Start from observable
behaviour change
Embrace different
perspectives
Recognise that all
interventions have
limited influence
Support people to build
their own well-being
Enable interventions to
adapt as they engage
Apply a systems
understanding
Source: Terry Smutylo / OM Lab 2012
A brief history of OM
• 1990s: post-Rio need to demonstrate ‘sustainable’
•
•
•
•
•
•
•
results
1998: Barry Kibel and Outcome Engineering
1999: Methodological collaboration with projects
2000: Publication of manual in English
2002: Training, facilitation & usage globally
2006: OM Learning Community
2008: CLAMA
2010: East Africa and beyond
“Being attentive along the journey is as
important as, and critical to, ariving at the
destination ”
Michael Quinn Patton
Three key concepts in OM:
1. Sphere of influence
2. Boundary Partners
3. Outcomes understood as
changes in behaviour
There is a limit to our influence
Project
Sphere of
control
People the
project works
with/through
Sphere of
influence
People who
benefit from
the project
Sphere of concern
There is a limit to our influence
Inputs,
activities,
outputs
Sphere of
control
Outcomes:
Changes in
behavior
Sphere of
influence
Impact:
Changes in
state
Sphere of concern
Farmers
participate
in field trials
Participating
farmers learn how to
use drip irrigation
equipment
Participatory
research on
demonstration
farms
to develop
approaches
to drip irrigation
Extension workers
visit demonstration
farms
Increased
knowledge of
techniques
Farmers adopting
drip irrigation
methods
Training of
extension
workers
Publication of
performance of
different setups
Reduced
numbers of
new wells
Greater quantities
of groundwater
available
Extension workers
promoting drip
irrigation
Source: Terry Smutylo
Who are your boundary partners?
Programme
Beneficiaries
Stakeholders
Boundary Partners
Focus of Outcome Mapping
Inputs
Activities
Outputs
Outcomes
Outcome Mapping
Impacts
Where is the map?
✓OM is a guide to the journey we take
with our partners. We co-create the map.
✓It focuses on the intention and what
happens along the way
✓The map is not the territory, it shows
the route taken
✓“The only real voyage of discovery
exists, not in seeing new landscapes, but
in having new eyes” (Marcel Proust)
Step 1: Vision
improved human, social, & environmental wellbeing
I have a dream!
Martin Luther King, Jr.
August 28, 1963
A vision statement..
•
Guides
• Motivates and inspires
• Is an ‘accountability-free zone’
Vision in graphic form, Nagaland (India)
33
Vision statement in narrative form
Local authorities, communities, and international organizations in developing
countries in Africa recognize the value of HIV/AIDS intervention as an integral part of
social & economic development. Municipal, regional, and national governments
actively support HIV/AIDS prevention activities by formulating and implementing
effective public health policies. Using research findings, they have developed a
comprehensive public health strategy to slow down the infection rate. Formerly
marginalized groups (e.g. women and youth) are organized into advocacy groups that
can effectively formulate their needs to policy makers. All groups have access to
reliable and relevant technical information about HIV/AIDS prevention and are able to
make informed choices. In essence, there are healthier, happier, and wealthier
communities.
Vision facilitation question
Imagine that, 5-10 years from now, the program has been
extremely successful. Things have improved beyond your most
ambitious dreams.
• What changes have occurred?
• What (& how) are your intended beneficiaries
doing?
• What are your partners doing?
• Describe the better world you are seeking.
Step 2: Mission
The mission is that “bite” of the vision
statement on which the program is
going to focus.
A mission statement describes:
• How the program intends to apply its resources
in support of the vision
• The areas in which the program intends to work
• How the program will support the achievement
of outcomes by its direct partners
Example Mission Statement
In support of this vision and on behalf of its donors, the program will work in the areas of
research, dissemination, capacity building, & coordination. It will contribute to the production,
synthesis, & dissemination of research data, position papers, & other information that will
sensitize local & international actors to HIV/AIDS prevention. The program will seek to expand the
range of disciplines involved in HIV/AIDS research. It will enhance HIV/AIDS research capacity in
order to produce credible information for local, national, & international policy-making &
program development. It will promote an interest in HIV/AIDS research among new researchers
by providing research fellowships, mentorship, & training opportunities. The program will
contribute to the development of linkages between Northern & Southern researchers &
encourage partnerships between research organizations, advocates, & decision makers. It will
increase its visibility & credibility among the donor community & will convince them of the utility
of supporting HIV/AIDS prevention.
Summary
Vision
Mission
✓ About the future
✓
Feasible
✓
Identifies activities and
relationships
✓
About the program
✓ Observable
✓ Idealistic
✓ Not about the program
Exercise 1:
Correcting Vision and
Mission Statements
Step 3:
Boundary Partners
Boundary Partners are...
Those individuals, groups, & organizations with
whom a program:
• interacts directly to effect change
• can anticipate some opportunities for
influence
• engages in mutual learning
Who are your boundary partners?
Programme
Beneficiaries
Stakeholders
Boundary Partners
Community
leaders
Network
member
District
official
Illegal
loggers
National
media
CSO
(TFCG)
Government
department
CSO
CSO
CSO
CSO
CSO
KPMG
DFID
example… AcT Tanzania
Strategic partners
• Selected for their potential to contribute to the
mission
• A person or group with whom the program
works directly to achieve the mission, without
necessarily wanting to change the partner’s
behaviour as part of the mission
• E.g. Donor agency, contracted service providers,
NGOs doing similar work, media agencies
Facilitation questions
• In which individuals, groups, or organizations is
your program trying to encourage change as a
contribution to the vision?
• With whom will you work directly?
• Are you choosing boundary partners because
you want to influence the ways they help or
influence others?
• On whose actions does your success depend?
Exercise 2:
Are you my boundary
partner?
Swayamsiddha Mission Statement
The Swayamsiddha Project works with governments, NGOs &
CBOs to improve women’s health and empowerment. It facilitates
the development of women’s self help groups. It provides them
with funding and training to help them influence community and
government services to be more responsive to their health and
livelihood needs. It fosters mutual respect and joint action
between these self-help groups and: banks; police; health and
social service providers; and government agencies. It researches
and promotes the application of ecosystem approaches to human
health in agriculture and in the provision of health and sanitation
services. Swayamsiddha addresses equity issues in all its
activities. It uses participatory methods to monitor progress, to
learn how to become more effective in supporting its partners and
to report on its results.
Step 5:
Progress Markers
Progress Markers
Love to see
(Deep transformation)
Like to see
(Active engagement)
Expect to see
(Early positive responses)
Love to
see
Like to see
Like to see
Like to see
Like to
see
Like to
see
Expect to
see
Expect to
see
Like to
see
Expect to
see
Why Graduated Progress
Markers?
• Articulate the complexity of the change process
• Allow negotiation of expectations between the program and its
partners
• Permit early assessment of progress
• Encourage the program to seek the most profound
transformation possible
• Help identify mid-course improvements
How can we measure:
Greater awareness…
Empowered women…
Community ownership…
Reduced conflict…
Increased collaboration…
Governmental commitment…
Gender sensitivity…
Equal access…
Budgetary transparency…
Active participation…
Poverty alleviation…
Strengthened capacity…
?
Examples
• Project expects to see Tanzania Forest Service
– Consult with communities on the new forest policy
and the national REDD strategy;
– Develop management plans for all Nature Reserves;
– Continue a policy that is supportive of participatory
forest management.
Examples
• Project would like to see Tanzania Forest Service
– respond promptly to information from communities about illegal
activities and other forest management issues and take action to
resolve the issue.
– incorporate recommendations from communities and civil society
organistions in the revised forest policy and the national REDD
strategy;
– approve voluntary social and environmental timber standards;
– implement and monitor management plans for nature reserves
and other high biodiversity forest reserves with the participation
of communities;
– approve independent forest monitoring and cooperate fully in its
implementation;
– incorporated Biodiversity indicators in NAFORMA
Examples
• Project would love to see Tanzania Forest Service
– successfully advocate for wider application of good
timber standards throughout Tz gov’t systems.
– consistently and effectively apply the Forest Act and
regulations within the timber and charcoal trade
thereby increasing the revenue that they capture and
reducing illegal timber harvesting and charcoal
production;
– incorporated social and environmental standards in
revised harvesting regulations and other documents
related to the national forest programme;
– Applying good social and environmental practices in
respect of reserve management
Facilitation questions
• How can the programme know the boundary
partner is moving toward the outcome?
• What would they be doing?
• What milestones would be reached as the
boundary partner moves towards their
intended role in contributing to the vision?
Progress Marker Checklist
Each Progress Marker:
 Describes a changed behaviour by the boundary
partner
 Can be monitored & observed
As a set, Progress Markers:
 Are graduated from preliminary to more profound
changes in behaviour
 Describe the change process of a single boundary
partner
Exercise 4:
Arranging Progress
Markers
Step 4:
Outcome Challenge
Outcome Challenge
• Describes behaviour of a single boundary
partner
• Sets out the ideal actions, relationships
activities
• Describes the boundary partner’s contribution
to the vision
Facilitation questions
• Ideally, in order to contribute to the vision,
what would the boundary partner be doing?
• With whom would they be interacting?
• How could this boundary partner contribute
maximally to the vision?
Women’s self help groups are taking action to make community and
government services more responsive to the health and livelihood needs of
women and girls.
They influence banks, police, health and social service providers, local
officials and state and national government agencies in relationships of
mutual respect and joint action to improve women’s well being.
Women’s self help groups arrange bank loans for members and for life skills
training for girls to be included in the school curriculum.
They influence local, state and national government policies and
expenditures on community improvement and transportation and support
women candidates to run for election to local government office.
Step 6:
Strategy Maps
Strategy Maps
For each Boundary partner:
• What will the Project do to SUPPORT desired changes in the
Partner towards the Outcome Challenge?
• Diverse range (an array) of ACTIVITIES:
- Aimed at Partner
- Aimed at Partner’s environment
- Cause, Persuade, Support
Strategy Map
causal
Partner
Environment
persuasive supportive
Facilitation Questions
causal
Partner
what will be
done to produce
immediate
outputs?
Environment
what will be
done to alter the
physical or policy
environment?
persuasive supportive
what will be
done to build
capacity?
how will
sustained
support,
guidance or
mentoring be
provided?
how will the
what networks or
media or
relationships will
publications be be established or
used?
utilized?
SM summary
• Outlines the programmes approach in working
with the BPs
• Indicates the relative influence the programme
is likely to have on the BPs
• Helps identify gaps in strategy – not to fill the
boxes
• Suggest the type of evaluation methods needed
Exercise 5:
Strategy Map
Step 7:
Organisational
Practices
Organizational Practices
How does your team or organization stay
relevant, viable and effective?
8 practices
1. Prospecting for new ideas, opportunities, and
resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest
power
4. Assessing and (re)designing products, services,
systems, and procedures
8 practices
5. Checking up on those already served to add
value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
Exercise 6:
Sharing organisational
Practices
M&E planning and
journals
M&E can be used for…
 Feeding
learning into management cycle
 Meeting accountability / reporting requirements
Reducing risk
 Multi-actor learning process
Decision-making
Inform publicity documents, communication
activities, or case-study materials
What else…?
Effective use of M&E depends on…
USES
EVENTS
USERS
INFO
Adjust Strategy
Annual Staff Mtg
Program Staff
Strategies
implemented
Improve
Implementation
Annual Report
Board
Morbidity & mortality
Donor
partner actions in
community
Project Partner
community
responses to proj.
Report
Performance
Quarterly Report
to Donor
3yr Prog Review
Risk Management
Program
Management
partner collab. with
ministry
Proj team Mtg
Build Capacity
community
participation
Lobbying
funding flows
Effective use 
“I can honestly say
that not a day goes by
when we don’t use
those evaluations in
one way or another.”
5 kinds of M&E Information
Contextual InformatIon
Program
Partner
State, status
or situational
data
Strategies
outcomes
(behaviour changes in the
relevance & viability
(actions of the program)
partners)
implementation
(interventions by the program)
Lets look at Swayemsida in a simple form…
Expect to see women victims of domestic violence:
Participating when abuse is discussed
2. Attending self help group meetings
1.
Like to see women victims of domestic violence:
3. Educating self about combating domestic violence
4. Setting an agenda for taking collective action
5. Raising funds for victim support services
Love to see women victims of domestic violence:
6. Taking action to protect self
7. Lobbying police to change enforcement practices
M&E Planning Worksheet
What
How will it be
information will collected, from
be collected?
what sources and
when?
Who will
collect it?
Who will make
sense of the info
& how?
Who will
manage the
collection-tousage
processes?
What
resources are
needed &
who
contributes?
Focus your questions…
• many interesting & important questions, but data costs money &
time
• vague questions yield vague answers
• avoid nice-to-know & might-as-well data
• OM can’t answer all questions but it can help illuminate:
• achievement of outcomes (expected & unexpected)
• factors and actors contributing to outcomes
• implementation of the program
Outcome Mapping M&E…
• systematic collection of data
• a regular learning & improvement cycle
• encourages the program to challenge itself
• framework for collecting & organizing, does not analyze data nor
tell you how to use it
Outcome journal
•
•
•
•
•
What, when, who, how much
Contributing factors
Sources of evidence
Unanticipated change
Lessons, required changes, action
Outcome journal
Strategy journal
•
•
•
•
•
What did you do, with whom, when
(How) did it influence change
Outputs
Suggested changes/follow up
Lessons
Performance journal
• Progress against internal commitments – e.g.
organisational practices
• Reviewing the logic of the programme
Journal for Monitoring Outcomes
Outcome Monitoring Journal
Period of monitoring and data collection:
Contributors to Monitoring Update:
Name of the person(s) who compiled the journal:
Outcome Challenge:
Based on your analysis of each progress marker, please indicate the number on the scale that best represents the boundary partner's progress in
achieving the progress marker (1 indicates no activity towards the progress marker while 7 represents full achievement of the progress marker)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
Progress markers
What happened (who, how, Date Strategy Used and its
Follow up/corrective measures
what, where)
(when) Efectiveness
1)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
2)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
3)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
4)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
5)
Not started yet/no activity towards progress marker 1 2 3 4 5 6 7 Progress marker reached/strong activity towards progress marker noted
Evidence
(documents
etc.)
Anecdote as evidence
Taking up M&E findings…
• What should we keep doing?
• What do we need to change in order to improve?
• What strategies/practices do we need to add?
• What strategies/practices do we need to drop? (i.e., they
produced no results, they require too much efforts or
resources to produce results?)
• Has any issue come up that we need to evaluate in greater
depth? What? When? Why? How?
Considerations for internalizing OM…
• Decision / will (of the organization, of those involved)
• Incentives (learning)
• One person or team that is responsible (that has the
support of governing bodies); facilitator; champion (at
least one, more is better)
• Management-donor buy-in (challenges of that)
• Experiment or to meet needs
Coming back to indicators …
Don’t get lost in music …
Appreciative inquiry
Case study
Composite Logic Model
Content analysis
Contribution analysis
Cost-Benefit Analysis (CBA)
Creative expression
Critical Stories of Change
Developmental Evaluation
Do No Harm
Dreams realised or visioning
Episode Studies
Expert review
GIS mapping
Graphing results
Group assessment
Historical trends and timelines
Horizontal evaluation
Impact evaluation
Impact flow diagram
Impact Pathway
Innovation Histories
Institutional history
Institutional linkage diagram
Learning alliances
Planning and Implementation
Learning-oriented evaluation
Framework Analysis (PIFA)
Lot Quality Assurance Sampling Poverty Impact Assessment
Logical Framework Analysis
(PIA)
M&E wheel (or "spider web")
RAPID Outcome Assessment
Method for Impact Assessment of Results Based Management
Programmes and Projects (MAPP) Rich pictures (or mind maps)
Modular Matrices
Rigorous Impact Analysis (RIA)
Monitoring of Effects (MOVIE) SAGE – Situational Analysis and
Most Significant Change (MSC) Goal Establishment
Network Functions Approach
Social Framework
NGO-IDEAs Toolbox
Social mapping or well-being
Non-random sampling
ranking
Observation
Social Network Analysis
Outcome Mapping (OM)
Sociograms
Participatory Impact Monitoring Stakeholder analysis
(PIM)
Sustainability Impact
Participatory Impact Pathways
Assessment (SIA)
Analysis
SWOT
Participatory Livelihood
Systems (or inputs-outputs)
Monitoring (PaLSA)
diagram
Peace and Conflict Impact
Theory of Change (ToC)
Assessment (PCIA) / Aid for
Theory-based evaluation
Peace (AfP)
Transects
Exercise 7:
Pros and cons of OM
Thank you!
For more info on OM:
Visit the OM learning community:
www.outcomemapping.ca
Contact:
Simon Hearn, [email protected]
Julius Nyangaga, [email protected]
Ziad Moussa, [email protected]
outcomemapping.ca