Transcript Boundary Partners

Challenging Evaluation:
An Introduction to Outcome Mapping
Amy Etherington & Rebecca Lee
Mini-training for IDRC Interns & PDAs
December 7 & 8, 2005
Session overview:
 Evaluation context – where does OM
fit?
 Introduction to OM methodology – main
concepts and tools
 OM book – it’s useful!
 How is OM being used – when is it
appropriate?
Warm-up question:
What words come to
your mind when you
think of “evaluation”?
IDRC’s Evaluation System
 Promotes ownership & use of findings at all
levels
 Decentralized
 Focus on evaluation processes
 Monitors quality
 Accountability for results at program &
project levels
IDRC’s Evaluation Unit
 Strategic evaluations
 Cross-cutting issues
 Corporate reporting
 To BoG on performance and results
 Information systems
 Capacity building
 With Southern partners
 Promoting M&E with programs
 Tools & methods
 Organizational Assessment
 Outcome Mapping
Monitoring & evaluation challenges:
1. Establishing cause & effect in open systems
2. Sharing ownership & participation
3. Recognizing the contributions of others
4. Tracking progress
5. Encouraging iterative learning
6. Measuring development results of research
7. Timing
Shifting influence over time
Influence
High
Low
Time
Beneficiaries/Indigenous Actors
Program
Are we
having an
impact yet ?
A framework that allows
researchers to plot human
behaviour and actions and
assess their contribution to the
aims of research projects and
programs
IDRC’s Evaluation Unit
(1998-2000)
Outcomes as changes in
behaviour, relationships,
activities or actions of the
people, groups and
organizations with whom a
program works directly
OM bumper-stickers:
The map is not the territory!
 Be prepared for
surprises along the way
Keep your eyes wide open!
Being attentive along
the journey…
This is as important as
the destination
OM t-shirts
CONTRIBUTION
ATTRIBUTION
OM t-shirts
INFLUENCE
IMPACT
OM recognizes that in a
development context, change is:





Complex (multiple actors and factors)
Continuous (not limited to the life of the
project)
Non-linear (unexpected results occur)
Beyond the control of the project (but
subject to its influence)
Two-way (program also changes)
Principles of use
 Flexible: modular - use adapted to
circumstances
 Participatory: seeks dialogue and
collaboration with boundary partners in
P,M,&E
 Evaluative thinking; culture of
reflection: promotes social and
organizational learning
Step 1: Vision
I have a dream!
Martin Luther King, Jr.
August 28, 1963
Vision is a guide:
North
Light house
Distant star
Vision statement:
Reflects the broad human, social, and
environmental betterment in which the
program is engaged and to which it is
contributing
Written in present tense as if the vision were already a
reality
“Those who dream
make a difference!”
Ms. Kalpana Pant
-Chaitanya
Women and girls in rural India enjoy full access to appropriate
health care, education, food and water security and freedom from
violence. They earn their own livelihoods with free access to
markets and to banking and municipal services. Women
knowledgeably use drudgery reduction technologies and
agricultural practices that support human and ecological wellbeing. Villages are served by public transport, are well lit at night
and have police forces that enforce laws fully and equitably. Girls
attend school full time and, with their families, have the
information and resources to make informed decisions regarding
their personal health, safety and social needs. Gender equity is
evident in the allocation of household labor and household
decision-making; and men in the community support genderresponsive initiatives. Schools reinforce the role of women as
educated, valuable and empowered community members.
Through community-based organizations, women participate in
and influence decisions, policies and programs affecting their
well-being and share their experiences with others.
Vision: facilitation questions
 What is the human, social and/or
environmental condition that you hope to
see in the region in which the program is
being carried out?
 Who is contributing to this situation - and
how?
Dream boldly !
Step 2: Mission
The mission is that “bite” of the vision
statement on which the program is going
to focus.
Written in future tense as something the program
will do
Your mission is your
“business”
 What do you do?
 Who do you do it with?
 Why do you do it?
Mission: facilitation questions
 How can the program best support
or contribute to the achievement of
the vision?
 What areas do you need to work in?
 Where do you have credibility?
 Who can you work with?
Vision
Mission
Step 3: Boundary Partners
Boundary Partners:
definition
Those individuals, groups, and
organizations with whom the program:
 interacts directly to effect change
 anticipates opportunities for influence
 engages in mutual learning
= partners
The rest of the
world
Program
Sphere of influence
The rest of the
world
Program
= partners
Boundary Partners have Boundary Partners
program
program’s bp
bp’s bp
Families
SHG
Banks
PHCs
Police
Community
Leaders
State
NGO
State
NGO
State
NGO
State
NGO
State
NGO
State
NGO
BAIF
IDRC
CIDA
Swayamsiddha
Boundary Partners:
Facilitation Questions
 In which individuals, groups, or organizations
is our program trying to encourage change so
that they can better contribute to the vision?
 With whom will we work directly?
 Are we choosing X BP because we want to
influence their behaviour and actions, or
because they will influence others? Or both?
 What behavioural changes do we (the project
and BPs, collectively) want to see in the BP
that will contribute to the vision?
Vision
Mission
BOUNDARY
PARTNER 1
BOUNDARY
PARTNER 2
BOUNDARY
PARTNER 3
Step 4 & 5:
Outcome Challenges and
Progress Markers
Outcome Challenge:
definition

One OC is about a single boundary
partner

Describes the ideal behavioural
changes, relationships, actions and
interactions in this partner

Describes how these changes will
contribute to the vision.
Outcome Challenge: characteristics

One for each boundary partner

Does NOT describe program
strategies

Written like this: “The program
intends to see [boundary partner]
who [description of behaviours in
the active present tense]”
Outcome Challenges: facilitation
questions
Ideally, how would your boundary
partner be acting?
With whom would they be interacting?
 What would they be doing?
Progress Markers:
description

A graduated set of statements describing a
progression of changed behaviours in the
boundary partner

Describe changes in actions, activities and
relationships leading to the ideal outcome

Articulate the complexity of the change process

Can be monitored & observed

Permit on-going assessment of partners’
progress (including unintended results)
3 Levels of Progress Markers
The program sets out what it would:
 Expect to see the boundary partner
doing?
 Like to see the boundary partner doing?
 Love to see the boundary partner
doing?
Progress markers = ladder of change
Love to see
Truly transformative
Set quite high
Like
to
see
More active learning,
engagement
Expect to see
Early response to
program’s basic activities
X
Why graduated progress markers?
Taken as a set, the progress markers:
 are graduated from easier to more difficult to achieve
changes in behaviour
 describe the change process of a single boundary
partner
 are more complete than a single indicator
 help the program think about how it can intentionally
contribute to the most profound transformation possible
 facilitate mid-course corrections and improvement
Vision
Behaviors, Actions, Relationships of
Boundary Partners
OUTCOMES
Mission
BOUNDARY
PARTNER 1
Outcome
Challenge
BOUNDARY
PARTNER 2
Outcome
Challenge
BOUNDARY
PARTNER 3
Outcome
Challenge
Progress Markers
Love to See
Progress Markers
Love to See
Progress Markers
Love to See
Like to See
Like to See
Like to See
Expect to See
Expect to See
Expect to See
Influencing Outcomes
Program
Program relevance
& viability
Org. Practices
Boundary
Partner
Program
Delivery
Strategies
Program Results
Progress Markers
Step 6: Strategy Map
Strategy Map
 Outlines the program`s approach in working
with the boundary partner
 Indicates the relative influence the program
is likely to have on boundary partner
 Helps pinpoint strategic gaps in the
approach or if the program is overextended
6 Types of Strategies
Strategy
Aimed at
the
Boundary
Parnter
Causal
Persuasive
Supportive
I-1
I-2
I-3
• Direct Output
• Arouse New
Skills/ Thinking
•Supporter who
guides change
over time
E-2
E-3
•Modify the
information system
•Create /
Strengthen a
Peer Network
E-1
Aimed at the
•Alter physical or
Boundary
regulatory
Partner`s
environment
Environment
Facilitation Questions:
I-1
I-2
I-3
What will be
done to produce
an “immediate”
output?
What will be
done to build
capacity?
How will
sustained
support,
guidance, or
mentoring be
provided?
E-1
E-2
What will be
done to change
the physical or
policy
environment?
How will you use
the media or
publications?
E-3
What networks/
relationships will
be established
or utilized?
Strategy images
Vision
Behaviors, Actions, Relationships of
Boundary Partners
Undertaken to Bring
about Outcomes
ORGANIZATIONAL
PRACTICES
STRATEGIES
OUTCOMES
Mission
BOUNDARY
PARTNER 1
Outcome
Challenge
BOUNDARY
PARTNER 2
Outcome
Challenge
BOUNDARY
PARTNER 3
Outcome
Challenge
Progress Markers
Love to See
Progress Markers
Love to See
Progress Markers
Love to See
Like to See
Like to See
Like to See
Expect to See
Expect to See
Expect to See
Strategy Map 1
Strategy Map 2
Strategy Map 3
Step 7: Organizational
Practices
Organizational Practices: definition
What the program does to:
look within
stay fresh, sharp, effective, healthy
better serve its partners
learn and change
Why Organizational Practices?
 Important to how the program is going
to function to effectively fulfill its
mission
 Supporting change in its boundary
partners requires that the program be
able to change and adapt
8 Organizational Practices
1. Prospecting for new ideas, opportunities,
and resources
2. Seeking feedback from key informants
3. Obtaining the support of your next highest
power
4. Assessing and (re)designing products,
services, systems, and procedures
8 Organizational Practices
5. Checking up on those already served to add
value
6. Sharing your best wisdom with the world
7. Experimenting to remain innovative
8. Engaging in organizational reflection
Vision
Behaviors, Actions, Relationships of
Boundary Partners
Undertaken to Bring
about Outcomes
ORGANIZATIONAL
PRACTICES
STRATEGIES
OUTCOMES
Mission
BOUNDARY
PARTNER 1
Outcome
Challenge
BOUNDARY
PARTNER 2
Outcome
Challenge
BOUNDARY
PARTNER 3
Outcome
Challenge
Progress Markers
Love to See
Progress Markers
Love to See
Progress Markers
Love to See
Like to See
Like to See
Like to See
Expect to See
Expect to See
Expect to See
Strategy Map 1
Strategy Map 2
Strategy Map 3
Step 8-11: Monitoring
priorities and Journals
Count every " F " in the following text:
FINISHED FILES ARE THE RE
SULT OF YEARS OF SCIENTI
FIC STUDY COMBINED WITH
THE EXPERIENCE OF YEARS...
M & E involves making choices
“You can’t always get what you want…
But if you try sometimes, you might find
You get what you need… ahhhhhhh, yeah…”
The Rolling Stones
Choosing WHAT to monitor
 Who will use the monitoring
information?
 What will it be used for?
 When is it needed?
Choosing HOW to monitor
 Which components will be monitored?
 How and when will data be collected?
 Who will collect it?
 Who will analyze, collate, package data?
 Where and when will it be discussed and
used?
Monitoring needs to be:
 Useful
 Affordable
 Light in work
 Understandable
Outcome Journal
Work Dating From/To:
Contributors to Monitoring Update:
Outcome Challenge:
Low =
Medium =
High =
Expect to see:
LMH
1
2
3
Who?
Strategy Journal
Working Dating From/To:
Contributors to Monitoring Update:
Strategy to be Monitored:
Effectiveness?
(How did it help
the boundary
partner?)
Outputs
Performance Journal
Working Dating From/To:
Contributors to Monitoring Update:
Practice 1: Prospecting for New Ideas, Opportunities, & Resources
Example or Indicators:
Sources of Evidence:
Lessons:
Practice 2: Seeking Feedback from Key Informants
Example or Indicators:
Sources of Evidence:
Lessons:
Step 12: Evaluation Plan
Evaluation Plan:
 A short description of the main planning
elements of an evaluation study
 For Whom? What? Where? How? By
Whom? When? How Much?
Elements of an Evaluation Plan
Evaluation Issue:
Date Cost
Evaluation Who Will
Who Will Use Questions Info.
Sources Methods Conduct & (Start &
to be
Evaluation,
Manage the Finish)
How, & When? Answered
Evaluation?
Setting Evaluation Priorities
Criteria with which to prioritize include:
 Learning needs
 Reporting requirements
 Partners’ information needs
How can OM be used?
 Designing and articulating the program’s logic
 Recording internal and external monitoring data
 Indicating cases of positive performance and
areas for improvement
 Evaluating intended and unexpected results
 Gathering data on the contributions that a program
made to bringing about changes in its partners
 Establishing evaluation priorities and an evaluation
plan
When is OM best used?
 Once strategic direction or primary program
areas are established
 Particularly effective for larger projects
 Best used at the start, but can also be used
as a midway or final assessment tool
 Activities must be sufficiently specific to
identify key groups who will be influenced
Is OM appropriate? Not always!




May require changes that are not possible
Able to focus on “OM” outcomes
Commitment to change and self-assessment
Not intended for technical evaluations
 (assessing relevance of a programming area or a costeffectiveness comparison)
 Credible, and compatible with donor reporting
requirements
 Team consensus
 Resource commitment