Outcome Mapping Team Meeting February 2010

Key evaluation challenges

• • • • • measuring development results establishing cause & effect in an open system timing encouraging iterative learning clarifying values 2

problem with « impact »

Impact Implies: Development Implies:

• Cause & effect • Positive, intended results • Focus on ultimate effects • Credits a single contributor • Story ends when program obtains success • Open system • Unexpected positive & negative results occur • Upstream effects are important • Multiple actors create results & need credit • Change process never ends 3

Recognizing that change is…

• • • • • Continuous Complex Non-linear Multidirectional Not controllable 4

OM - a flexible, multiple-use tool • Planning • Monitoring • Evaluation

5

Boundary Partners Ultimate beneficiaries

Thanks to Paul Crawford

IMPACT OUTCOMES OUTPUTS RESOURCES SPHERE OF CONTROL SPHERE OF INFLUENCE SPHERE OF INTEREST GeSCI Strategic partners Strategic partners

focus on direct partners

• Key concept is « boundary partners » • The individuals, groups, and organizations you work with directly and anticipate opportunities for influence 7

Focus of Outcome Mapping

Inputs Activities Outputs Outcomes Impacts

Behavioural Changes

Why behaviour changes?

• To stress that development is done by and for people • To illustrate that although a program can influence the achievement of outcomes, it cannot control them because ultimate responsibility rests with the people affected 9

Contribution not attribution

You can influence but not control change in your partners 10

Primary uses

• PLANNING: articulate goals & define activities • MONITORING: assess program performance & partners’ outcomes • EVALUTION: design & conduct a use oriented evaluation 11

The Questions that Outcome Mapping will help to answer :

•

In designing and articulating the program’s logic

- What are our development goals?

- How can our programme contribute to those development goals?

- Who are our beneficiaries?

- How can we help our beneficiaries contribute to the broader development goals?

•

Recording internal and external monitoring data

- How far have our beneficiaries progressed towards achieving outcomes?

- What are we doing to support the achievement of those outcomes?

- How well have we performed?

•

Indicating cases of positive performance and areas of improvement

- What worked well? Why?

- How can we maximise our contributions?

Kiely, Team Meeting, December 2007

The Questions that Outcome Mapping will help to answer (continued) :

•

Evaluating intended or unexpected results

- Who changed / what changed? How?

- If they did not change as expected, do we need to do something different or reorient our expectations?

•

Gathering data on the contribution that a program made to bringing about changes in its partners

- What activities / strategies were used?

- How did the activities influence the individuals, groups, or institutions to change?

•

Establishing evaluation priorities and an evaluation plan

- What strategies, relationships, or issues need to be studied in depth?

- How, and from where, can we gather relevant data?

Components of Outcome Mapping

•

A description of the macro-level changes to which program intends to contribute

vision, mission, boundary partners and outcome challenges, progress markers •

Strategy maps

•

Organisational Practices Journal

•

A monitoring framework

outcome journal, strategy journal, performance journal •

Evaluation Plan

boundary partners

Those individuals, groups, and organizations with whom the program • • • interacts directly to effect change anticipates opportunities for influence engages in mutual learning 15

Boundary partners have boundary partners

program program’s bp bp’s bp 16

Outcome challenges

describe: • the ideal behavioural changes (relationships, activities & actions) of a boundary partner • how they will contribute ideally to the vision.

17

Example: Outcome Challenge Statement (IDRC)

Vibrant and dynamic

regional and national knowledge building and sharing communities

which will produce and share high quality evidence based research and experiences on ICT4E and KS. These communities will work with GeSCI in a pro-active and committed way to identify critical issues, develop appropriate solutions and support one another on a peer basis. They will make optimal use of the GeSCI resources, tools and expertise. They will commit themselves to participate in leadership capacity building for ICT4E and KS and acquire and apply critical identified skills in their respective fields for system transformation and innovation. 18

Progress markers

✓ ✓ ✓ ✓ ✓

A graduated set of statements

describing a progression of changed behaviours in the boundary partner Describe changes in actions, activities and relationships leading to the ideal outcome; shows story of change Articulate the complexity of the change process Can be monitored & observed Permit on-going assessment of partner’s progress (including unintended results) 19

Example

Boundary partner Outcome challenge Government officials and policymakers Ministries of Education(s) Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation.

Progress markers

Expect to see 1.

2.

3.

Negotiating with SchoolNet and other service providers on details of implementation Setting a priority list for implementation Taking control of the initiative Like to see 1.

2.

3.

4.

Efficient coordination of partnerships for delivery of ICT services to schools.

Establishing guidelines for integrating ICT effectively into learning and teaching Setting operational objectives for implementation based on the ICT policy for education Direct budgeting for expenditure by the Ministries of Education on the programme to be specified in the Policy Implementation Plan Love to see 1.

2.

3.

Creating the critical mass of ICT skills at school level to be able to realise the country’s vision for the future.

Setting specific guidelines on cost-effectiveness for ICT solutions for schools or putting implementation out to tender to see who offers the best option Using the ICT initiative to facilitate a change in emphasis to student-centred learning and teaching.

Taken together, progress markers

 facilitate mid-course corrections and improvements  stimulate the program to consider how it can contribute to the most profound transformation possible  articulate the complexity of change 21

X 22

Strategy Maps

• Outlines the program`s approach in working with the boundary partner • Breaks strategies down into different types so you can be as strategic and influential as possible 23

6 types of strategies

Strategy Aimed at the Boundary Parnter Causal I-1 • Direct Output Persuasive Supportive I-2 • Arouse New Skills/ Thinking I-3 •Supporter who guides change over time Aimed at the Boundary Partner`s Environment E-1 •Alter physical or regulatory environment E-2 •Modify the information system E-3 •Create / Strengthen a Peer Network 24

Practice: Facilitation Questions

I-1 What will be done to produce an “immediate” output?

E-1 What will be done to change the physical or policy environment?

I-2 What will be done to build capacity?

E-2 How will you use the media or publications?

I-3 How will sustained support, guidance, or mentoring be provided?

E-3 What networks/ relationships will be established or utilized?

25

M&E Planning

Use(s) and User(s)

– Who will take action(s) or make decisions with the help of the M&E process or findings? What actions and/or decisions will be taken?

Questions Timing Methods & Data Sources Responsibility & Roles Dissemination & Communication Costs

What do you want to know about our programmes?

When are the findings needed in order to be useful to the user(s)?

When will the data be collected?

How will data be collected? (I.e., instruments) From what sources?

Who is going to lead the process? Who needs to play what role to accomplish the work? (eg., who will manage collection? analysis? writing/reporting?) How will the findings be communicated to the intended user(s)?

What should the final products look like in order to be useful?

Who else will be interested in the findings and should be informed of them? What human and financial resources are needed and who contributes?

What resources are available or not?

27

different tools you can use to collect data

• Outcome/ strategy mapping journals • most significant change methodology • project cycle analysis • activity system analysis • any social science tool or method • ….

28

outcome journal

To understand the change process in boundary partners. Collects information about: • Story of change and reasons for change • Unexpected changes • The actors and factors that contributed to that change • How we know the change occurred • Learnings 30

outcome journal

Work Dating From/To: Contributors to Monitoring Update: Outcome Challenge: Low = Medium = High = Expect to see: LMH 1 2 3 Who?

31

set values for low, medium & high

• Can capture either quantity or quality of change(s): – Quantity (number of boundary partners) – Quality (depth of change) 32

strategy journal

To understand how the programme is contributing to changes in boundary partners. Collects information on: • • • • • • Resources that have been invested Strategies and Activities Products Effectiveness Changes that need follow up Learnings 34

strategy journal

Working Dating From/To: Contributors to Monitoring Update: Strategy to be Monitored: Effectiveness?

(How did it help the boundary partner?) Outputs 35

how to use the journals

• alone or combined with other tools and methods • focus groups, interviews, e-surveys, conference calls, revision of documents • can incorporate quantitative and qualitative methods • face-to-face workshops • other… 36

new technologies for data collection

• • • • • • • • discussion forums handhelds participatory video survey monkey blogs chat skype what else…?

37

Expanding accountabilities

OM helps you: • manage multiple accountabilities: up, down, horizontal by: – being able to document & communicate outcomes (traditional understanding of accountability) – engage in social learning with your boundary partners making you more accountable to yourself, your organization, and them 38

Taking up M&E findings

• What should we keep doing?

• What do we need to change in order to improve? • What strategies/practices do we need to add?

• What strategies/practices do we need to drop? (i.e., they produced no results, they require too much efforts or resources to produce results?) • Has any issue come up that we need to evaluate in greater depth? What? When? Why? How?

39

Reviewing programme design

1. Read the vision statement 2. Read the mission statement Does this still reflect the program's dream?

Is this the greatest contribution our program can make? Have we been doing this? Why? Why not? Should we add anything or take anything away?

3. Review boundary partners 4. Review outcomes 5. Review progress markers Is this who we are working with directly? Do we need to work with anyone else?

Do these accurately describe the ideals way that our boundary partners could act to contribute to the achieving of the vision?

Was the change process we set out accurate and useful? What now needs to be added or taken out?

6. Review strategies 7. Review organizational practices What did we plan to do? Have we implemented these activities? Why? Why not?

Are we doing everything we can to maintain our capacity to support our partners?

40

Discussion questions

• What elements of OM would we like to integrate that we think would be helpful? Why? • What do we need to do to make this happen? • What spaces, methodologies & tools do we already use that OM could complement? (how can we integrate OM into what we are already doing?) • What challenges will we face? What opportunities are there to take advantage of? • How will we communicate to others? • What will be our next steps? 41

OM 2009

Review of Pilot OM Implementation within Culture for Organizational Learning STRENGTHS Identify areas where OM was used to provide a benefit to organization, teams and partners in 2009 WEAKNESSES Identify areas of organizational practice which needed improvement in order to achieve desired outcome of OM integration in 2009 OPPORTUNITIES Identify areas of OM that could support organizational learning and development in 2010 - that are not currently being utilized THREATS Identify possible obstacles or problems to the integration of OM horizontally across our programmes in 2010