Transcript Activity results frameworks

Partnerships Fund Round 5:
Design Workshop
Developing Activity Results
Frameworks to support design and
implementation
February 2015
SESSION OVERVIEW
Objective
• To develop a shared understanding of Activity Results Frameworks to
support Activity design and Activity management during implementation
Approach
1. Activity Results Frameworks overview (15 mins)
• What are Activity Results Frameworks?
• Why are they important?
• How are they used in design and management of Activities?
2. Activity Results Frameworks - 3 components in-depth (3hrs)
• Key messages, good practice example
• Group exercises using your designs
Time: 3hrs 15mins
WHAT ARE ACTIVITY RESULTS
FRAMEWORKS?
• NZ Aid Programme uses a results-based approach for designing and
managing Activities.
• Activity Results Frameworks key mechanism to drive results efforts
 Design phase: agreed logic, intended results, approach to monitor and
evaluate
 Implementation phase: collection, analysis of results information to
support Activity management
 Overall: supports learning, improvement, decision-making,
accountability and communications
• Also referred to as Monitoring and Evaluation Frameworks, Log-Frames etc.
WHAT ARE ACTIVITY RESULTS
FRAMEWORKS?
NZ Aid Programme Activity Results Framework – 3 components
1. Results Diagram
 Identifies logic supporting the Activity and intended results (based on
results chain, causal-logical relationship)
2. Results Measurement Table
 How results will be measured (indicators, targets, etc)
3. Monitoring and Evaluation Work Plan
 Tasks and resources required to support measuring results - crucial for
implementation
We will look at each in depth later in the session- see example
ACTIVITY RESULTS FRAMEWORKS
WHY ARE THEY IMPORTANT?
Measuring Results – New Zealand Aid Programme Priority
• Internationally recognised as essential for effective aid and development
– Key aid and development principle
– Links: ownership, transparency and mutual accountability
• Results at forefront of our work - planning, implementation and completion
– Developing greater evidence base to support: management, learning,
decision-making, accountability, communications
• For all NZ Aid Programme Activities – we need to know:
– Intended results to be delivered (outputs) and supported (outcomes)
– Resources ($$) required to deliver and achieve results
HOW?
THE RESULTS CHAIN AND
RESULTS LANGUAGE
Results Chain
• Central to Results Frameworks - diagrammatically presents the logic / theory
supporting results (outputs to outcomes)
• Multiple results chains build a single results diagram
Results Language
• Results include both outputs and outcomes
• Clearly defining results is crucial:
– We need common agreement of what are the intended results
– Activity budgets are based on outputs (strongest accountability)
– We implement Activities to achieve outcomes
HOW?
RESULTS LANGUAGE
Outputs
• Describes services, capital goods and products
• Stated within budget (clear accountability) - can be purchased
Outcomes
• Describes the change or effect that occurs (uses a directional verb)– can not
be purchased
– An output: Teacher training provided
– An outcome: Improved teacher performance
RESULTS CHAIN
Long-term
Outcomes
Influence and ability to
attribute reduces as the
level of outcome
hierarchy increases
(i.e. shift to contribution
with others)
Medium-term
Outcomes
Managing for
Development
Results
Short-term
Outcomes
Outputs
Control / Accountability
Attribution strongest
Inputs / activities
Activity
Management
HOW?
RESULTS TIMING / DURATION
When is short-term, medium-term and long-term?
• Dependent on each Activity and its design – and “pitch” of results.
General rules:
• Outputs delivered during implementation
• Short-term outcomes - progressed / achieved during implementation
• Medium-term outcomes - progressed / achieved during/ post implementation
• Long-term outcomes - achieved post implementation.
• Level of attribution / contribution changes for different levels of results
Terminology – Results Chain – Examples
Long-term
Outcomes
Long-term (Ultimate) results:
Changes in state - conditions,
social, economic, civic,
environmental, etc.
Medium-term
Outcomes
Medium-term (intermediate)
results:
Change in behaviour and
practice, etc
Short-term
Outcomes
Outputs
Improved health of local
community
Improved quality of Results
Frameworks across Aid
Programme
Increased use of clean water
Participants able to develop a
Results Framework to
standard and better engage
with stakeholders
Short-term (immediate) results:
Change in ability, skills,
awareness, knowledge, skills, etc
Improved water quality
Improved knowledge of
Results Frameworks policy
and guidelines
What we do:
deliver services, develop
products, construct, provide
materials, train, etc
Wells built
Results Framework Training
provided
HOW - ACTIVITY RESULTS
FRAMEWORKS DURING DESIGN
Design Process
• Results Frameworks are an essential part of design process
• Developed to a good standard during design - some tasks, minor tweaks can
be made in implementation (include in M&E Work Plan)
• Results Frameworks should be developed in a participatory manner
– Supports buy-in, ownership, accountability and better design
• Consider partner capacity to support results management
HOW - ACTIVITY RESULTS
FRAMEWORKS DURING IMPLEMENTATION
AND COMPLETION
Implementation
• Early in implementation - ensure the Results Framework is fit for purpose
– Check monitoring tasks (i.e. baselines), systems, processes and capacity is
adequate
• Key purpose of Results Frameworks is to support Activity Management
– Timely, accurate performance and results information is essential for good
management of performance
Completion
•
•
Results Frameworks supports evidence-based decisions to transition to next
phase or completion.
Supports possible Activity evaluation
Activity Results Frameworks
3 components
1. Developing Results Diagrams
RESULTS DIAGRAMS
KEY MESSAGES
A results diagram provides:
• A clear description of key results – those that are critical
to success
• A demonstration of underlying logic or ‘theory of change’
supporting the Activity – using vertical arrows
• A visual representation of how the outputs will contribute
to the outcomes and ultimately the goal of an Activity
Get the diagram right before moving on to other
elements of the ARF
Results Diagrams also include:
• A goal: intention of the Activity at sector, regional or country level
(not measured)
• Activities and inputs table: summary of the activities and inputs
required to achieve the outputs.
Results Diagrams are crucial to build a common understanding among
stakeholders
Test: Can an outsider pick-up the results diagram and take away a
good understanding of the Activity and what will be achieved (and
what NZ funding will contract for)?
RESULTS DIAGRAMS
KEY MESSAGES
Results Diagrams – Tips
1. Focus on well-defined key results only – this makes the task of
monitoring and reporting more manageable.
– Be strict – we can’t monitor every result
– Each result should include one idea
– M&E requires resourcing (focus our efforts)
2. Aim for a pyramid shape
– More results should be described at output and STO levels vs.
MTO and LTO levels (reflecting focus on accountability/ attribution/
contribution)
RESULTS DIAGRAMS
KEY MESSAGES
3. Test logic between results – causal relationship (e.g. cause and
effect)
• To test: ask the “If…then….because” statement
• No one correct logic - but easy to see where logic is flawed
If we produce output A then what will be the effect / change (outcome)?
For example:
If we successfully provide teacher training, then the effect will be increased
knowledge of students, because teacher knowledge of learning techniques
is important to student learning.
USEFUL TIPS: ASSUMPTIONS AND
RISKS
4. Key assumptions and risks should be noted
• Assumptions are a positive statement of conditions or events needed to
achieve results
• Risk are a negative statement of events/conditions that could adversely
impact on achievement of results (list in Risk Matrix)
For example:
If we successfully provide teacher training, then the effect will be increased
knowledge of students, because teacher knowledge of learning techniques
is important to student learning (as long as training is based on agreed
teaching techniques and curriculum).
USEFUL TIPS: CROSS-CUTTING
ISSUES
6. Cross-cutting issues
• These should be considered as part of the broader design – and
included in ARF where relevant
Three approaches for gender:
1. Specific results that address gender equality
2.Gender-specific indicators
3. Sex-disaggregated data (for all indicators where the number of
recipients is measured)
PROCESS FOR DEVELOPING
RESULTS DIAGRAMS
General Approach
• Facilitator is important to lead the process
• Workshopping is best
• Usually takes – back and forth – more than a single meeting
What do we already know?
• Work from the top down- what will success look like?
• Part of design process – draws on broader design thinking
– Context analysis, problem analysis, etc (not in isolation)
– What is scope of Activity?
PROCESS FOR DEVELOPING
RESULTS DIAGRAMS
Process
1. Ensure a shared understanding of ARFs
2. Discuss already identified aspects:
– Design aspects – context analysis, problem analysis, etc
– Goal, outcomes, outputs
3. Brainstorm other potential outcomes and outputs
- An opportunity for stakeholders to identify “their” results
4. Use arrows from the outputs linking (logically) to intended outcomes
at the higher levels (i.e. causal relationship between results)
5. List any key assumptions being made about the logic / results
EXAMPLE RESULTS DIAGRAM
Goal: Improved livelihoods through increased protection of the environment in x
region
Long-term
outcome(s)
Mediumterm
outcomes
Short-term
outcomes
Outputs
Increased protection of immediate environment including coastlines (where applicable), and within
villages
Tree and plant regrowth on
previously
unutilised or
damaged soil
Increased food
security for
community members
and crops
Fruit trees provide
sources of food
Tree planting
initiatives are
established
successfully
Fruit and stability
trees planted
Communities have
increased
understanding of
land/ coastal
protection
Communities have
increased
understanding of
land/ crop/ livestock
management
Communities trained in
Landcare, gardening,
livestock and
plantations
Improved animal
management and
increased production of
eggs, poultry meat etc.
All households have
raised vegetable
beds with newly
planted crops
Raised garden
beds built
Livestock is protected
from predation (e.g.
poultry) and contained
to prevent damage to
crops (e.g. pigs)
Livestock
management
techniques
implemented
EXERCISE:
RESULTS DIAGRAMS
• As a group identify one of your Activities, and develop a draft results
diagram for the Activity – using butchers paper and post-its.
• Include a goal
• Note assumptions and risks
• Remember to address cross-cutting issues
2. Developing Results Measurement
Tables
RESULTS MEASUREMENT TABLES
KEY MESSAGES
RMT provides the means to measure Activity achievement (intended
results)
Requirements
• All results in Results Diagram are included in the RMT
• Baseline and time bound targets for each indicator
• Data source/ methodology information stated
Remember
• Use the same unit of measurement for indicator, baseline, target
• Make use of existing partner M&E systems, indicators, etc.
• Use a participatory approach – ownership, buy-in, accountability, etc
• Make use of NZ Aid Programme Headline Results Indicators
RESULTS MEASUREMENT TABLES
KEY MESSAGES
1. Data availability challenges
– We need to be realistic about what’s available – important to focus
limited resources on a concise and meaningful set of data.
2. Challenge of identifying a concise set of indicators
– Need to be stringent – ideally two indicators per result
3. Challenge of collating baseline data
– Often not possible to collect full set of baseline data.
– What data is essential now in design and what can wait to early
implementation?
– Link to problem analysis - what evidence to say problem exists?
INDICATORS
An indicator clearly and neutrally identifies a relevant means to measure
achievement of an output or outcome
• For example:
– number of training courses conducted
– quality of water
– perception of community heath
Indicators for each result need to:
• Provide a balanced view of performance (quantity and quality)
• Draw from existing data sources wherever possible
Where new data needs to be collected – this needs to fit within available
capacity and resource.
TYPES OF INDICATORS
QUALITY AND QUANTITY FOCUSED
Quantity focused indicators
Quality focused indicators
Provide a measure of an amount or
quantity.
Provide a measure of quality – this maybe based
on people’s perceptions or judgements /
assessment (including scientific means) against a
criteria or standard
1. Kilometres of road built,
2. Ratio of women to men in decision
making positions in an organisation
3. Percentage of girls in school
4. Cost per unit of …
1. Community perceptions that health in the
community has improved
2. Teaching practice complies with standards
(assessed by auditor)
3. Community perceptions that trust in the
police has improved
4. New roads comply with international
highway standards
Note quality focus indicators can have quantity targets.
BASELINES
Baselines describe the situation prior to the intervention enabling
us to measure change
• Where it is not already available, baseline may need to be measured
early in implementation- note in M&E workplan.
• The RMT would then be updated to include the baseline data when it
becomes available.
• Good baseline data enables us to set realistic targets
TARGETS
For each result, the target specifies the quantity or quality that is
intended to be achieved by a stated time period.
This gives an understanding of what success, or achieving the
result, looks like.
Targets:
• Are always time bound:
– Year 1, Year 2, Year 3, total
• Can be set by looking at baseline, similar Activities, data trends or by
gaining opinions of partners, and /or stakeholders.
• Are often determined subjectively - it is important to have complete
agreement with partners.
TARGETS
• Targets should be realistic and achievable
• Regular review is required to ensure they remain so
• Hard targets and soft targets – recognising we have greater control over
results at output and short-term outcome level (compared to medium and
long-term results), it is important to treat targets accordingly.
Hard (or specific) targets
should be agreed for outputs
and short-term outcomes
For example:
90% of water samples
150 people trained (80 women and 70 men).
Soft (or directional) targets
are usually more appropriate
for medium and long-term
outcomes
For example:
Improving educational achievement
Decreasing rates of disease
EXAMPLES: INDICATORS, BASELINES AND TARGETS
Result (Output
or Outcome)
Training
completed
Indicators
Baselines
Targets
1. Number of training courses
conducted
2. Number of men and women
trained
1. 18 courses in 2011
Skills and
knowledge in
small business
improved
1. Percentage of trainees
(men/women) that improve
skills/knowledge when tested
before/after training
1. 85% of men and
90% of women in
2012
1. 16 courses by 31 Dec 2013
(14 by 31 Dec 2014)
2. 250 men and 300 women
by 31 Dec 2013 (200 m / 300
w by 31 Dec 2014)
1. 90% of men and 90% of
women by 30 June 2014 (90%
m / 90% w by 30 June 2015)
Operations
manual complete
2. Manual meets quality
standards (as perceived by
manager)
Nil
1. Manual meets standards by
30 June 2013
Improved water
quality
1. Percentage of water samples
that comply with agreed
drinking water standards
2. Number of incidence of
waterborne disease in the
community
1. Percentage of students who
achieve national test levels at
age 10 for reading.
1. 30% in 2009
1. 50% in 2012 (60% in 2013;
70% in 2014)
Improved student
achievement
2. 250 men and 180
women in 2011
2. 18 confirmed
incidence of disease x
in 2012.
1. 70% in 2011.
2. No more than 10 confirmed
incidence of disease x in 2013
(5 by 2014).
1. Increasing percentage
annually (or 80% by 2015)
EXERCISE 2
RESULTS MEASUREMENT TABLE
• Using the results diagram you developed in the last session, write
indicators, baselines, targets for selected results (at output and
outcome level)
• Write up on butcher’s paper
Remember:
• Indicators must be neutral (i.e ‘number of’ rather than description of
change i.e. ‘increase in number of’)
• Balance indicator focus on quality and quantity
3. M&E Workplan
M&E WORK PLAN
This session will cover
• What it [the M&E Workplan] is
• Why we have it
• What to include in it
• A practical exercise
So who’s developed an M&E Workplan before?
M&E WORK PLAN
What is the M&E Workplan?
• Sets out an implementation plan for M&E of an Activity
• Includes the methods, approaches, tools, responsibilities,
timeframes and resources required to support effective
results-based practice over design, implementation and
completion
• Builds on results diagram and RMT
M&E WORK PLAN
Purpose of the M&E Workplan
• To ensure the monitoring, evaluation and reporting
‘system’ functions effectively by identifying key tasks and
approaches
• To assess capacity to collect data, monitor, report and
evaluate, and identify where support may be required
• To ensure results information is effectively used for
learning and decision-making, and is at the forefront of
discussions
M&E WORK PLAN
What to include in your M&E Workplan
Monitoring includes:
• Initial data collection (incl baseline)
• Strengthening or creating monitoring systems, processes and
capacity
• Regular data collection, monitoring visits and reporting
• Regular results discussions (incl annual review of ARF)
M&E WORK PLAN
What to include in your M&E Workplan
Evaluation includes:
• In-depth partner or MFAT-led assessments (incl independent
evaluations) of what is and is not working well, and why
• Some lighter touch than others, but same process and principles
• Formative or process evaluations that occur early in the life of an
Activity and that aim to inform ongoing implementation
• Summative or outcome evaluations that occur late (or after) an
Activity and aim to make a judgement about the value of an Activity
• The use of data gathered through monitoring and through evaluation
methods to answer specific questions
M&E WORK PLAN
Example
M&E Workplan
Tasks
Approach
Timeline
Roles &
Deliverables
responsibilities & reporting
Indicative
costs
Monitoring Tasks
i. Initial data collection (including baseline)
ii. Create/ strengthen systems, processes and capacity
iii. On-going data collection tasks (including methods used for data collection), monitoring, formal visits and reporting
iv. On-going results discussions (include annual review of Activity Results Framework involving MFAT and partners)
Evaluation Tasks
i. Partner-led evaluation
ii. MFAT-led evaluation
Overall M&E Budget
Total Indicative Costs
$
TOTAL BUDGETED
$
M&E Workplan
Tasks
Approach
Timeline
Roles &
responsibilities
Deliverables &
reporting
Indicative costs
Monitoring Tasks
i. Initial data collection (including baseline)
ii. Create/ strengthen systems, processes and capacity
iii. On-going data collection tasks (including methods used for data collection), monitoring, formal visits and reporting
Trainee course
evaluation and
selfassessment
Develop and
implement prepost- and follow
up course
evaluation for
trainees
Trainees
surveyed
before, after
and 6
months
following
module
delivery
Training
partner
develops and
implements
course
evaluation
Annual
reporting
against RMT
$2,500
iv. On-going results discussions (include annual review of Activity Results Framework involving MFAT and partners)
M&E Workplan
Tasks
Approach
Timeline
Roles &
responsibilities
Deliverables &
reporting
Indicative costs
Evaluation Tasks
i. Partner-led evaluation
Mid-term
• Desk based
process
review
evaluation to
• Engagement
assess
with key
relevance: how
stakeholders
relevant is the
in-country
training to the
needs of small
holder
farmers?
ii. MFAT-led evaluation
After 1 year • Project
• Evaluation
• $25,000
implementmanager to
plan by
• Est. 25-30 days
ation
draft TOR &
31/8/15
consultancy
lead day-to- • Workshop to
time & $3,000
day
discuss key
expenses
management
findings by
of evaluation
30/11/15
• In-country
• Final
partners to
evaluation
advise on
report by
TOR and
31/1/16
comment on
deliverables
• External
consultant to
implement
evaluation
M&E WORK PLAN
When to develop and review your M&E
Workplan
• Plan for M&E early: develop the Workplan during the
design stage, providing as much detail as you can
• Check, refine and update: review the M&E Workplan
early in implementation, and regularly (at least every 12
months)
M&E WORK PLAN
Key points
• This is the document that brings the ARF to life
• Each Activity will have unique M&E requirements
• Poor planning and inadequate resources are a common reason
for failure to measure and track results
• Work with your partners to build capacity to measure and monitor
results
• Make it a live tool that is used to actively manage M&E efforts:
develop – refine – review
• In the evaluation section, define purpose, objectives or questions
M&E WORKPLAN
Exercises
1. Monitoring vs evaluation
2. Work together as a group to develop M&E tasks for initial data
collection (baseline) and a mid-term evaluation
See handout for instructions
ACTIVITY RESULTS FRAMEWORKS:
KEY MESSAGES
ACTIVITY RESULTS FRAMEWORKS
KEY MESSAGES
Key Messages
1. Keep simple and practical
• small Activity budget – light ARF approach
• The more complex, the more challenging to implement
2. Use partner systems where possible (people, systems, indicators, data, etc.)
– Support existing systems rather than build competing systems
– Proven to have greater success where we draw on partners approaches
3. Spread of results across the Results Diagram
– More results should be described at output and STO levels vs. MTO and
LTO levels (reflecting focus on accountability/ attribution/ contribution)
ACTIVITY RESULTS FRAMEWORKS
KEY MESSAGES
4. Focus on key results only – this makes the task of monitoring and reporting
more manageable.
– We need to be strict – we can’t monitor every result (resourcing required)
5. Financial resources to support measurement – funding may be needed to
collate and report critical data – or to support capacity development
6. Incorporate cross-cutting issues identified during design
7. M&E Work Plan – essential to support good Activity Management during
implementation.
– What tasks, resources and capacity are needed to provide timely and quality
performance and results information.
ACTIVITY RESULTS FRAMEWORKS
KEY MESSAGES
Remember Activity Results Frameworks are a key tool
(and process) to maximise achievement of results - it’s
the results that count!
More information and guidance available on the New Zealand Aid
Programme website:
http://www.aid.govt.nz/about-aid-programme/how-we-work/tools-andtemplates/activities