Transcript Module 8: Monitoring, evaluation and learning – for increased impact and improvement of the IEA process.

Module 8: Monitoring, evaluation
and learning – for increased
impact and improvement of the
IEA process
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Developing a Monitoring and
Evaluation Plan
• Session 4: Framework, Attributes and
Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Exercise: Previous Experience with
Monitoring and Evaluation (30 minutes)
Discuss with your neighbour (10 minutes)
• When have you been involved in an
evaluation/monitoring process?
• What were some of the keys to success?
• What were some of the challenges?
Discuss in plenary (20 minutes)
• Your stories, key elements of success and key
challenges in plenary.
Monitoring Defined
• Monitoring is a planned, systematic process
that closely follows a course of activities, and
compares what is happening with what is
expected to happen.
• Monitoring the IEA process makes sure the
assessment meets its goals, while working
within the scope of allocated resources (i.e.,
time, financial, human, informational and
technical).
Evaluation Defined
• Evaluation assesses an achievement against preset
criteria. Evaluations can have a variety of purposes,
and follow distinct methodologies (process, outcome,
performance, etc.).
• Evaluation of the IEA process determines the extent to
which achievements (outputs, outcomes and impacts)
are comparable with the originally intended purpose, and
what lessons can be learned for the next
environmental assessment and management cycle.
The evaluation of the process is, first and foremost,
a capacity-development opportunity.
Compare Monitoring and Evaluation
Monitoring
Evaluation
1.
2.
3.
4.
5.
6.
7.
1.
2.
3.
4.
5.
6.
7.
8.
Collecting data
Sense of progress
Present time
Attention to details
Inspires motivation
Occurs continuously
Requires management
skills
8. Asks: What needs to
happen now to achieve
our goal
Assessing data
Sense of achievement
Past / future time
Attention to bigger picture
Inspires creativity
Occurs intermittently
Requires leadership skills
Asks: Have we achieved
our goal? Can we do better
next time?
Learning Defined
• An emotional and/or cognitive
transformation taking place during
information collection and information
processing, bringing about behaviour
change or change in the ability to act
differently.
Key Questions for
Developing an Evaluation Plan
1. What is the purpose of the evaluation?
2. Who will use the evaluation results?
3. Who will do the evaluation?
4. What evaluation framework is practical?
5. What needs to be monitored and evaluated?
6. What are the steps to develop a selfassessment matrix?
7. How can I use the evaluation to enhance a
learning culture that keeps improving my IEA
process?
Stages of National IEA Process
GEO as a Reporting Process
Plan - Do - Review Model
Why Monitoring and Evaluation?
• Lessons learned from IEAs help improve policy
making processes, policies and state of
environment.
• The IEA process builds capacity for periodic
policy revision through monitoring and
evaluation.
• Capacity is built among individuals and
organizations.
Conceptual Understanding of the National IEA Process,
With Links to Ecosystem Health and Human Well-Being
Example: Ozone-related
Treaties
• Continuous improvement of ozone-related
international treaties was due to researchbased evidence.
• This involved close monitoring of ozone
concentrations in the stratosphere, simulation of
possible scenarios, and revision of
international agreements.
• Monitoring took into account processes,
products and impacts.
Example: SoE Monitoring in India
• Capacity building and preparation of
the SoE report are the main activities of
the SoE Monitoring Program.
• National Host Institutions (NHI) and State
Host Institutions (SHI) lead the program.
• A two-stage monitoring and evaluation
program is in place.
SoE Monitoring in India
• Stage 1:
– NHI performance evaluation (done by the
Ministry) is indicated by the number of states
that have progressed in their SoE programs
and published final SoE reports.
• Stage 2:
– SHI performance as measured through the
number of SoE reports prepared.
– The success of NHIs depend on SHI success,
and SHIs depend on NHIs for funding.
Coupling Science and Policy
• Communication between the cycle of
scientific data collection and processing,
and the cycle of policy making is
facilitated by a GEO-type process.
• It can take decades to develop the right
set of policies after the first evidence of an
environmental issue.
Example: Lake Balaton, Hungary
•
The first scientific warnings of eutrophication
in were published in the early 1970s.
•
The process of policy response included:
1.
2.
3.
4.
5.
6.
unusual scientific observation;
scientific debate;
acceptance by the policy-makers;
policy development;
policy enforcement; and
a long time period required before the cumulative
impact was apparent.
Example: United Kingdom
• The UK government has been publishing
environmental indicators since 1996.
• The UK sustainable development strategy in
1999 significantly incorporated indicators.
• Headline indicators are used to communicate
progress.
• An interdepartmental management board
coordinates indicator development.
• Indicators were more successful in
communicating progress than policy review.
Example: Ministry of Environment,
Hungary
• Over a decade of published environmental and
sustainable development indicators.
• Trends are distributed to a wide audience of
technical experts, policy-makers and the general
public.
• Regularly updated information has been a
factor in the reports’ policy impacts.
Discussion: Additional Examples
• Do you know of other examples of how
research-based evidence can guide policy
making and lead to improvements?
• What are some examples in your country
at the national level?
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Developing a Monitoring and
Evaluation Plan
• Session 4: Framework, Attributes and Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Foundation Steps for Effective
Monitoring and Evaluation
1. Identifying your purpose.
2. Identifying your primary users.
3. Deciding whether internal or external
evaluators best serve your purpose.
1. Types of Purpose
1. Judgment: Sets clear criteria and
standards to judge performance. Can
increase credibility of the GEO-process.
2. Improvement: Open-ended evaluation
that measures change over time. Often
applied to cyclical activities, like the GEO
Process.
3. Knowledge-creation: Identifies
emerging knowledge and insights.
Increases the saliency of the GEO
process.
2. Users of a GEO-type Evaluation
People who:
• can revise the GEO process: have the
mandate, knowledge and skills; and
• want to revise the GEO process: have a
vested interest in influencing the design
and implementation of the GEO process.
Primary Users
The primary users of the evaluation may include:
• IEA core team (includes policy-makers);
• policy and decision-makers who are primary
users of the IEA; and
• the evaluation team (internal and/or external).
Exercise: Purpose for Monitoring
and Evaluation
In small groups, discuss the following:
1. Why do you need a plan for monitoring and
evaluation of your IEA process and impact at
the beginning of your planning process?
2. Why is improvement-oriented evaluation
relevant to your IEA process?
3. Using the information above, what would you
state as the purpose of your evaluation?
Exercise: Continued
4. Who are potential users?
– What are their interests in using the
evaluation results?
– Do they have a mandate to revise the GEO?
Group discussion (30 minutes), plenary
presentations (15 minutes)
3. Evaluators
Evaluators may include:
• A small internal evaluation task force (including
the IEA core team, which is recommended).
• External evaluators (consultants and internal
evaluators of another IEA).
• A combination of internal and external parties.
Plenary Discussion: Identifying
Evaluators
(40 minutes)
In small groups, for 20 minutes,
• Brainstorm key criteria for selecting
evaluators; and
• Identify types of evaluators and possible
names.
In plenary, discuss your results (20
minutes).
Plenary Discussion Continued:
What are some other parameters to
consider for a monitoring and evaluation
plan to be a success in my context?
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Steps to Developing a Monitoring and
Evaluation Plan
• Session 4: Framework, Attributes and Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Corresponding Attributes of
Effective Evaluations
Conceptual Understanding of the National IEA Process,
With Links to Ecosystem Health and Human Well-Being
Framework for Monitoring and Evaluating
the National IEA Process
Monitoring the timely Completion of Key
Activities and Outputs
Possible Measures for Effective
Knowledge Management
Discussion: Effective Knowledge
Management
(30 minutes)
In small groups, consider the following (15
minutes):
1. Can you think of additional important
measures of effective knowledge
management?
2. What do you think are reasonable targets for
the measures you identified?
Possible Measures for Effective
Opportunity Management
Discussion: Effective Opportunity
Management
(30 minutes)
In small groups, consider the following (15
minutes):
1. Can you think of important measures of
effective opportunity management not
included in the table above?
2. What do you think are reasonable targets
for the measures you identified?
Measures for Effective Relationship
Management
Identifying relationships based on behaviour
Behaviour Type
Behaviour Exhibited
1. Receiving (individual,
organization)
2. Seeking (individual,
organization)
3. Acting Upon (individual,
organization, institution)
1. Reports, e-mails,
listserv
2. Target users seek new
information.
3. Technical expertise is
sought to revise
policies.
4. Specific needs, such as
monitoring data for the
next IEA cycle.
4. Demanding (individual,
organization, institution)
Possible Measures for Effective
Relationship Management
Discussion: Measuring
Relationship Management
(30 minutes)
In small groups, consider the following (15
minutes):
•
Can you think of any other important measures
of effective relationship management?
2. What reasonable targets would you
recommend for various measures?
Measures for Improvements in
Policy and Policy Making
• Measurement should relate to the change
statement you identified in your impact
strategy.
• Measurement should also track other
observed improvements in policies and
policy processes.
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Steps to Developing a Monitoring and
Evaluation Plan
• Session 4: Framework, Attributes and Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Planning a Self Assessment
Step 1
Identify major issues and monitoring questions,
and develop specific measures.
Step 2
Identify sources of data and data collection
methods.
Step 3
Set priorities and frequency of monitoring
Step 1. Identify Major Issues and Monitoring
Questions, and Develop Specific Measures
Outcomes
• Your Change Statement
• Effective Relationship Management
Activities and Outputs
• Effective Knowledge Management
• Effective Opportunity Management
• Timely completion of activities and outputs
Outcome-based Measures: Possible Organization of Your SelfAssessment Matrix
Key Issues/Questions
Specific Measures
and Target
Your Change Statement
Have the desired improvements
in policies and policy processes
that you identified in your impact
strategy been realized?
What other improvements in
policies and policy processes
have you observed during and
following your national IEA
process?
Effective Relationship
See Table 4 for
Management
guidance
What changes in the thinking
and actions of policy-makers and
decision-makers (and other
important relationships) have you
observed?
Data Source
Data Collection
Method
Activity and Output-based Measures: Possible Organization of Your
Self-Assessment Matrix
Stage of the
IEA
Stage 1
Inception
Key Issues /
Questions
Specific Measures and
Targets
Timely completion of activities
and outputs
See Table 7 for guidance
Effective Knowledge and
Opportunity Management
See Table 5 and 6 for guidance
Stage 2
Institutional
Setup
Timely completion of activities
and outputs
Stage 3
Scoping and
Design
Timely completion of activities
and outputs
Stage 4
Planning
Timely completion of activities
and outputs
Effective Knowledge and
Opportunity Management
Effective Knowledge and
Opportunity Management
Effective Knowledge and
Opportunity Management
Stage 5
Implementation
of IEA
Timely completion of activities
and outputs
Stage 6
Communication
and Outreach
Timely completion of activities
and outputs
Stage 7
Evaluation
Timely completion of activities
and outputs
Effective Knowledge and
Opportunity Management
Effective Knowledge and
Opportunity Management
Effective Knowledge and
Opportunity Management
Data
Source
Data Collection
Method
Step 2: Identify Sources of Data and Data
Collection Methods
Collecting Data for Monitoring
Effective Relationship Management
• Requires that changes in behavior be
identified and mapped:
– these incremental changes will lead towards
the decisions or changes you are seeking.
• Can be a time intensive process, so it is
important to set up simple ways to monitor
your strategy against those measures.
• Can set up a small contacts database with
a journaling function (see next slide).
Collecting Data for Monitoring
Effective Relationship Management
Step 3: Set priorities and Frequency of
Monitoring and Evaluation
• Ensure that critical indicators are
monitored.
• Establish the frequency of monitoring for
each indicator.
Process indicators are needed throughout the
IEA.
Progress indicators are needed less
frequently, but long after the IEA is complete.
Exercise: Preparing a SelfAssessment
1.
2.
Form working groups of 4–5 persons.
Complete steps 1–3 for preparing a self-assessment
matrix.
–
–
–
3.
Step 1: Identify major issues and monitoring questions, and
develop specific measures. For activity and output-based
measures, assign different stages to different groups and
compile in plenary.
Step 2: Identify sources of data and data collection methods.
Step 3: Set priorities and frequency of monitoring
Plenary Session to compile the results and discuss
them.
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Steps to Developing a Monitoring and
Evaluation Plan
• Session 4: Framework, Attributes and Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Improvement Opportunities
• Thinking of an IEA as a capacity-building
process significantly increases its impact.
• The more that monitoring and evaluation is
treated as an organizational learning
opportunity (versus a value judgment),
the more effectively the IEA supports
improvements in policy making.
Exercise: Learning
• In pairs, for 10 minutes:
Think of the last time you learned
something new. What was it, how did you
learn it and what did you do differently as
a result?
What is learning?
• Learning is more than knowledge creation;
• Learning is demonstrated by behaviour
change; and
• Information processing, in addition to
information collection, is of paramount
importance.
Learning Cycle
Exercise: Learning
A.
Write what comes to mind based on the
following four questions (5 minutes):
1. What did you hear during the GEO module training
course (e.g., Stage 1) that you already knew?
2. What new information and insight did you gain?
3. How are you going to use this new insight?
4. How else and when could you use this new
information? How could you improve policy making
with this new insight?
Exercise: Learning
• Discuss your findings with your neighbors
(5 minutes).
• In plenary, discuss the insights you have
gained from this exercise.
– How did the group discussion help you to
recognize improvement opportunities for the
national IEA process and ways to have
impact, such as causing changes in policy
making? (15 minutes.)
A Few Prerequisites for
Learning
• motivation, which often is the urgency to solve a
problem, or act with the support of new knowledge;
• trust to discuss values, assumptions and ideas without
repercussions;
• mandate and opportunity to apply the new knowledge;
and
• shared understanding of the importance of learning
(not only what to learn but also how to learn).
Learning Cycle with GEOSpecific Questions
Organizational Learning Cycle
The Role of Learning in Improving
the IEA
• Formulate your change statement (Module 3).
• Identify measures for your change statement and
other supporting measures for key outcomes and
activities/outputs (Module 8, Tables 8 and 9)
• Examine performance against making the desired
changes and summarize results
• Formulate recommendations and lessons learned.
• Integrate (feedback) recommendations and lessons
learned to improve the next planning cycle and
other decisions.
How Can We Use Learning
Opportunities? – The Monitoring Meeting
• Learning opportunities naturally present themselves at
the beginning and end of each IEA stage and each IEA
cycle. These are the times when you need to reflect and
articulate lessons learned to improve the next course of
action.
• Given the limited time available, we suggest that your
core IEA team organize regular, mid-stage and/or stageend monitoring and evaluation meetings to serve two
purposes:
– 1. Monitor progress toward and capture lessons learned to
improve the next IEA stage and the next IEA cycle.
– 2. Cultivate a learning, improvement-oriented approach
throughout the whole IEA process.
Questions for Monitoring Meetings
1. What were the most revealing lessons for you in this
stage?
2. How can you use this knowledge, attitude to improve
the next stages process, outputs or targeted changes?
3. Have you achieved what we planned for in this stage? If
yes, what factors helped? If not, what factors hindered.
How can we reach the desired goal?
4. Are there any unexpected results, emerging
phenomena, trends or questions you want to discuss?
5. How did this stage contribute to the perception of
saliency, legitimacy and credibility of the national IEA
process?
Suggestions for General and Stage-Specific Questions and Exercises for
Monitoring Progress and Promoting Learning at Specific GEO Stages
General questions
for monitoring
meetings
(1) What were the most
revealing lessons for you
in this stage?
(2) How can you use this
knowledge, attitude to
improve the next stages,
process, outputs or
targeted changes?
(3) Have you achieved
what we planned for in
this stage? If yes, what
factors helped? If not,
what factors hindered?
How can you reach the
desired goal?
(4) Are there any
unexpected results,
emerging phenomena,
trends or questions you
want to discuss?
(5) How did this stage
contribute to the
perception of saliency,
legitimacy and credibility
of the national IEA
process?
GEO
Stage
Major activity
Prevailing learning
condition
Stage-specific activities
for monitoring meetings
(see Box 7)
1
Inception
2
Institutional
setup
Mandate
Articulate, confirm
mandate
3
Scoping and
design
Trust
Nurture trust
Force-field analysis
4
Planning
Information
processing
Create opportunities for
collective information
processing
5
Implementation
Shared
understanding of
learning
Remind participants of
how they learn best
Keep, start, stop doing
6
Communication
Motivation
Revisit motivation
Carousel discussion
7
Evaluation
Application
Focus on lessons learned
for application
Samoan circle
Exercise: Designing a
monitoring meeting
• In groups of 4 or 5, prepare an agenda for halfday monitoring meeting for any stage of your
IEA process (30 min). Address the following:
– What specific progress measures will be part of your
meeting?
– What learning opportunities might be identified for the
stage of the IEA process you have selected?
• In plenary, discuss the results from two groups
(15 min).