Transcript Training Modules: Monitoring and Evaluation of Population, Health and Environment (PHE) Programs MEASURE Evaluation Developed by Caryl Feldacker.

Training Modules:
Monitoring and Evaluation of
Population, Health and Environment
(PHE) Programs
MEASURE Evaluation
Developed by Caryl Feldacker.
Agenda
Training Modules
1. Introduction to PHE
2. Frameworks, Part 1 – Conceptual
3. Introduction to M&E
4. Frameworks, Part 2 – Logic models
5. Indicators
6. Frameworks, Part 3 – Indicator matrixes
7. Assessment and evaluation design
Starting on the Same Page…
First, everyone should complete this free, online,
two-hour course:
http://www.cpc.unc.edu/measure/training/mentor/me_fundamentals
Then, all participants should be able to:





Identify the basic purposes and scope of M&E
Identify the basic criteria for the selection of indicators
Describe how indicators are linked to frameworks
Identify types of data sources
Describe how information can be used for decision
making
Module 1: Training Overview
5
Training Programmatic Focus
This course is geared for programs that:
 Involve active partnerships between communities, nongovernment organizations, or government entities;
 Promote an integrated, community-based approach to
problem solving;
 Operate in limited resource settings (smaller budgets);
 Collaborate between population, health and environment
(PHE) sectors by implementing activities in multiple
programmatic areas including, but not limited to: family
planning, education, income generation, natural resource
management, gender, sanitation, or agriculture.
Training Target Audience
This training course and its modules are most useful for:
 Organizations that are familiar with PHE programs and
the PHE approach
 Staff who are involved with monitoring and evaluation
in an integrated programming context
 Managers and staff who design, plan, and implement
PHE programs
 Consultants and trainers of technical staff in integrated
PHE programs
Training Objectives
 To increase understanding of the basic concepts and
practical approaches for performance monitoring and
evaluation for integrated population, health and
environment programs
 To learn state-of-the-art tools and techniques used
specifically to monitor and evaluate population, health
and environment programs
 To gain hands-on experience in designing
components of monitoring and evaluation plans for
population, health and environment programs
Training Learning Methods
 Lectures
 In-class exercises
 Small group work
 Group presentations
LAYING THE FOUNDATION
What is PHE?
DEFINITION:
 A community development model that links population,
health and environment (PHE) sectors and supports
cross-sectoral collaboration and coordination, particularly
in biodiversity-rich areas (BALANCED:
http://balanced.crc.uri.edu/)
 Within one or more communities, these programs actively
link environmental activities with activities that seek to
improve reproductive health, always including, but not
limited to, provision of family planning services (Robert
Engelman, Worldwatch Institute)
What Does PHE Stand for?
P = Reproductive health, family planning
H = Water, sanitation, malaria, vaccines, HIV/AIDS, MCH
E = Natural resource management, biodiversity conservation,
fisheries management, eco-agriculture
Amenable to other sectors
+E = Education (non-formal & formal)
Energy (fuel-efficient smokeless stoves, biogas)
Eco-tourism, sustainable tourism development
Enterprise/livelihood development/integrated development
Why Integrate?
Social and environmental factors linked to food insecurity in coastal Philippines
13
Reasons for Integration








Integrated lives
Address root causes
Enhance relationship
Access different communities
Economic efficiencies
Build on existing frameworks
Broader community participation
Value added from integration
Synergy from Combination
 Social systems & ecosystems are interrelated and linked
 Whatever effects one system, affects the other
 Small improvements in either system reinforce each
other
 To turn around both systems from degradation to health
Types of Integration
 Parallel: projects conducted in a single area without
coordination among organizations
 Coordinated: projects conducted in the same area with
some level of coordination
 Cross-sectoral: projects involving several sectors that
are conceptually linked
 Integrated: projects involving several sectors that are
conceptually and operationally linked
Achieving Integration
 Conceptual Integration
 Field Integration
 Community Involvement
 PHE Partnerships
 Mobilizing Resources
 Planning Ahead for Sustainability
“Valuing” Gender as Integral to PHE
Consideration of gender begins with attention to
two central questions:
 How will gender relations (relationships between men
and women) affect the achievement of sustainable
results?
 How will my program affect, influence, or change the
relative status of men and women? (Will it reduce
differences in men’s and women’s status? Will it help
create equality between men and women?)
Module 2
Frameworks, Part 1
Conceptual frameworks
Module Outline
 Introduction to frameworks
1. Conceptual frameworks
• Including goals and objectives
2. Logic models
3. Indicator matrixes
 Exercises developing frameworks
 Linking frameworks and M&E planning
 Work on group project
Brief Summary of Three Frameworks
Type of
Brief Description
Framework
Use for Program
Management
Basis for Monitoring
and Evaluation of
PHE
Conceptual
Determine which
factors the program
will influence and
how
Helps explain and
promote the multisector influences and
integrated approach
Logic model Logically links
inputs, processes,
outputs, and
outcomes of
planned activities
Shows the causal
relationship between
inputs and reaching
the objectives
Shows intended
progress of activities
at all stages of the
program – aids
indicator selection
Indicator
matrix
Shows the causal
relationship and
steps between
activities and
meeting objectives
Guides monitoring and
aids assessment by
stating the information
you will collect, how,
when, where and why.
Shows the
interactions of
various factors that
operate in your
program
environment
Logically links
program goal,
objectives, outputs,
outcomes, and
activities
Brief Summary of Three Frameworks
Type of
Brief Description
Framework
Use for Program
Management
Basis for Monitoring
and Evaluation of
PHE
Conceptual
Determine which
factors the program
will influence and
how
Helps explain and
promote the multisector influences and
integrated approach
Logic model Logically links
inputs, processes,
outputs, and
outcomes of
planned activities
Shows the causal
relationship between
inputs and reaching
the objectives
Shows intended
progress of activities
at all stages of the
program – aids
indicator selection
Indicator
matrix
Shows the causal
relationship and
steps between
activities and
meeting objectives
Guides monitoring and
aids assessment by
stating the information
you will collect, how,
when, where, and why.
Shows the
interactions of
various factors that
operate in your
program
environment
Logically links
program goal,
objectives, outputs,
outcomes, and
activities
Module Learning Objectives
At the end of the session, participants will be able to:
1. Explain why conceptual frameworks are helpful for PHE
programs
2. Draw a conceptual framework for a specific program
3. Design goals and objectives for specific intervention
programs to match the conceptual framework
Conceptual Frameworks
Definition: Diagram that identifies and illustrates the
relationships between all relevant systemic, organizational,
individual, or other salient factors that may influence
program/project operation and the successful achievement
of program or project goals.
Purpose for M&E:
 To show where the program fits into wider context
 To clarify assumptions about causal relationships
 To suggest causal pathways
Why Are Conceptual Frameworks Useful
for M&E of PHE?
 To demonstrate knowledge or understanding of the
problematic context – situation analysis.
 To show the linkages between PHE program areas,
demonstrating the synergies between integrated problems
and integrated solutions.
 To draw or explain relationships between the problems that a
program/project addresses and the external context
(environmental factors), demonstrating ways that program
activities may affect knowledge, attitudes, skills, and
behaviors of the target population.
 To help identify what evaluation information might be useful to
measure program success.
Causes of Malnutrition in Society
Child malnutrition,
death and disability
Inadequate
dietary intake
Insufficient
access to food
Outcomes
Immediate
causes –
individual
level
Disease
Inadequate
maternal & child
care practices
Poor water/sanitation
& inadequate health
services
Quantity & quality of actual
resources – human, economic and
organizational – and the way they
are controlled
Underlying
causes at
household/
family level
Macro-level
causes at
societal
level
Potential resources: environment,
technology, people
Source: UNICEF, State of the World’s Children, 1998.
Example of a conceptual model for the IPOPCORM
project in the Philippines
Result Chain 1:
FP intervention to achieve a conservation outcome
Community
Based
Distribution
(CBD)
of
Contraceptives
Increase
Access
to FP
Info and
Products
Decrease
Fertility
Fewer
Children
Couples
Have
More
Time
for CRM
Activities
Habitats
Protected
Intervention: Community based distribution (CBD) of
contraceptives in which community residents are trained and
equipped to promote and distribute contraceptives using social
marketing approaches.
Improved
Conservation
of the
Marine
Environment
Activity: Making a Model
29
Brief Presentations & Discussion
Presentation of the conceptual frameworks ~ 5-10 minutes
per group
 Each group presents their framework;
 Feedback from other participants?
As a large group, let’s discuss:
 What is different about conceptual frameworks for
integrated development programs?
 How can you make the linkages between diverse
sectors explicit in your conceptual framework?
Integrated Goal and Objectives
Philippines: IPOPCORM Program
• Community-based
Family Planning
• Policy Advocacy &
Community IEC
• Community-based Coastal
Resource Management
• Economic Development
(environmentally-friendly)
Goals
Goal: a broad statement of a
desired, long-term outcome of the
program.
For PHE, it should reflect an
integrated outcome.
Source: GAP 2003
Integrated Goals: IPOPCORM Example
To improve food security and quality of life in coastal
communities while maintaining the biodiversity and
productivity of life-sustaining marine ecosystems
34
Hints for Writing Quality Goals
GOALS:

Present the desired INTEGRATED outcomes, accomplishments, result or
purpose sought (not the process).

Capture broad changes in conditions, answering the “so what” question.

Often goals reflect behavior, attitude, or economic change and show how our
activities contribute toward a larger development impact.

Usually goals reflect a result achieved in an intermediate time period (2-5
years).
(I want) TO
(action verb)
of/among ____________
(what)
(whom)
What behavior/practice, attitude, economic, knowledge, capacity, or access
change do you seek?
Goals Should Be Specific and Tailored!
No!!
Yes!!
To improve health
To improve the health practices of
people living in targeted pastoralist
communities.
To reduce conflict
To increase the effective management
of critical conflicts among local
communities in the Bungoma
Biosphere Reserve.
To increase standards of
living
To increase the price of livestock sold
on the local market by women in the
National Park buffer zone.
Objectives
Objectives: statements of desired, specific, realistic, and
measurable program results. They should be SMART!!
 Specific: identifies concrete events or actions that will take
place.
 Measurable: quantifies the amount of resources, activity,
or change to be expended and achieved.
 Appropriate: logically relates to the overall problem
statement and desired effects of the program.
 Realistic: Provides a realistic dimension that can be
achieved with the available resources and plans for
implementation.
 Time-based: specifies a time within which the objective
will be achieved.
Source: GAP 2003
Hints for Writing Quality Objectives
Remember objectives…
 Are linked directly to the goals and the intended outputs
(information, products, processes, services, or results) you
hope to produce to reach the goal.
 Reflect what you hope to produce by undertaking a specific
activity. For example: You decide you want to train people
(the activity), then the objective will relate to the knowledge
you increase.
 Usually reflect a result achieved in a relatively short time
period (0-2 years).
To Write an Objective, Fill In the Blanks:
TO __________ (action/direction of change)
the ________ (what knowledge, attitude, practices will be changed)
In _______ (targeted population or area of change)
by _______ (the expected amount of change in number or percent)
as measured by ______(how will you measure the change)
by _______ (time frame)
Examples
1. To increase the number of children in primary school in Addis
Ababa by 25% (as measured by school records) by December
30, 2010.
2. To provide four workshops on peace building methods to 100
community members as measured by field staff workshop reports
by December 30, 2008.
Integrated Objectives: IPOPCORM Example
Objectives (short term outcomes)
• By 2004, improve RH outcomes among people living in coastal
areas.
• By 2006, build community capacity to implement coastal
resource management including alternative livelihood schemes.
• By 2007, increase public and policymakers’ awareness and
support for integrated PHE approaches.
Are the objectives SMART?
Specific, Measurable, Appropriate, Realistic, and Time Limited?
40
Goals and Objectives
 Goal: To improve access to family planning, community
health, and natural resource management through integrated
development approaches in target communities.
 Objective 1: Reduce barriers for access to voluntary FP and
RH services in target communities.
 Objective 2: Improve family health in priority biodiversity
areas in target communities.
 Objective 3: Improve community management of natural
resources and habitat conservation in target communities.
 Objective 4: Document and promote sustainable and
successful PHE approaches in target communities through
improved monitoring and evaluation.
Goals and Objectives: Activity
Child malnutrition,
death, and disability
Inadequate
dietary intake
Insufficient
access to food
Outcomes
Disease
Inadequate
maternal & child
care practices
Poor
water/sanitation &
inadequate health
services
Quantity & quality of actual
resources – human, economic,
and organizational – and the way
they are controlled
Potential resources:
environment, technology,
people
Immediate
causes
Underlying
causes at
household/
family level
Basic
causes at
societal
level
Source: UNICEF, State of the World’s Children, 1998.
Small Group Activity
In your groups of 4-6, draw up a simple conceptual framework of
your program
 This will be the basis of your work for the training
 You will present this framework and talk the group through it
(Re) Write your goals and objectives
 The goal must reflect integration of at least two sectors
 At least one population/health objective
 At least one environment/conservation/non-health objective
Do your goals and objectives match your model?
Are your goals and objectives written effectively? SMART?
Conceptual Framework, Goal, and
Objectives Group Report
 Each group should present their conceptual framework,
goal, and objectives for the group.
 Walk the group through your conceptual framework,
showing the links between problems/issues.
 Present your goal and objectives.
 Did you all manage to integrate PH&E components into
your conceptual framework, goals, and objectives?
Module 3
Introduction to M&E
Objectives of this Module
1. To define monitoring and evaluation
2. To review elements of an M&E plan and necessary
steps toward constructing the M&E plan
3. To discuss implementation of M&E plans
4. To discuss specific considerations for M&E in the
small-scale, integrated development programming
context
5. To introduce “A Guide for Monitoring and Evaluating
Population-Health-Environment Programs”
Monitoring and Evaluation is an
Essential Process to:
1. Determine if the program is on track in carrying out
planned activities
2. Decide whether the program is having desired
outcomes in all sectors (health, population,
environment, gender, livelihoods, etc.)
3. Help make informed decisions about needed
corrections, new services, use of program resources,
etc.
47
Monitoring is the routine
tracking of program activities
to help program and project
managers measure progress.
 Are we completing
activities according to
plan?
 What are the costs?
Evaluation is a specific study
that helps determine program
achievement. A well-planned,
high-quality evaluation can help
answer:
 How well was the project
implemented?
 Were the desired changes
achieved?
 Who is using our services?  If the change was achieved,
to what extent can it be
attributed to the project?
Intervention
Outcome
Why Do M&E in PHE programs?
1. To collect information to make informed, evidencebased decisions.
2. To make mid-course adjustments and refine project
activities.
3. To demonstrate progress and explain unique PHErelated challenges to stakeholders, funders, & partners.
4. To create records of past and present performance for
future assessment and institutional memory.
5. To demonstrate the advantages of working across
sectors rather than working in a single sector, showing
the value of our integrated approach!
Planning for M&E with an M&E Plan
 Links information from various data sources to decisions
that will improve programs.
 States how the program will measure achievements
(ensure accountability).
 Documents consensus (encourage transparency and
responsibility).
 Guides M&E implementation (standardization and
coordination).
 Preserves institutional memory.
 Measures success.
The M&E plan needs to be adjusted when a program is
modified.
Functions of an M&E Plan: by Programmatic
Level
Community:
 Determine participation levels within all sectors
 Supervise community-based workers
Facility:
 Facilitate cooperation with local providers/partners
 Manage logistics among partner organizations
District:
 Request resources/encourage collaboration
National:
 Request resources/advocate for policy changes
When to Plan for M&E
The initial project activity plan should include routine
monitoring and evaluation.
While planning:
 Identify potential, existing data sources and tools.
 Create appropriate forms, tools, and guidelines to
collect information during implementation.
 For integrated development programs, plan with
partners for shared M&E responsibilities before you
begin!
Considerations for Implementing the
M&E Plan in Integrated Programs
 Utility
 Feasibility
 Propriety
 Accuracy
Guidance for M&E for PHE
What’s in the Book? Why Use It?
 Overview of PHE, its history, and its rationale
 Overview of M&E
 Hints on data quality and collection
 Steps for developing a M&E plan
 A set of standard indicators for PH&E sectors, including
integration and value added indicators
Guide to Indicators
1. Population Indicators
2. Health Indicators
3. Environment Indicators
4. Integration Indicators
5. Value-Added Indicators
CHALLENGES
FOR M&E FOR PHE
How Do We Demonstrate “Integration?”
 Show partnerships and collaborative efforts between
sectors
 Highlight the diverse PHE messages and services
provided to communities
 Demonstrate community buy-in to the integrated
approach
 Promote the sustainability of PHE
 Indicate the cost-effectiveness of cooperative efforts
What Do We Mean by “Value Added”?
The value added of integrating PHE, instead of parallel PH-&E efforts, include our programs’ potential to:
 Achieve results outside of the initial program plan,
including positive effects on, for example, gender,
governance, livelihoods, and youth
 Create broader community-level changes than single
sectors programs
 Provide more bang-for-the-buck for donors than more
traditional interventions by saving costs and using
funds more effectively
Special Considerations for M&E for PHE
1. Multi-sector measurements
2. Limited budgets
3. Need for lessons learned/best practices in M&E
4. Use of both primary and secondary data sources
PHE Considerations Continued…
5. Collaboration/cooperation between organizations
6. Demonstrating integration
 How do we show others how we work across
sectors?
7. Issues of “value added”
 How do we demonstrate that our programs are
more than the sum of their parts?
Overcoming Challenges of M&E for PHE
The completion of this training and its modules address
these challenges:
 Measurement across sectors
 Working on a limited budget
 Alternative assessment techniques
 Determining the quality of secondary data
 Measurement of both integration and value added
indicators
 Using standard indicators to measure results
Special M&E Considerations: Stakeholders
What is a stakeholder?
Any person or group with a particular interest in a
policy or program
In PHE programs, stakeholders may be:
 Government Program Managers
Service Providers
 Policy Makers
Journalists/Media
 Private Sector
Donors
 Non-Governmental Organizations Beneficiaries
 Community-based Workers
Who else?
Special M&E Considerations:
Decision Making
In PHE programs, our complex programs require
collaborative decision making in three distinct categories:
1. Advocacy and policy formation (community, district,
etc.);
2. Program design and planning; and
3. Program management and operations.
Special M&E Considerations: Data Use
To make decisions well, all decision makers & stakeholders
need to understand:
 How the programs work, including how they collect
information.
 What are the specific reporting and documentation
procedures used by all partners?
 Among partners and staff, who will analyze the data that is
collected?
 Who needs what specific information?
 How will it be disseminated?
Factors Other Than Evidence-based
Information Influence Decisions
 Power relationships
 Timing
 Local culture surrounding
decision making
 Competing priorities
 PHE partners
 Public opinion
 Other information sources
 Political ideology
 Community needs
Further Readings/Tools on Decision Making,
Information Use, and Stakeholder Analysis
1. Decision Calendar:
 A tool to help explicitly and systematically align and prioritize decisions with
appropriate data resources.
2. Assessment of Data Use Constraints
 A rapid assessment tool designed to identify barriers and constraints,
including those at the organizational, technical, and individual levels.
3. Information Use Mapping Tool
 A tool to help assess and identify opportunities for improved data use and
feedback mechanisms for stakeholders across different levels.
4. Stakeholder Engagement Tool
 A tool to help ensure that the appropriate stakeholders in decision
processes have been identified and involved and identifies key areas of
support for scale up.
Tools are available at MEASURE Evaluation, search term: DDIU:
http://www.cpc.unc.edu/measure/tools/tools-by-title
ACTIVITY: Starting Group Projects
 Return to your groups of 4-6 for the project
 Discuss the program that you will be working on in the
training, including review/creation of:
• Conceptual framework, goals, objectives
• Again, if you don’t have an existing program, you may
design an example program
 Write (or develop) an abridged program plan/description to
use as a basis for your M&E work
• Include at least 10 key activities to reach your
goal/objectives (must include P,H, & E activities)
• Project timeline – how long will your program be funded?
• Anticipated partners
• Key stakeholders/decision makers
Module 4
Frameworks, Part II
Logic Models
Brief Summary of Three Frameworks
Type of
Brief Description
Framework
Use for Program
Management
Basis for Monitoring
and Evaluation of
PHE
Conceptual
Shows the
interactions of
various factors that
operate in your
program environment
Determine which
factors the program
will influence and how
Helps explain and
promote the multisector influences and
integrated approach
Logic model
Logically links inputs,
processes, outputs,
and outcomes of
planned activities
Shows the causal
relationship between
inputs and reaching
the objectives
Shows intended
progress of activities at
all stages of the
program – aids indicator
selection
Indicator
matrix
Logically links
program goal,
objectives, outputs,
outcomes, and
activities
Shows the causal
relationship and steps
between activities and
meeting objectives
Guides monitoring and
aids assessment by
stating the information
you will collect, how,
when, where, and why
Logic Models
Definition:
 Diagrams that identify and illustrate the linear relationships
flowing from program inputs, processes, outputs, and
outcomes.
 Inputs or resources affect processes or activities which
produce immediate results or outputs, ultimately leading to
longer term or broader results, or outcomes.
Purpose:
 Provides a streamlined interpretation of planned use of
resources and desired ends
 Clarifies project/program assumptions about linear
relationships between key factors relevant to desired ends
 Other terms used: M&E Frameworks, indicator matrixes
Key Terms:
1. Inputs: human, financial, social, political, or other
resources needed to begin and complete the program.
2. Processes: activities which produce immediate results
– what will you be doing?
3. Outputs: immediate results from your processes – what
are the results of your activities?
4. Outcomes: longer term or intermediate results – the
progress towards achieving your objectives.
5. Impact: long-term changes necessary to sustain your
goal.
Logic Model for Integrated Development
Logic Model: Training Activity
INPUT
Times and
skills to
develop new
agricultural
technology
training
curriculum
74
PROCESS
OUTPUT
OUTCOME
IMPACT
Conduct
training
events
Local farmers
trained in new
techniques
Increase in
Increased food
agricultural
security in
production
target
from those who
population
adopt new
technologies
Logic Model: Family Planning Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Educate men
and women
about the
advantages of
modern
method use
• Sessions
held in
community
about family
planning
methods
1. Increased
access to
contraceptive
methods
• Increased
contraceptive
prevalence
• Demand for
FP in the
community
• Distribute FP
methods in
the community
• Train program
staff in
providing FP
information
and methods
• Increased
interest in FP
• FP methods
distributed in
communities
• Clinic staff
trained in FP
method
counseling
2. Increased
access to FP
counseling
3. Increased
number of new
users of
modern
methods
4. Increased male
participation in
FP decisions
Logic Model: Diarrhea Prevention Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Train park
guards in
diarrheal
disease and
use of oral
rehydration salts
• (ORS)
packets
distributed
1. Reduction in
acute diarrhea
cases among
children
• Reduced
morbidity
/mortality
from diarrheal
disease
among
children
• Demand for
disease
intervention
in
community
• Train local
community
members in
safe water
storage
• Form natural
resource
management
committees in
10 communities
near park
• Community
members
trained in
safe water
storage
• NRM
committees
formed
2. Increased safe
water storage
3. Involvement of
NRM
committees in
promotion of
safe water
Logic Model: Voluntary Counseling and Testing for HIV
INPUTS
PROCESSES
OUTPUTS
Financial
resources
Supervision and
training for VCT
personnel
People
know their
HIV status
Provide pre-test
counseling
HIV+ people
are referred
to
appropriate
services
Demand for
services
Infrastructure
VCT MIS
VCT protocols,
guidelines, and
training documents
Counseling and
testing
personnel
Provide HIV testing
Provide post-test
counseling
Refer HIV+ clients to
appropriate services
(PMTCT, care and
support, treatment)
Maintain patient
records
HIV test kits
Referral system for
prevention & Tx
services
Complete
reporting
requirements
OUTCOMES
Clients (HIV+
and -) develop &
adhere to
personalized HIV
risk-reduction
Client
records are
available
HIV+ clients
develop &
adhere to
personalized
HIV care,
support, and
treatment
plans
Service
reports are
produced
Continuity of
care is
available
Program and
services are
improved
Risk
behaviors
decreased
Increase in care,
prevention, and
treatment
services for
HIV+, HIV-, and
discordant
couples
Health
outcomes of
HIV+
improve
IMPACTS
HIV
transmission
rates
decreased
HIV
incidence
decreased
HIV
morbidity
& mortality
decreased
Group Activity: Creating a Logic Model PHE Project in
Nepal
Goal: Improve the quality of life of people living in forest corridors and ecosystems while
insuring sustainable management of Nepal’s natural resources
Objective 1:
Health outcomes
of forest-dependent
communities
improved
by 2008
Activity 1:
Promote
Family
Planning
as a
Sustainable
Livelihood
Approach (SLA)
using
Traditional
Media (street
Drama)
Activity 2:
Develop
Communitybased
Distribution
System for
FP Products
Activity 3:
Disseminate
FP
HIV/AIDS
& PHE
Information
through
Non-formal
Education
Channels
Objective 2:
Enhance capacity
of user groups to
promote human
and ecosystem
health by 2007
Activity 4:
Promote the Use of
Improved Cooking
Stoves
Create Livelihood
Opportunities that are
Eco-friendly
(i.e., briquette
making from invasive
plant species)
Objective 3:
Increase support
In forest user
groups for PHE
by 2008
Activity 5:
Empower Community
Forest User Groups
(CFUGs) to Manage
Forest Assets and
Integrate P&H
Interventions into Their
Five-year Operational
Plans
Small Group Activity
 Return to small groups
 Determine/select three activities for your program:
• One activity from P or H
• One activity from E
• One activity that may work as integration or value added.
Give it a try! Use the Guide for help.
 Develop logic model components for the three
activities/processes of your program
 Work through inputs, processes (activities), outputs, outcomes,
and impact
 After ~30 minutes, a member of each group will share one
logic model
Module 5:
Indicators
Indicators Module Objectives
At the end of the session, participants will be able to:
1. Critique indicators
2. Identify criteria for selection of sound indicators
3. Understand how indicators are linked to frameworks
4. Select indicators and complete an Indicator Reference
Sheet
What Is an Indicator?
An Indicator is a variable that measures one aspect of a
program/project or outcome.
An appropriate set of indicators includes at least one
indicator for each significant aspect of the program or
project.
***For PHE, try to always include integration and
value-added indicators too***
Characteristics of Good Indicators
 Valid: accurate measure of a behavior, practice, or task
 Reliable: consistently measurable in the same way by
different observers
 Precise: operationally defined in clear terms
 Measurable: quantifiable using available tools
and methods
 Timely: provides a measurement at time intervals
relevant and appropriate in terms of program goals and
activities
 Programmatically important: linked to a public health
impact or to achieving the objectives that are needed
for impact
Characteristics of Good Indicators: Valid
 Indicator measures what it is supposed to measure
 Accurate measure of a behavior, practice, or task
• Direct measures
• Proxy measures (substitute measures)
 Straightforward interpretation: change in value signals
a change in concept or behavior
Validity: Class Discussion
1. Is “contraceptive prevalence rate” a valid indicator
of the impact of a family planning program for small
scale programs?
2. Is “number of trees planted” a valid measure of
reforestation efforts?
3. Is “# of women on natural resource management
committee” a valid measure of women’s
participation?
4. Does “new users of modern contraception” measure
women’s attitudes towards family planning?
Characteristics of Good Indicators: Reliable
 Consistently measurable in the same way by different
observers or people
 Types of errors in measuring indicators:
• Bias: estimates of condom use may reflect people’s
desire to answer questions to please you!
• Personal opinion: indicators that ask for personal
judgment such as “quality,” “positive environment,”
and “progress.”
Characteristics of Good Indicators: Precise
 Operationally defined in clear and understandable terms
 Partner Activity: each two-person group will develop two
definitions:
Select from:
1. New user to modern contraceptives
2. Knowledge of PHE issue
3. Adoption of improved agricultural technique
4. Participated in income generation activity
Characteristics of Good Indicators:
Measurable
 Quantifiable using available tools and methods
 Examples of measures:
• Days of drug stock outs
• Number of hectares under improved management
• % of children presenting at health clinic with acute
diarrhea
• % of households participating in income generating
activities
Characteristics of Good Indicators:
Timely
 Provides a measurement over period of time
with data available for all appropriate intervals
 Timeliness Considerations:
• Reporting schedules
• Recall periods
• Survey schedules
• Length of time over which change can be detected
Characteristics of Good Indicators:
Programmatically Important
Linked to a programmatic impact or to achieving the objectives
needed for impact
1. Example: Program to distribute and promote condoms
 Indicator: # of sex partners in last four weeks
2. Example: Fuel efficient stove distribution program
 Indicator: % of community households with fuel efficient stoves
3. Example: Program to increase access to oral rehydration salt
(ORS) through community based distributors
 Indicator: # of ORS packets distributed by clinics in past month
Factors to Consider When Selecting
Indicators
 Logical – are they linked to framework?
 Programmatic needs – do they get you the information you
need for decision making?
 Resources – can you afford to collect it?
 External requirements – do you need it for government,
donor, headquarters?
 Data availability – can you get the data you need for both
numerator and denominator?
 Standardized indicators – can you compare across
programs/countries? Is there a “gold standard” for this
indicator? Are they in the PHE Indicator Guide?
Operationalizing Indicators
 To operationalize an indicator is to identify how a given
concept or behavior will be measured
 Challenges:
• Subjective judgment
• Local conditions
• Unclear yardsticks
• Available data
CHALLENGES FOR PHE
INDICATORS
Indicators for M&E of PHE programs
PHE Indicators
 Population Indicators
• Reproductive health; family planning; births
 Health Indicators
• Vaccines; safe water; bed nets
 Environment Indicators
• Legally-protected area; community-based NRM;
species richness
 Integration Indicators
• Linked messages; collaboration; awareness
 Value-Added Indicators
• Gender; governance; economic development; youth
Each Indicator Includes:
 Description/definition
 Timing of indicator – measurement
• Activities (processes/inputs) or Results (outcomes/impact)
 Calculation of the measure
• Counts (number of guards trained and posted; number of
trees planted)
• Percentages, rates, ratios (contraceptive prevalence rate;
% of planted trees surviving)
 Purpose of the indicator
 Data source and disaggregation
 Frequency of data collection
 Strengths and weaknesses
Integration Indicators
 Help us measure our integrated PHE efforts across
programs
 Demonstrate commitment to providing diverse PHE
messages and services
 Show the importance of PHE issues to the communities
and countries where you work
 Help demonstrate community support, ownership, and
sustainability of PHE
 Could indicate the cost-effectiveness of these
cooperative efforts
Integration Indicator #3: Number and frequency of PHE sessions provided in
the target community (p. 108)
Definition: Educational sessions should be on topics related to integration of
population, health and environment issues rather than sessions on specific and
single-sector topics. These sessions could occur in any context such as
presentations to local officials, a community theater presentation, or as part of a
community educator’s typical work schedule.
Purpose: This indicator captures the extent to which the project is educating the
target population on the linkages between humans and the environment. Counting
the number of sessions can provide an indication of project progress.
Data Sources: Project records
Time frame: Monthly, quarterly
Data Collection Considerations: Sessions may include community visits,
outreach home visits, educational activities, videos, or communication programs
with integrated approaches.
Strengths & Limitations: This indicator is easy to collect with good project
record-keeping; however it does not give indication of whether the target audience
received the message or of their understanding and acceptance of the linked
message.
98
Value Added Indicators
 Demonstrate what was achieved outside of the initial
program plan
 Show how the unique aspects of PHE programs create
larger changes than single sectors programs
 Illustrate how PHE programs are more than the sum of
their parts
 Offer a way to “gain credit” for the additional changes
we create in gender, governance, livelihoods, and
youth
Value added Indicator #8: % of leadership positions held by women on
community NRM committees (p. 129).
Definition: A “leadership position” is any position where the chosen person has
commanding authority or influence.
Purpose: An NRM committee may make decisions regarding tasks completed by
women; therefore this indicator measures the ability of women to have a decisionmaking role in committee plans, actions, and control of resources. Increased
participation of women may lead to decreased gender inequality.
Data Sources: Project records
Time frame: Annually
Data Collection Considerations: Qualitative interviews with women may assess
their perceptions of involvement in the committee and obtain exact details
of their leadership responsibilities.
Strengths & Limitations: Women involved in decision-making for the NRM may
also have influence in education, health, and family planning decisions. However,
there may be cultural factors that prevent female leaders from speaking up; there
is no equality if women play only a ceremonial role.
Value Added Indicator #10: Number of fuel-efficient stoves distributed (page
133).
Definition: Fuel-efficient stoves are enclosed stoves that often employ an
elbow shape to provide a combustion chamber and insulation that increases the
heat available to cook food.
Purpose: Switching to fuel-efficient stoves can have direct impacts on both
forest and human health by limiting wood collection and by minimizing human
exposure to pollutants.
Data Sources: Project records
Time frame: Annually
Data Collection Considerations: A standard form should be used to keep
track of these data. The form should include the dates of distribution, total # of
stoves distributed, a list of recipients, and the village or community name.
Strengths & Limitations: Data on the number of stoves distributed should be
easy to obtain and track over time. However, the distributed stoves only have an
impact if they are used.
Special focus: Integrating Gender
 Gender sensitive indicators
• Women’s participation in program activities
• Women in leadership roles
• Young women’s decision making, etc.
 Collect program-related information among men and women
• Hold focus groups, conduct informational interviews with
men and women
• Collect household information from men and women
• Sex-disaggregated data – report information separately for
men and women (participation, leadership, etc.) when
possible
Common Indicator Metrics
 Counts
• Number of providers trained
• Number of condoms distributed
 Calculations: percentages, rates, ratios
• % of parks with trained guards
• Contraceptive prevalence rate, denominator and
numerator!
 Index, composite measures
• Quality index comprising the sum of scores on separate
outcome indicators
 Thresholds
• Presence, absence
• Pre-determined level or standard
The Right Information for the Right
Indictor: Indicator Metrics
Indicator 1: (p.49) Percent of program staff trained to work
with or provide reproductive health services to adolescents
 What does “trained” mean?
 What information do you need to collect this?
Indicator 2: (p.87) Percent of trees planted that survive
 Numerator: what would this be?
 Denominator: what would this be?
 What does “survive” mean?
Always Specify the Details!
 For example: Environment Indicator #8 – “improved
technology adoption” (p.92)
 How would you define “adopt” ?
Sources of Indicators: Pre-defined
Indicators
 From past years of the program
 From related or similar programs
 From lists of global or recommended indicators
• MEASURE Evaluation PHE Indicators – (Guide)
• Other resources listed in the training course, slides
150+
Adding Indicators to Logic Model
Input
Activity/
Process
Quantifiable
1. What you
resources
do to
going in to your
accomplish
activities – the
your
things you
objectives?
budget for.
Indicators
(example)
Output
Outcomes
Impact
Immediate
results
from your
activity –
people trained,
services
provided
Longer-term
change in
knowledge,
attitude,
behaviour,
etc. related to
program goal
Long-term,
population level
change. Can
relate to a
program or
organization’s
vision/mission
statement
How do you
measure this?
How do you
How do you
measure this? measure this?
Logic Model: Family Planning Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Educate women
about the
advantages of
modern method
use
• Increased
interest in FP
1. Increased
access to FP
services
• Increased
contraceptive
prevalence
• Demand for
FP in the
community
• Distribute FP
methods in the
community
• Sessions
held in
community
about family
planning
methods
• Train program
staff in providing
FP information
and methods
• Clinic staff
trained in FP
method
counseling
2. Increased
condom use
3. Increased
male
participation in
FP
Adding indicators: Family planning activity
INPUT
• Human and
financial
resources
• Demand for
FP in the
community
PROCESS
• Educate
women about
the advantages
of modern
method use
• Distribute FP
methods in the
community
• Train program
staff in
providing FP
information and
methods
OUTPUT
• Increased
interest in FP
• Sessions held
in community
about family
planning
methods
• Clinic staff
trained in FP
method
counseling
OUTCOME
OUTCOME
1. Increased access to
1.
FP Increased
services
access condom
to FP
2. Increased
useservices
IMPACT
• Increased
contraceptive
prevalence
3. Increased male
2.
Increased
participation
in FP
condom use
3. Increased
male
participation in
FP
Indicators:
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
Adding Indicators: Family Planning Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
IMPACT
• Human and
financial
resources
• Educate
women about
the advantages
of modern
method use
• Increased
interest in FP
1. Increased access
to FP services
• Sessions held
in community
about family
planning
methods
2. Increased condom
use
••Increased
Increased
•Increased
contraceptive
contraceptive
contraceptive
prevalence
prevalence
prevalence
• Demand for
FP in the
community
Indicators:
• Distribute FP
methods in the
community
• Train program
staff in
providing FP
information and
methods
• Clinic staff
trained in FP
method
counseling
OUTPUT
INPUT
PROCESS
• % of program
staff trained to
provide FP
information to
youth (Pop # 1)
• # of educational
session held in
the community
about FP methods
3. Increased male
participation in FP
OUTCOME
% of women who
were clients of
community-based FP
distributor last year
(Pop #2)
• # of new
acceptors of FP
methods (Pop #6)
• % of men who
approve of FP for
themselves or their
partners (Value
added #13)
IMPACT
• Contraceptive
prevalence
rate (Pop #7)
Logic Model: Diarrhea Prevention Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Train park guards
in diarrheal disease
and use of oral
rehydration salts
• (ORS)
packets
distributed
1. Reduction in
acute
diarrhea
cases among
children
• Reduced
morbidity
/mortality
from
diarrheal
disease
among
children
• Demand for
disease
intervention
in community
• Train local
community
members in safe
water storage
• Form natural
resource
management
committees in 10
communities near
park
• Community
members
trained in
safe water
storage
• NRM
committees
formed
2. Increased
safe water
storage
3. Involvement
of NRM
committees in
promotion of
safe water
Adding Indicators: Diarrhea Prevention Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Train park guards in
diarrheal disease
and use of oral
rehydration salts
• (ORS)
packets
distributed
1. Reduction in
acute diarrhea
cases among
children
• Reduced
morbidity
/mortality
from diarrheal
disease
among
children
• Demand for
disease
intervention in
community
Indicators:
INPUT
• Train local
community
members in safe
water storage
• Form natural
resource
management
committees in 10
communities near
park
PROCESS
• Community
members
trained in safe
water storage
• NRM
committees
formed
2. Increased safe
water storage
3. Involvement of
NRM committees
in promotion of
safe water
OUTPUT
3 HERE – 2
from the GUIDE
*including at
least one value
added
Integration
indicator???
OUTCOME
3 HERE – 2 from
the Guide
*including at least
one value added!
Integration indictor?
IMPACT
Adding Indicators: Diarrhea Prevention Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Train park guards in
diarrheal disease and
use of oral rehydration
salts
• (ORS) packets
distributed
1. Reduction in acute
diarrhea cases
among children
• Reduced
morbidity
/mortality from
diarrheal
disease among
children
• Demand for
disease
intervention in
community
Indicators:
INPUT
• Train local community
members in safe water
storage
• Form natural resource
management
committees in 10
communities near park
PROCESS
• Community
members trained
in safe water
storage
• NRM committees
formed
2. Increased safe
water storage
3. Involvement of
NRM committees in
promotion of safe
water
OUTPUT
OUTCOME
• # of NRM committees formed in
the 10 communities near the
national park
% of households with access to n
improved water source (health #7)
• # or ORS packets distributed
(health 3)
• Number of educational sessions
held about safe water sources,
storage, etc.
• % of households with access to
improved water source (health 7)
• Guide Integration indicators # 2,
3, 4, or 5 may work!!!!
% of communities in target area with
community based (NRM) plans (Env’t 1)
Oral rehydration therapy use rate
(health 13)
% of households storing drinking water
safely (health 11)
Value Added Indicators:
Guide value-added indicators #1, 7,or 8
might work!
IMPACT
Indicator Reference Sheet
Definition: detailed documentation for each indicator
 Basic information
 Description
 Plans for data collection
 Plans for data analysis, reporting, and review
 Data quality issues
 Performance data table (baseline and targets)
Population-Based Indicator Reference Sheet
Core Indicator Relating to Key Result (Objective): Required by Projects Conducting Surveys
Indicator Description
Indicator:
Contraceptive Prevalence Rate (Met Need)
Definition:
Percentage of women of reproductive age (WRA) 15-49 who are married or in union using (or whose partner is
using) a modern method of family planning
Unit of Measure:
Percentage
How to Calculate:
NUMERATOR: No. of women age 15–49 years who are married or in union who are not pregnant (or unsure) AND
who report using (or whose partner is using) a modern method of family planning.
DENOMINATOR: Total number of women age 15–49 years who are married or in union included in the survey
Indicator = (Numerator/Denominator) * 100.
Important Background
Characteristics to
Consider When
Assessing This Indicator:
At minimum, you are encouraged to examine this indicator by age group.
What It Measures:
Population coverage of current family planning use
Important Notes:
The conventional indicator is limited to women in union/married.
Modern methods: The following methods are usually counted as modern methods in the indicator: female
sterilization (tubal ligation), male sterilization (vasectomy), pills, IUD, injections, implants (NORPLANT), condoms,
female condom, diaphragm, and foam/jelly and LAM. The Standard Days Method (SDM) is also in the process of
being designated as a modern method of FP.
Data Collection
Data Source:
Population-based household survey (Flexible Fund Family Planning Survey)
Data Collection Method:
To obtain the CU/CPR, please refer to the Flexible Fund Family Planning Survey for model questions required to
construct the indicator. The most recent version is found on the website: www.childsurvival.com
Frequency/Timing of
Data Collection:
Baseline, midterm (if applicable), and final evaluation
Proposed Data Use:
The overall purpose of this indicator is to assess whether all your project activities are contributing to the ultimate
objective of increased contraceptive use. If the CU or CPR is lower than expected, consider all the possible factors
that contribute to end use (e.g., knowledge/interest, quality of care, access).
Setting Indicator Targets: Useful
Information Sources
 Past trends
 Client expectations
 Donor expectations
 Expert opinion
 Research findings
 What has been accomplished elsewhere
Common Pitfalls in Indicator Selection
 Indicators not linked to program activities
 Using outputs as outcomes
 Poorly defined indicators
 Data needed for indicator is unavailable
 Indicator does not accurately represent intended
outcome
 Too many indicators in P, H, and E
Pitfalls with Selecting Indicators
Indicator not linked to program activities
IR: Expanded access to diarrhea treatment services
Activities: Train providers in treating acute diarrhea
Inappropriate Indicator: % of facilities with adequate
conditions to provide care
Better indicators: # of clinicians trained, # of facilities with a
trained provider, % of clinicians with 100% on post-test
The program is not aiming to affect facility conditions, only
provider skills.
Pitfalls with Selecting Indicators
Using outputs to measure outcomes
Problem: routine monitoring data (outputs) is available from
households but outcomes are needed for reporting
You collect: # of households that use your new crop rotation
technique
You need: % of households in target areas who use your crop
rotation technique
Routine monitoring data should not be reported as outcome
data. Compile your routine data as well as information about
your target area to turn outputs into outcome measures.
Pitfalls with Selecting Indicators
Indicator poorly defined
Activity: radio campaign about PHE linkages or program
components
Inappropriate Indicator: % of population who have heard of X
campaign
Better indicators: % of population who can relate correct
information about a PHE issue
Campaign knowledge does not indicate what is
desirable: knowledge that the PHE issues are important,
what PHE issues are.
Pitfalls with Selecting Indicators
Data needed for indicator not available
Data issue: Information on stock-outs may not be
collected daily
Inappropriate Indicator: % of days per quarter that service
delivery points have stock-out of drugs
Better indicators: % of service delivery points that had a stock
out of drugs at some time during the last quarter
If relying on routine data, indicator definition will depend
on how data are collected.
Pitfalls with Selecting Indicators
Indicator does not accurately represent desired
outcome
IR: Expanded access to family planning methods
Inappropriate Indicators: % of women using family planning
methods
Better indicator: # of new users of family planning;
contraceptive prevalence rate; estimated CPR
What does it mean if inappropriate indicators increase?
Decrease? Do they reflect the desired program effect?
Pitfalls with Selecting Indicators
Too many indicators for P, H, and E
Rules of thumb:
• One or two indicators per key activity or result (ideally, from different
data sources) (e.g., training, committee building, new users of
family planning)
• Try to include at least 1 “value added” indicator per objective area
• Try to include at least 1 “integration” indicator to represent each
linked activity, partnership, or outcome
• Not more than 8 -10 indicators per programmatic area, for P, H or E
There is no set formula for how many indicators. Consider
your budget, stakeholders, and program plans – then,
consider: is it reasonable? Feasible?
Indicator Pyramid
Global
Compare countries
Overview world-wide situation
National/Sub-national
Assess effectiveness of response
Reflect goals/objectives of national/
sub-national response
District, Facility, Community
Identify progress, problems, and challenges
Decreases
Number of
Indicators
Increases
Key Questions to Ask When Selecting
Indicators
 Do they meet programmatic needs?
 Will they give you useful information for decision making?
 Are they feasible considering time, money, and staffing?
 Do they match external requirements?
 Are the data available? How accurate are the data?
 Are they standard indicators used across projects,
programs, countries?
 Are you collecting the information appropriately?
Small Group Activity




Return to small groups
Return to the logic models from Module 4
Choose ONE of the logic models you completed
Like the examples, fill in the indicators for your logic model.
• You may need to add additional detail for the output and
outcome sections of your logic model before adding
indicators.
• You must have at least 2 output and 2 outcome indicators.
• REMEMBER: In general, output indicators are counts while
outcome indicators measure longer term changes in % or
proportion in knowledge, attitudes, or practices
 ~20-30 minutes for group work
 ~30 minutes for group presentation/group critique
Small Groups Activity: Adding Indicators
to Your Program Logic Model
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
PROCESS
OUTPUT
OUTCOME
IMPACT
Indicators:
INPUT
Module 6:
Indicator Matrixes for M&E
Brief Summary of Three Frameworks
Type of
Framework
Brief Description
Use for Program
Management
Basis for Monitoring
and Evaluation of PHE
Conceptual
Shows the interactions Determine which
of various factors that factors the program
operate in your
will influence and how
program environment
Helps explain and
promote the multi-sector
influences and integrated
approach
Logic model
Logically links inputs,
processes, outputs,
and outcomes of
planned activities
Shows the causal
relationship between
inputs and reaching
the objectives
Shows intended progress
of activities at all stages
of the program – aids
indicator selection
Indicator
matrix
Logically links
program goal,
objectives, outputs,
outcomes, and
activities
Shows the causal
relationship and steps
between activities and
meeting objectives
Guides monitoring and
aids assessment by
stating the information
you will collect, how,
when, where, and why.
Brief Summary of Three Frameworks
Type of
Framework
Brief Description
Conceptual
Shows the interactions Determine which
of various factors that factors the program
operate in your
will influence and how
program environment
Helps explain and
promote the multi-sector
influences and integrated
approach
Logic model
Logically links inputs,
processes, outputs,
and outcomes of
planned activities
Shows the causal
relationship between
inputs and reaching
the objectives
Shows intended progress
of activities at all stages
of the program – aids
indicator selection
Indicator
matrix
Logically links
program goal,
objectives, outputs,
outcomes, and
activities
Shows the causal
relationship and steps
between activities and
meeting objectives
Guides monitoring and
aids assessment by
stating the information
you will collect, how,
when, where, and why.
130
Use for Program
Management
Basis for Monitoring
and Evaluation of PHE
Indicator Matrix
Definition: a standardized summary of the project and its logic.
Purposes:
 Summarizes what the project intends to do and how
 Summarizes outputs and outcomes that will be monitored
and evaluated
 Details the indicators that will be used to measure
progress and achievement of objectives
 Encourages discussion of data sources
Filling In the Blanks: Data Needs for M&E
Before you begin the indicator matrix, it’s important to think about
data!
 What data do you need?
 What data do you have?
 What are the gaps in data availability?
 How can you fill the gaps internally?
 How can you get the data externally?
How Do We Get Data for Decision Making?
Qualitative Methods
 Focus groups
 Key informant interviews
 Participatory rural appraisal/rapid appraisal techniques
Quantitative Methods
 Adequacy and plausibility assessments
 Surveys
 Pre-post training tests
Internal Data Sources
 Data that is generated by your own
organization
 Routine sources: procurement
records, trip reports, budgets,
facility records, training records
 Non-routine sources: surveys,
participatory research findings
External Data Sources
 Who else is working where you are working?
Health
NGO
District
Health
Officer
Family
Planning
Services
NGO
Women’s
Self-help
Group
Conservation
Partner
Community
Council
Park
Ranger
District
Level
Environment
Officer
Farmer’s
Cooperative
Data Source Issues for PHE: Sharing Data
 If other organizations are gathering data your organization
could also use, you could establish a data sharing partnership.
 Sharing data with other organizations reduces the burden of
data collection on any one organization.
 Contributes to stronger relationships.
 What are the barriers to data sharing?
Evaluating Secondary Data
Characteristics of quality data:
 Valid: accurate measure of a behavior, practice, or task
 Reliable: consistently measurable in the same way by different
observers/promoters/supervisors
 Precise: clearly written, well-kept records
 Measurable: does the data answer your question?
 Timely: available at time intervals relevant and appropriate in
terms of program goals and activities
 Programmatically important: linked to achieving the objectives
that are needed for your monitoring, evaluation, or impact
assessment
Additional Questions for Consideration
Although it is a good idea to use existing data to save time,
energy, and costs, you must also consider:
 Who collected the original data?
 How complete is it?
 What is its intended purpose?
 Is it available?
 Is it sensitive?
 What does it cost to use in terms of time, $, travel,
etc.?
Giving Credit
 Data and information sources always need to be
properly credited in any reports, publications,
presentations, or even internal meetings.
 This adds credibility to your information and builds
trust in data sharing.
 Your organization can ask that your data is also
properly credited when other organizations use it.
Pulling It All Together…
Adding Indicators: Diarrhea Prevention Activity
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
• Human and
financial
resources
• Train park guards in
diarrheal disease and
use of oral rehydration
salts
• (ORS) packets
distributed
1. Reduction in acute
diarrhea cases among
children
• Reduced
morbidity
/mortality from
diarrheal
disease among
children
• Demand for
disease
intervention in
community
• Train local community
members in safe water
storage
• Form natural resource
management committees
in 10 communities near
park
Indicators:
• Community
members trained
in safe water
storage
• NRM committees
formed
PROCESS
3. Involvement of NRM
committees in
promotion of safe
water
OUTPUT
OUTCOME
• # of NRM committees formed in the
10 communities near the national
park
% of households with access to an
improved water source (health #7)
• # or ORS packets distributed
(health 3)
INPUT
2. Increased safe water
storage
• Number of educational sessions
held about safe water sources,
storage, etc. (in project/target
area)
• % of households with access to
improved water source (health 7)
• Guide Integration indicators # 2, 3,
4, or 5 may work!
% of communities in target area with
community based (NRM) plans
(Env’t 1)
Oral rehydration therapy use rate
(health 13)
% of households storing drinking
water safely (health 11)
Value Added Indicators:
Guide value-added indicators #1,
7,or 8 might work!
IMPACT
Section of PHE Indicator Matrix
Overall Objective 2: Reduce childhood morbidity in priority biodiversity areas in target communities
Intermediate Result
2.1: Improved
information about and
access to safe
drinking water
Data
Source
Frequency
Baseline
End-ofProject
Target
Discussion
Points/Comments
Information
User
This does not cover
how many people
attended, or if the
people who
attended were
those that required
training or whether
they adopted the
information.
Who wants
this
information?
Donors?
Program
managers?
Outputs (what is the specific result of your activity?):
Educational sessions
held about safe water
sources, storage, etc.
(In project/target
area.)
Program
Records
Biannually
Outcomes (what change do you expect to see in knowledge, attitudes or practices as a result?):
Percent of
households storing
drinking water safely
(in project/target
area).
Household
survey;
household
visits
Annually
Check
90% of
available housedata
holds
The distribution of
water storage units,
and training to
properly use the
units, are listed as
output indicators
above.
Who wants
this
information?
Donors?
Program
managers?
Large Group Activity
In one group we will:
 Examine pieces of the example PHE indicator matrix
Let’s think through:
 The four overall objective indicators – think about 2-5 years
– do they work?
 Note that each objective is further divided into intermediate
results.
 Under each objective sub-heading – population, health,
conservation, and M&E – we will look at one outcome
indicator (total of four outcomes)
 Under each objective sub-heading – population, health,
conservation, and M&E – we will look at one output
indicator (total of four outputs)
Preparing to Create Your Indicator Matrix
Questions for you to discuss as you plan your indicator matrix:
 What data is available to you?
 What data do you need?
 What could you collect?
 What would you like if it were available?
 What sources of data could you explore?
 How many “standard indicators” can you include?
Small Group Projects
Use the PHE programs M&E indicator matrix as an example.
Using your own project/program, you will:
 Use your two outputs and two outcome indicators from the
previous activity on adding indicators to logic models.
 Are the indicators P,H, E, I, or V?
 Write your own indicator matrix, including filling in across
all rows, for these four indicators (follow the format of the
indicator matrix example).
 For one indicator, complete the indicator reference sheet.
Present your findings to the group.
Module 7: Assessment and Evaluation
Designs for PHE
Objectives of the Session
By the end of this session, participants will be able to:
 Define what is an adequacy assessment and an
evaluation study
 Describe what assessments and evaluation studies are
most appropriate for low resource PHE programs
 Understand how to select the best design for their
specific study
Capacity and Resources for M&E
Capacity for M&E
• M&E unit or dedicated M&E person
• The M&E person should know if the survey is done well,
even if not doing the survey in house
Scale of program funding and resources for M&E
• Ten percent of resources devoted to M&E
• Costs related to data collection systems & information
dissemination
• Costs may be higher during the first year
• Other helpful resources include indicator guides,
manuals, communication tools
Type of
assessment
Answer what kind of questions?
Adequacy
•
•
•
•
•
Plausibility
• Did the program seem to have an effect above
and beyond the contribution of other, nonprogram influences?
• Requires comparison groups.
• May benefit from a pre-/post-test design.
Did the expected changes occur?
Are objectives being met?
Were activities performed as planned?
Does not require a comparison group.
May benefit from a pre-/post-test design.
Adequacy Assessment
Adequacy studies describe whether an objective (outcome)
is met or not.
 Typically addresses provision, utilization, or coverage
aspects. No need for a comparison group.
• Question answered: Are expected levels achieved?
 Can also answer questions of evaluation (amount of
change) if you do pre- and post-tests.
• Question answered: Is the difference between
program and non-program communities equal or
greater than expected?
Features of Adequacy Assessment
 Pros:
• If only input or output (activity) results are needed, this
may be a good design for you.
• Simplest (and cheapest) of evaluation models.
• Data are needed only for outcomes (think about your
logic framework!).
 Cons:
• If there is no change, you will not know whether the
lack of change is due to program failure or not.
• You cannot use this design to measure impact – you
will not know if changes are due to the program alone.
Adequacy assessments designs
1. One program group – post-test only (weakest design)
IMPLEMENT
PROGRAM
ASSESS TARGET GROUP AFTER
PROGRAM
2. One group, pre- and post-test (better)
ASSESS
TARGET GROUP
BEFORE
PROGRAM
IMPLEMENT PROGRAM
ASSESS TARGET
GROUP AFTER
PROGRAM
Plausibility Assessment: Post-test Only
3.
BEFORE
PROGRAM
Community 1
Community 2
AFTER PROGRAM
IMPLEMENT
PROGRAM
NO PROGRAM!
ASSESS TARGET
GROUP AFTER
PROGRAM
ASSESS
COMPARISON
GROUP AFTER
PROGRAM
Plausibility Assessment: Post-test Only
 Identify people/communities who received the program.
 Identify similar people/communities who did not receive
the program.
 Test both groups USING THE SAME QUESTIONNAIRE
OR FOCUS GROUP OR EVALUATION METHOD.
 Compare results between program and non-program
groups.
 This is better than an adequacy assessment because of
the comparison group.
Plausibility assessment: pre- and post-test
4.
BEFORE
PROGRAM
ASSESS TARGET
GROUP BEFORE
PROGRAM
ASSESS
COMPARISON
GROUP BEFORE
PROGRAM
AFTER PROGRAM
IMPLEMENT
PROGRAM
NO PROGRAM!
ASSESS TARGET
GROUP AFTER
PROGRAM
ASSESS
COMPARISON
GROUP AFTER
PROGRAM
Plausibility Assessment: Pre- and Post-test
 Identify people/communities who get the program.
 Identify similar people/communities who do not get the
program.
 Pre-test both groups USING THE SAME QUESTIONNAIRE
OR FOCUS GROUP OR METHOD.
 Deliver the intervention/program only to the program group.
 Post-test both groups with the same method after the program
finishes.
 Compare results between program and non-program group;
consider assistance from a skilled evaluator or consultant.
Choosing the Right Design for You
 Adequacy designs will allow you to link program outcomes to
program activities.
 If you want to have more certainty, try to use a pre- and posttest design.
 If you want to be more confident, you should try to use a
comparison group – it will be more costly.
 If you have the resources, hire an evaluator to help determine
which design will maximize your program’s resources and
answer your team’s evaluation questions with the greatest
degree of certainty.
 Disseminate your findings and share lessons learned!
Areas of Concern to Different DecisionMakers
Type of
Evaluation
Provision
Utilization
PHE Programs
Adequacy
Health center manager
International Agencies
Plausibility
Coverage
Impact
District health managers
International Agencies
Donor
agencies
International Agencies
Scientists
Probability
Donor Agencies & Scientists
Source: Habicht, Victora and Vaughan (1999).
Summary
Assessment
Objective
What It Says
Data Needs
Adequacy:
Assess whether
impact was
reached.
Indicates whether
resources were
well spent or not.
Outcome data collected
among participants.
Baseline data improves
study design.
Understand what
affects the
outcomes.
Helps understand
how and why the
program
succeeded or
failed.
Outcome data collected
among participants and
comparisons. Baseline
data improves study
design.
(Assessment of
change in
outcome.)
Plausibility:
(Uses
comparison
group.)
Small Group Activity – Assessment Planning
 Take the four indicators you detailed in your indicator matrix (you
may consider more indicators if you choose);
 Review the data source for your output and outcome indicators;
 Now, considering your data needs, your actual budget, and your
program stage, discuss an assessment strategy:
1. What type of assessment could you do (of four designs)?
• Can you afford a baseline? Comparison group? Is it too late?
2. How will you collect the information for your indicators?
(Household survey, facility survey, household visits, program
records, focus groups, interviews, community mapping, etc.)?
3. What specific information do you need? (Think about
numerator and denominator.) What questions will you ask?
Who will you ask?
4. What forms might you need for monitoring outputs?
5. What skills/knowledge will your assessment team need to look
at outcomes?
Small Group Activity – Presentations
 One person from each group should present your major
discussion points
 10 minutes per group
 In your brief presentation, include information on:
1. Indicators you are using
2. Possible assessment design to collect the data
(numerator/denominator)
3. Data collection type (surveys, focus groups, etc.)
4. Anticipated difficulties
5. Expected successes
Conclusion, Wrap-up, and Evaluation