Transcript First Client Exit Interview MIP Akwa Ibom State

CDI Module 13: Basic M&E Concepts for
Integrated Community Management of Malaria
©Jhpiego Corporation
The Johns Hopkins University
A Training Program on CommunityDirected Intervention (CDI) to Improve
Access to Essential Health Services
Module 13 Objectives
By the end of this module, learners will:
 Describe participatory monitoring and evaluation
(PME)
 State the roles of beneficiaries in PME
 Define a community-based information system
and state why it is important
 Explain the meaning of recordkeeping and state
why it is important
 Describe reporting and state why it is important
2
Learners Will Also …
 Define basic monitoring and evaluation (M&E)
concepts and state the differences between
monitoring and evaluation
 State why M&E is important in programming
 Differentiate between data and information
 Describe data and identify qualities of good data
 Identify sources of data
 Demonstrate how to fill out the register for home
management of malaria (HMM) and the referral
form
3
Key M&E Questions
 Did the program achieve its objectives?
 Did the target population benefit from the
program?
 At what cost?
 Can improved health outcomes be attributed
to program efforts?
 Which program activities were more (or less)
important/effective?
4
What Is PME?
PME is a process of involving beneficiaries in
tracking program activities as well as in
assessing the extent to which desired goals and
specific objectives are met
Participatory monitoring involves local
beneficiaries who measure, record, collect,
process, analyze and communicate data

These data relate to program activities that are
directed at meeting the beneficiaries’ needs
5
What Is PME? (continued)
Participatory evaluation is a process of
reflecting on the extent to which changes have
occurred
The goal of PME is to use available community
resources to track the progress of community
development programs
6
PME’s Role in CDI
PME can help a community-directed intervention
(CDI) program identify:
 What is working well
 What is not working well
 How to proceed next
The PME process, however, requires the technical
assistance of a specialist
7
Principles of PME
 Participation: Involve the community in contributing
ideas, decision-making and taking responsibility
 Negotiation: Create an understanding among
conflicting community issues, and take the lead in
proposing community-based solutions that are a win-win
for all
 Flexibility: Use an approach that is democratic and
makes the project all-inclusive
8
More Principles of PME
 Learning: Involve stakeholders as active
participants, and offer them the opportunity to
explore new ways to assess and learn from
change—ways that are more inclusive and
reflect the perspectives and aspirations of those
most directly affected
 Varied evaluation methods: For example, use
simple surveys, in-depth interviews, focus
groups, observation
9
PME Methods/Tools
That Can Be Applied to CDI
Community/social mapping is a process in which
researchers/informants gather data and draw maps
Maps:
 Show important locations
 Give useful information that can:
– Guide discussions and interventions, or
– Support the project
Ranking can be used to:
 Prioritize problems in the order that they affect the community or
people most
 Consider potential resources available to solve the problems
 Evaluate options in a sequence (the same as giving a rank to each
student in a class after an exam)
10
More PME Methods/Tools
That Can Be Applied to CDI
Pair-wise ranking compares two attributes and places
one above the other (similar to preference ranking in that
people are asked to choose between items)
Flow diagram/charting can be used to:
 Analyze the sequence of activities that occur in a
particular process
 Break up a routine process, or set of activities, into a
series of sub-steps that make up the process
 Define the relationship between activities and the
desired outcomes of those activities
11
More PME Methods/Tools
That Can Be Applied to CDI
Matrix scoring can be used for comparing two or
more activities or projects
 Each activity/project is assigned a number
 A matrix is drawn with a box for each activity or project being
compared
 Participants are asked to vote for (or they can write on a piece of
paper) the activity/project they prefer most
 The number of votes is written in the box containing the
activity/project; the one with highest number of votes is ranked first
Matrix scoring can be used to compare successful projects with
unsuccessful projects, according to various criteria (e.g., number
of people who donated labor, amount of external resources
obtained, number of times the community met to work on the
project)
12
More PME Methods/Tools
That Can Be Applied to CDI
Seasonal calendar: Used to illustrate trends in
disease patterns or product outputs (e.g., number
of trees planted)
Scoring: Like ranking, scoring provides an
opportunity to evaluate different choices—very
similar to ranking; however, scoring provides
additional analysis
13
More PME Methods/Tools
That Can Be Applied to CDI
Village meeting/open
discussion: Provides
community members
an opportunity to give
feedback on programs
14
More PME Methods/Tools
That Can Be Applied to CDI
Interviews are:
 Two-way conversations on topical issues
 Conducted with guidelines that will bring out ideas,
opinions or perceptions of the people being
interviewed
 For example, discussions with community members
about:
 Their perceptions of the project’s success
 Problems with project implementation
 The direct and indirect effects of the project on the lives of
community members
15
What Is a Community-Based
Information System (CBIS)?
CBIS is the process of collecting information for
services that are provided at the community level
16
CBIS
CBIS:
 Documents services that
are outside of—but
linked to—health
facilities
 Involves planning,
collecting, managing,
reporting and using
information
CBIS activities are done
with community
participation
17
Examples of Information That Can Be
Collected and Reported by Communities
 Births and deaths (vital events registration)
 Notification of cases of infectious diseases and
outbreaks (disease surveillance)
 Data on routine service activities (routine health
service statistics)
 Tracking of defaulters of critical services (client
tracking)
 Identification of pregnant women
 Identification of orphans
18
Different Actors/Levels in CBIS






Community leaders
Volunteers, community health workers
Community groups/organizations
Field supervisors
State and district officers
National program (e.g., the Ministry of Health
[MOH] and relevant agencies like the National
Malaria Control Program [NMCP])
 Donors
19
Different Actors Have
Differing Information Needs
Information is needed to:
 Inform communities about health risks, how to
access services, etc.
 Help community workers manage their work, and
report results to higher levels
 Allow supervisors to monitor the outputs and
outcomes of programs, and provide support to
community workers
 Enable districts, states and the MOH/NMCP to
identify and prioritize health problems, and to
allocate resources
20
CBIS for Community-Based
Malaria Programs






HMM for children and adults
Rapid diagnostic testing
Malaria in pregnancy
Home-based prevention of malaria
Referral for case management
Others
21
What Is Recordkeeping?
If it is not in writing, it does not exist
Recordkeeping is a process of documenting services
provided over a period of time using specific formats
designed for the purpose
22
Why Recordkeeping Is Important
Records provide us with what
we need to know:
 How many people receive
services over a period of time
 How many pregnant women
receive doses of intermittent
preventive treatment in
pregnancy (IPTp)
 How the program is
performing
 Information for planning
purposes
23
Records Help Us
Records help us to:
 Quantify the different drugs and other supplies
that we need for serving clients over a period of
time
 Know if the right people are receiving the
services
 Encourage service providers to improve services
24
Setting Up a User-Friendly
Recordkeeping System
 Review program objectives in line with the
relevant indicators
 Identify the categories of data/information you
need to plan, monitor and evaluate the project
(input, output, etc.)
 Identify where (source) data will be collected
 Review any existing data collection formats
25
Setting Up a User-Friendly
Recordkeeping System (continued)
 Adapt the formats to meet your project needs
(e.g., forms, checklists, questionnaires)
 Determine the frequency of data collection
 Identify the persons who will collect the
necessary data
 Train staff in use of the formats
 Establish a data collation process, and set up a
data analysis reporting and feedback system
26
What Is Reporting?
A process of providing a summary of activities carried
out over a period of time
Involves summarizing the number of times different
services are provided in a given time
27
Why Is It Important to
Report Services Provided?
Reporting:




Provides a summary of accomplishments
Shows progress at a glance
Is useful for planning
Makes it easy for the supervisors to know what
activities each of the motivators carried out over
a period of time
 Provides feedback to funders, fund recipients
and program beneficiaries
28
What Is Monitoring?
A continuous systematic process of collecting,
analyzing and using information to track the efficiency of
achieving program goals and objectives—often called
process evaluation
29
What Is M&E?
Monitoring
 A process of routine
tracking of program
activities on a regular
basis
Evaluation
 A process of
measuring the extent
to which program
outcomes and outputs
are achieved
30
What Is Monitoring?
 Monitoring provides regular feedback that
measures change over time in any of the
program components
 For example, costs, personnel and program
implementation
 An unexpected change in monitoring data may
trigger the need for a more formal evaluation of
activities
31
What Should Be Monitored?
 Progress of project activities
 Services, training, outreach, meetings, information,
education and communication (IEC) activities
 Use of resources
 Finances, commodities, equipment, supplies, staff
 Results
 Achievement of project objectives, coverage, outputs,
impact, effect on project beneficiaries
 Institutional development
 Staffing, policies and procedures, organizational
systems
32
What to Monitor at the Community Level
 Stock levels of all the Roll Back Malaria Partnership
(RBM) commodities and medicines








Artemisinin-based combination therapies (ACTs)
Rapid diagnostic tests (RDTs)
Long-lasting insecticide-treated nets (LLINs)
Data capturing tools
Types of ACTs provided
Storage facility and condition of ACTs and RDTs
Training in all aspects of malaria control
Use of data capturing tools
33
Why Monitor?
The purpose of monitoring is NOT:
 To criticize your colleagues and their work
 To please international donors (and spend lots
of their money)
 To exert power—keep other people running
around, compiling tables, writing up reports
34
What Is Outcome Evaluation?
Outcome evaluation is a systematic process—
limited in time—of collecting, analyzing and using
information to assess the effectiveness, relevance
and impact of achieving your program’s goals
35
What Is the Purpose of
Outcome Evaluation?
Outcome evaluation:
 May involve measuring changes in knowledge,
attitudes, behaviors, skills, community norms,
utilization of health services and health status at
the population level
 Provides regular feedback that helps programs
analyze the consequences, outcomes and
results of their actions
36
What Should Be Evaluated?
 Effectiveness
 Is the project achieving satisfactory progress toward
stated objectives?
 Efficiency
 Are the effects being achieved at an acceptable cost,
compared with alternative approaches?
 Relevance
 Are the project objectives still relevant?
 Impact
 What difference has the project made?
37
What Can We Evaluate
at the Community Level?
 Availability and accessibility of RBM commodities
and medicines
 Use of LLINs by the community
 Appropriate and effective use of ACTs
 Appropriate storage conditions (temperatures) of
ACTs and RDTs
 Reduction in fever/malaria cases from routine
reports
 Number of referrals
 Number of structures adequately treated with indoor
residual spraying (IRS)
38
Requirements for Effective M&E
Programs should:
 Institutionalize the M&E process
 Establish an operational M&E plan using a
participatory approach
 Strengthen personnel capacity to conduct M&E
at all levels
 Set up a data collection system
 Set up a data entry and reporting template
 Establish a reporting, feedback and data use
mechanism
39
What Are the Differences between
Monitoring and Evaluation?
Ask learners to brainstorm and identify the differences
40
Indicators and Data
Indicators are measurable variables that provide us with
directions
The program goal and objectives inform the selection of
the indicators
Indicators are searchlights and guides in program
implementation
41
Levels of Indicators
Type
Example
Input
Resources used
Process
Programs implemented on time
Output
Number of radio programs aired
Outcome
Number of people sleeping under nets
Impact
Number of low birth weight babies born
42
What Are Data?
 Data simply means an amount of something
 Data often come in figures or tallies
 Data can be collected at both the community
and health facility levels
Reminder: If it is not in writing, it does
not exist
43
Difference between Data and Information
 Data are raw and unprocessed; information is
processed data
 Example—a health provider can use a tally to
represent an amount of something (e.g., ////)
 These strokes (data) convey no information until they
are processed
 A health provider may process these strokes
(data) as //// = four children under five years of
age who have received LLINs
 Once processed, these strokes (data) convey
the required information
44
What Makes Good Data?
 Good data are:
 Reliable
 Valid
 Complete
 Timely
 Precise
 Truthful (have integrity)
 Good data should be easy to collect, costeffective and sufficiently updated
45
What Can Reduce the Quality of Data?
 A poor data collection instrument
 Untrained data officers, or not selecting the right
people for training
 Poorly trained data officers
 An unreliable data entry system
 An insecure data storage system
 A change in data officers
46
Sources of Data
How can we get these
data?
 Routine sources
 Continuous data
collection
 Non-routine sources
 Special data collection
procedures
47
Routine Data Sources
 At the village Level
 Village register
 Service provision form
 Monthly service summary form
 At the clinic Level
 Antenatal care (ANC), treatment and other registers
 Client cards (ANC, child welfare, immunization)
 Monthly summary forms
Can you think of other routine data sources?
48
Non-Routine Methods




Community surveys
Focus groups of clients, health workers
Client exit interviews
In-depth Interviews of clients, community
leaders, health workers
These non-routine methods are useful for baseline
and endline data collection and “spot checks”
during a program
49
Who Is Responsible for Data?
 First, the client provides accurate information
 The community volunteers and leaders maintain
a village register and other forms
 Service providers keep records on the client’s
card and in the register
 The staff in charge of each facility summarize
data regularly
 District, state and NMCP M&E officers collect
and analyze data, and give feedback to service
providers and communities
50
Demonstrating How to Fill Out
the Data Collection Tools for HMM
 HMM/community daily case register
 Exercise on filling out the community daily case
register
 Referral form
 Exercise for filling out the referral form
(At this point the facilitators will provide samples of
actual data collection forms from your country for
review and practice)
51
Summary/Review
 What are the
following?
 Data
 Why do we need it?
 Monitoring
 What are its sources?
 Evaluation
 Who is responsible for
 Indicators
it?
 How can we improve
it?
 How can we use it?
 Data
52