MONITORING AND EVALUATION TOOLS

Monitoring and Evaluation

The National Democratic Institute

INTRODUCTIONS/ GROUND RULES

• • • Introductions Ground Rules Ice Breaker Exercise Photo: Sanja Gjenero, RGBstock.com

MONITORING AND EVALUATION (M&E) TOOLS OBJECTIVES

• • To become familiar with and practice using a range of M&E tools To consider when and how the tools can be employed during the project life cycle

M&E TOOLS TOPICS

• • • • • • • What is M&E?

Gender analysis Project lifecycle Data collection Program design tools Evaluations Other tools Image: www.pixabay.com

KEY TERMS

• • • • • • Indicator Input Output Intermediate result Outcome Objective • • • • • • Goal Target Baseline Impact Gender Stakeholder

M&E DEFINED

• •

Monitoring

— Continuous — Ensures project on track — Identifies problems

Evaluation

— Assesses whether project is achieving objectives — Periodic — Focuses on outcomes and impacts

GENDER AND M&E

• • Gendered priorities, constraints and impact Failure to address gender leads to inefficient and unsustainable results and exacerbates inequities

PROJECT LIFE CYCLE

Phase 4: Project Evaluation -Collect and analyze data -Produce reports -Identify lessons learned Phase 1: Project Design -Identify problem -Develop logframe -Develop M&E plan M&E Life Cycle Phase 3: Project Management -Analyze data and produce reports -Manage activities Phase 2: Project Start-up -Conduct baseline -Establish targets -Establish M&E systems

DATA COLLECTION TOOLS

•

Quantitative methods

– Household surveys – Public opinion polls/surveys •

Qualitative methods

– In-depth interviews – Focus groups – Direct observation – Document review – Participatory assessments

EXERCISE: WHICH TOOLS? WHEN?

• • • Multi-workshop training program for women political activists Objective: Improve the capacity of women to run for elected office Which data collection tools? When?

Photo: Amy Hamelin, NDI

• • • EXAMPLE: WHICH TOOLS? WHEN?

Workshops – Pre-and post questionnaires – Evaluation forms In between events – Direct observation End of project – Focus groups – In-depth interviews – Election results Photo: Amy Hamelin, NDI

PROGRAM DESIGN TOOLS

• • • • • • • • Gender, context and stakeholder analysis Problem and objective analysis Activity plan Indicators Critical assumptions Risk analysis and management Resource plan Results framework Image: www.pixabay.com

GENDER ANALYSIS

• • • • Can be undertaken at any stage but most effective if included in design Systematic way of analyzing different roles and impacts Asks the “who” questions How this will affect women and men?

CONTEXT ANALYSIS

• • • Institutions or vehicles that have positive or negative impact?

Key actors or factors that help or hinder?

Socio-cultural, political and economic environment?

CONTEXT ANALYSIS EXAMPLE Factors that have +/- impact Actors that have +/- impact Institutions that have +/- impact SADC declaration (+) Political parties (-) Gender ministry (+) High unemployment (-) Gender roles (-) Employers (-) Family members (-) Schools (+) Churches (+)

STAKEHOLDER ANALYSIS

• • • • All people who have a stake – Participants – Implementers – Decision-makers – Donors Who are they?

What is their interest? Influence? Power?

Relations with other stakeholders

•

STAKEHOLDER ANALYSIS

Survey all people who have a stake

High Keep satisfied Engage Low High Monitor Keep informed Low POWER

EXERCISE: STAKEHOLDER ANALYSIS

• • • Program to enhance the capacity of the Liberian legislature to represent citizen’s interests Identify stakeholders Plot them on chart Photo: NDI

PROBLEM ANALYSIS

EXERCISE: PROBLEM TREE

• • • • Problem: Low level of women in elected bodies Identify root causes Identify effects Link root causes to their effects Photo: www.pixabay.com

OBJECTIVE ANALYSIS

• • • • • • Links problem analysis to program design Provides visual map Defines scope of program Maps out pathway of change Identifies what will be monitored and evaluated Facilitates planning and management

OBJECTIVE ANALYSIS

• • Core problem reframed as objective Root causes reframed as intermediate results Core problem: Legislature does not effectively represent citizens Root cause: Legislators don’t know how to represent citizen interests to government Objective: Legislature more effectively represents citizens IR: Legislators articulate constituent needs to ministries

INDICATORS

• • • Qualitative versus quantitative

SMART

– Specific – Measurable – Achievable – Relevant – Time-bound Photo: www.pixabay.com

Negative effects reframed become indicators

CRITICAL ASSUMPTIONS

• • • • Factors outside our control Conditions under which program logic will hold true Conditions likely to affect achievement of results Identify at each level of framework

RISK ANALYSIS AND MANAGEMENT

• • Every program entails risk Identify risks and contingency plans/mitigation measures for each level of framework Image: www.pixabay.com

RESOURCE PLAN

• What resources are needed for: – Staff time and materials for setting up an M&E system – Evaluation activities

RESULTS FRAMEWORK

• • • • • Links evaluation strategy to problem analysis Helps identify indicators Aids in M&E Clarifies logic of project Provides roadmap for planning and management

EXAMPLE: FRAMEWORK

Development Hypothesis Results Statement Indicator of Change Goal (impacts)

The Liberian legislature plays a more effective role in government Legislature assumes its constitutional duties of oversight, lawmaking and representation

Objective Intermediate results Outputs

The legislature more effectively represents citizens The quality of legislation responds better to citizens needs MPs put in place mechanisms for outreach CSO fair held Civic groups and MPs have ongoing communication or dialogue Number of CSOs and MPs participating

Activities

Consult with MPs and CSOs about objectives; Train CSOs on presentation; Create CSO database for MPs Number of meetings between NDI, MPs and CSOs; Follow-up on decisions made

Inputs

Time, money, people, etc Number of staff and trainers; Burn rate

EXERCISE: RESULTS FRAMEWORK

Results Statement Indicator of Change Goal (impacts) Objective Intermediate results Outputs Activities Inputs

PROGRAM EVALUATION

• • • • • Types of evaluations: Baseline Mid-term Final Impact Internal versus external

EXAMPLE: ASSESSMENTS Win with Women Political Party Assessment Tool

PROGRAM EVALUATION

• • • • Steps Determine evaluability Select research questions Photo: Sanja Gjenero, RGBstock.com

Identify methods for data collection Develop and test data collection instruments and protocols

EVENT EVALUATIONS

• • • Verbal feedback Evaluation forms Pre- and post-workshop questionnaires

PUBLIC OPINION RESEARCH

• • • Qualitative – Focus groups/in-depth interviews Quantitative – Surveys/polls Combination of both

IN-DEPTH INTERVIEWS

• • • • Key informant/one-on-one interviews Explore thoughts on deeper level More flexible Avoid influence of group dynamics Photo: NDI

FOCUS GROUPS

• • • • • Why and how strongly people hold opinions Can’t project results to wider population Groups formed around common characteristics Six to ten people and moderator Open-ended questions

SURVEYS

• • • • Identical set of close-ended questions Representative sample of population Results projected to larger population Predict future behaviors and trends

Can men represent women?

80 60 40 20 0 Yes No Unsure Female 22.81

54.39

22.81

Male 61.9

26.19

11.9

EXAMPLE: PUBLIC OPINION RESEARCH IN UKRAINE • • • Nearly 50% thought women underrepresented More likely to vote for party with outreach to women Perceive women as better managers Photo: NDI

M&E TOOLS REVIEW

• • • • Monitoring: continuous activity to ensure project is on track Evaluation: assesses impact M&E must address gender Wide range of qualitative and quantitative data collection tools that can be used throughout project life cycle