Transcript Slide 1

REPUBLIC OF SOUTH AFRICA
DEPARTMENT OF WATER AFFAIRS
AND FORESTRY
PUBLIC SERVICE M&E NETWORK RELAUNCH
National Treasury
14 May 2007
Monitoring, Evaluation & Reporting – an
essential tool for sustainable service delivery
A case study on water sector
Presented by:
Kalinga Pelpola and Telly Chauke
Index
1.
Background and context
2.
Moving from project, programme to SWAP
3.
Masibambane Programme – Vehicle for sustainable, SWAP to
service delivery and M&E&R
4.
Development of M&E&R to support sustainability of the
Masibambane Programme
5.
M&E&R Indicator and Framework Development
6.
Water and Sanitation Spotcheck Assessments
7.
Quality of Life Evaluations and External Evaluations
8.
Challenges
9.
Conclusion
1. Background and Context
• Pre 1994 Water services
– Fragmented management- homeland and self governing states
– Department of Water Affairs and Forestry (DWAF) – only
responsible for water resources
– Ineffective or lack of LG especially in rural areas
• Post 1994,
– Review legislation leading to WS Act
– addressing services backlog to marginalised communities;
• Water backlog – 14million people
• Sanitation backlog – 24million people
2. Challenges facing the Water Sector in
South Africa (post 1994)
Challenges post 1994 democratisation:
–
Address the services backlogs and social injustices of post apartheid
South Africa;
–
Improve quality of life of previously marginalised communities;
–
Create institutional structures and delivery mechanisms for coordinated
and collaborative approach to services delivery;
–
Provide an enabling environment for stakeholder participation and
alignment of inter-department and inter-government involvement;
–
Establish frameworks for measuring and reporting service delivery
implementation; and
–
Ensure sustainability of services provision.
MULTISECTORAL
Gearingup
Programme Man
Moving project to programme to Sector
wide approaches
MSB II
Project Man
MSB I
CWSS
RDP
1994
1996
Project to
Prog: Focus
2001
2004
SWAP
Focus
2007
Multi-Sectoral
Focus
3. Masibambane Programme – Vehicle for
sustainable, sector wide approach (SWAP) to
service delivery and M&E&R development
•
•
The overall objective of the Masibambane Programme was in line with the
Millennium Development Goals to improve the quality of life and
contribute to poverty eradication;
Masibambane (meaning “let’s work together”), was led by DWAF and
supported by the donor organisations and further supported the
continuous development of a water sector M&E&R strategic framework;
The M&E&R strategic framework is supported by the SWAP.
•
The Masibambane SWAP ensured the collaboration & participation of:
•
– govt. departments – DWAF (sector leader), Dept. of Provincial and Local
Government, National Treasury, Education, Health, Housing;
– Civil Society Organisations (CSO’s); and
– the South African Local Government Association (SALGA)
MSB : SECTOR WIDE APPROACH
SA Strategic
Objectives
MDGs
Need for
Accelerated
delivery
MASIBAMBANE
SWAP
Multi-Sectoral Approach
MOVING FROM PROGRAMATIC TO SWAP
• WHAT IS SWAP?
– Is defined as a way of working together between government
and development partners
– The aim is :
• to broaden Government ownership over public sector policy and resource
allocation decisions within the sector,
• to increase the coherence between policy, spending and results; and
• to harmonise donor requirements into Govt processes thus reducing
transaction costs.
•
– It involves progressive development of a
• comprehensive and coherent sector policy and strategy,
• unified public expenditure framework for local and external resources; and
• common management, planning and reporting framework.
MOVING TO SWAP
SWAPs typically have six components:
GovernmentLed process of
Donor
Coordination
Agreed
process for
harmonization
of systems
Systematic
Mechanism for
Consultation of
beneficiaries
Common
Performance
Monitoring/
reporting
Clear & agreed
Sector policy
And strategy
SWAP
Sector mtef
(all local and
External
Resources)
MOVING TO SWAP (cont)
7 assessments for a SWAP
1. Macro-economic
framework
2. Sector policy and
national
strategic framework
7. Institutions and
capacities
6. Performance
monitoring & client
consultation systems
5. Donor
coordination
systems
SWAP
3. Medium term
expenditure
framework for the
sector
4. Accountability &
public
finance management
systems
Dec-97
People Served
Expenditure
Months
Cost per Capita
Jul-05
1,000
Apr-05
Jan-05
Oct-04
Jul-04
Apr-04
Jan-04
Oct-03
Jul-03
Apr-03
Jan-03
Oct-02
Jul-02
Apr-02
Jan-02
Oct-01
Jul-01
Apr-01
Jan-01
Sep-00
Jun-00
Mar-00
Dec-99
Sep-99
Jun-99
Mar-99
Dec-98
Sep-98
Jun-98
5,000
Mar-98
11,000
Gearingup
BoTT Impact
12,000
Sep-97
6,000
Jun-97
7,000
Mar-97
8,000
1 MILLION th
9,000
Dec-96
10,000
Sep-96
People Served (1000s); Expenditure (R Million);
Cost-per-capita (R-c)
Impact of MSB on expenditure?
National - Expenditure vs. People Served (Water)
11,127.060
8,018.57
4,000
3,000
2,000
720.64
0
4. Development of M&E to support
sustainability of Masibambane Programme
• Sustainability of MSB programme dependent on monitoring,
evaluation and reporting of measurable sector indicators and
targets:
– physical services delivery;
– financial execution; and
– “soft-issue” components (such as gender, training, employment
creation and environment)
• M&E&R targets were defined in the Strategic Framework for Water
Services:
– national for strategy and policy;
– regional for tactical planning; and
– local Water Services Authorities for implementation and operations
The water sector M&E&R strategic framework is embedded
in the context below
International
ALIGNMENT OF WATER SECTOR TARGETS
Millennium Development Goals
SFWS
NWR strategy
Forestry strategy
Strategic Alignment
National
Medium Term Strategic Objectives
(6 Objectives)
DWAF
dplg
Other
SALGA, DoH, DoE, NGO’s etc.
Key Focus Areas
Key Focus Area
Key Focus Area
Projects
Projects
Projects
WS Stakeholder
Specific
Water Sector
Strategic Plan of each Sector Stakeholder
Monitoring Framework
Vision / Mission
Strategic Objectives
Policy
Strategic Level
Key Performance Indicators
Perspective
Key Activities
Project
List
Workplan
Budget
Implementation,
Operation and
Maintenance
Service Delivery Paths
For KFA’s
Tactical Planning Level
Key Performance Indicators
Operational Level
Key Performance Indicators
5. M&E&R Indicator Development
• Incremental development of M&E&R system
• Initially main DWAF indicators focused on measuring services
aimed at backlog eradication:
–
–
–
–
Number of people served with water supply to the standard set;
Number of People served with sanitation facilities;
Number of schools and clinics served with water and sanitation facilities;
Number of bucket toilet systems eradicated.
• From 1994 to 1998 additional indicators were included to monitor
amongst others:
–
–
–
–
employment creation;
gender equality;
Training;
health and hygiene promotion, etc
• In 2003/2004, sector stakeholders developed a comprehensive list
of indicators to monitor and evaluate the effectiveness, efficiency
and sustainability of the programme aligned to the deliverables
described in the Strategic Framework for Water Services.
Incremental development of the
Indicator reporting
• Indicators were added to the report during each
reporting period
• The selection of indicators added was prioritised
according to requirements of the EU and Sector
Departments, as well as availability of data
– Start of with quantitative information (Year 1)
– Expand to qualitative information found commonly within
Departments (Year 2)
– Add water services specific indicators required (Year 3)
Conceptual development of the
quarterly reporting
BENCHMARK REPORT
ACCORDING TO SFWS
Q1
Q4
Q3
Q2
Q4
Q3
Q2
Q1
Q4
Q3
Q2
Q1
Year 1
Year 2
Year 3
Breakdown of development per
Indicator Framework Topic
Indicator framework topics
Access to Basic Sanitation Infrastructure
Quality of Service
Financial Viability
Economic Development and Growth
Civic Society
Resource Management and Environmental impact
Gender and Equity Mainstreaming
Governance, Capacity and Skills
Assets and Appropriate Technology
Community Awareness and Education
Policy, Legislation (by-laws) and Agreements
Base Statistics
Total
Cumulative total
Q1
3
6
0
0
0
0
0
0
0
0
0
0
9
9
Year 1
Year 2
Year 3
Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
4 16 18
6
0
0
4
0
0
0
0
14 23 16
0
0
0
0
0
0
0
0
0
0
0
0
2
2
3
9 10
0
0
0
0
0
0
0
7
7
7 10 10 14
0
0
0
0
7
6
0
0
0
0
0
0
0
0
0 15
0
0
0
0
0
0
0 14
0 11
0
0
0
0
0
0
0
0
0
0
6
4 10
0
0
0
0
0
0
0
0
0
1
5 18
4
0
0
0
0
0
0
5
9
0
0
0
0
0
0
0
0
0
0
8
4
1
0
0
0
0
0
0
0
0
0
0
5
0
0
0
0
18 53 34 28 46 34 38 20 20 10 14
27 80 114 142 188 222 260 280 300 310 324
6. Water Sector M&E&R Framework
Development
• Further support for the development of a sector M&E&R strategy
was contained in a proposal by the Presidency to operationalise
the GWM&ES, wherein it was stated that “…all public service
entities need to undertake their own credible M&E processes that
meet clearly defined standards and deliver information on their
progress and performance.”
• DWAF, as the water sector leader (on the success of the CWSS
and other Departmental M&E&R programmes), embarked on the
development of a robust water sector M&E&R strategy
framework, to address the challenges of a transforming water
sector, the requirements by donor agencies, and the priorities of
the Presidency.
DWAF National
•Water sector
(i.e. DOH etc)
Water &
Sanitation
Projects:
Internal
Spotchecks (SQN)
Monthly
Monitoring
and Reporting
Quarterly
Water Sector
Reports
Independent
annual Spotcheck
evaluations
External
consultations and
assessments
through sector
collaboration
Regional DWAF
M&E Unit (W&S)
Coordinators
Process
Evaluations
Quality of
Life/ Impact
Assessments
Process/programme
efficiency, effectiveness
& sustainability
Independent
Spotchecks (SQN)
Water &
Sanitation
Projects:
•Water sector
(i.e. DOH etc)
Monitoring
Evaluation
Impact on
Beneficiaries
7. Water & Sanitation Monitoring (spotcheck
assessment)
• DWAF has undertaken independent external spot-check
assessments on water and sanitation projects implemented by
the sector piloting the projects executed through the dplg MIG
programme..
• The spot-checks focused on the standards, norms and quality of
water sector projects.
• Rigorous Scientific methodology was employed in the research
study, tools, project sample size.
• The results and analysis of the spot-check assessments is aimed
at guiding the sector in understanding the quality of delivery
mechanisms.
• The 2006/07 spot check project sampled assessed 10% of the
total population of water & sanitation projects in the database
utilized. Due to time and resource constraints, the initial exercise
was conducted by the CSIR.
8. External Impact Assessments (Process
& Quality of Life Evaluations)
• For the purposes of DWAF’s M&E&R processes, quality of life
means a personal statement of the positivism or negativity of
multiple attributes that characterise one's life. It is personal. It is
subjective. It may vary from person to person. It may change from
day to day, but it is the best indication of overall satisfaction or
assessment of services received or not received by the community
members.
• The QoL evaluations aim to investigate the impact of water and
sanitation projects/programmes on beneficiary communities, as well
as assess the efficiency, effectiveness and sustainability of such
projects/programmes. This research is scheduled to commence in
the 2007/2008 financial year.
• The methodology for conducting QoL evaluations will involve:
–
–
–
–
Process evaluations, through one-to-one interviews and focus group sessions;
Desktop study and literature reviews;
Impact assessment on beneficiaries; and
Consolidation of analysis and reporting.
9. Reason behind external Evaluations
• Internal progress monitoring and reporting is performed on a
monthly, quarterly and annual basis by sector stakeholders and
is subjective in nature;
• External independent evaluations, can be conducted by local
and or international experts.
– the effectiveness, efficiency and impact of the programme against the
indicators formulated by the sector.
– The continued relevance and sustainability of the programme is also
evaluated to determine whether the programme should continue or
whether programme funds should be directed elsewhere;
– and to make recommendations for improvement.
• Best practice suggests a combination of both internal and
external assessment of successful outcomes.
10. Challenges
•
KPIs used in the early years of the programme were effective in measuring
whether targets were achieved, but did not cover quality or sustainability
issues;
•
“double counting” of beneficiaries a real problem, specifically between dplg
and Dept. of Housing, who each have its own M&E system;
•
Integration of KPI’s of DWAF and dplg (as well as the other sector players)
has been problematic as each of the systems had a different focus.
•
standard definitions were required;
•
Maintaining DWAF’s M&E function whilst transforming itself from service
delivery implementer to sector regulator;
•
Change management challenges (resistance to change; mutual trust and
sector cooperation)
•
Institutional transformation challenges
•
Operational information acquisition from WSA level.
•
Sector collaboration on M&E&R is not mainstreamed and should be
promoted to facilitate cooperative development of M&E&R frameworks
between sector departments.
11. Conclusion
• M&E&R is a combination of both quantitative and qualitative
analysis of performance.
• Greater emphasis needs to be placed on issues of operability,
specifically in terms of post-construction M&E, with a focus on
“value for money” and enhancement of QoL;
• M&E&R frameworks need to be developmental in nature;
• M&E&R sustainability dependent on consistency, accuracy and
timeous information flow, providing different perspectives
(operational, planning, strategic) to sector roleplayers;
• Establishment of DWAF Regional M&E&R units will facilitate
the development of regional sector M&E&R.