Transcript The implication of Performance Monitoring and Evaluation

The Presidency, Department
of Performance Monitoring
and Evaluation
Dr Solomon T. Bhunu
Chief Director-Business
Systems & IT
[email protected]
Content of presentation
1. Background
2. Role of DPME
3.Outcomes Approach
4. Way-Forward
2
Background
Services not integrated across agencies ( Direct Impact)
There are no effective systems for coordination,
Lack of incentives for coordination,
Competition between agencies e.g Local & District municipalities,
Lack of rigorous whole-of-government planning
Inappropriate delivery mechanisms in some cases ( Indirect Impact)
Lack of awareness of different models,
Weariness of trying different models,
Inadequate use of new technology,
3
……background
Inappropriate things are being done ( Direct impact)
Lack rigorous planning e.g. poor theory of change,
Lack of focus on key priorities/ drivers of change,
Lack of feedback on what is working or not.
Not using services from other stakeholders appropriately,
4
…….background
Inadequate data to inform management ( Direct Impact)
Lack of culture of using data to improve performance,
Suitable data not available
Staff not performing effectively ( Indirect Impact-outcome 12)
Lack of technical skills in public service
Incentives for not taking risk does not improve implementation
Bad implementation of management system
Lack of accountability ( Indirect Impact)
5
……background
Initial
M&E
thoughts
Post 1994 massive policy overhaul
Gradual awareness of M&E systems
M&E practices inconsistent, quality uneven
•Ensure transparency and accountability
•Promote service delivery improvement
•Ensure compliance with statutory and other
requirements
•Promote the emergence of a learning
culture in the public sector
6
……background
Cabinet
Memo
2005
Principles
Monitoring
Evaluation
Early warning
Verification
Data collection
Analysis
Reporting
7
Cabinet approved an
Implementation plan to develop
an M&E system across the whole
of government
Based on standard,
shared indicators - dashboard
Implementation plan
Composite system
Extractive
Built over time
Roll-out
……background
Period of
hiatus
Assumption that Departmental M&E
systems will allow for extractive M&E
system
 Did not take complexity of inter governmental system
(sub national) into account
 Ignored extensive monitoring work in Treasury on
budget performance measures and accountability cycle
 Uninformed position on institutional alignment &
coordination
 Need specific capacity in place (not recognised)
8
..background
PRESIDENCY
PoA
DPSA
NATIONAL
TREASURY
During
hiatus
Dev Ind
NATIONAL SECTOR DEPT
PoA
OTHER NATIONAL
DEPT
PREMIERS
OFFICE
PROVINCIAL
TREASURY
Human resource info
?planned other info
Financial info
PPI info
Output & outcome info
Policy outcome info
PROVINCIAL DEPARTMENT
•Systems initiated in response to specific needs
•Hard working, but not smart working systems
•Smart analysis
to align information
9
…..background
2007
New direction
 Shift from Task Team to Coordinating Forum, chaired by Pres
 Close cooperation Treasury
 Members: Treasury, DPSA, StatsSA, DPLG(CoGTA),
PALAMA, AG (Premier’s Office, DoE)
 Simultaneous creation of two aspects


Create policy platform that can provide direction to M&E
environment
Produce real M&E products for national oversight
10
The Role of Department of Performance
Monitoring and Evaluation (DPME)
The mandate of the Department of Performance, Monitoring and Evaluation is (DPME)
is derived from Section 85 (2) (c) which states that the President exercises the executive
authority, together with the other members of the Cabinet, by coordinating the functions
of state departments and administrations.
MAJOR MANDATES
1. Outcomes approach
To coordinate and facilitate the development and ongoing performance
monitoring and evaluation (PM&E) of the prioritised whole-of-government
outcomes
To ensure that interventions take place where necessary to unblock
bottlenecks in implementation
2. Performance monitoring of individual departments
Carry out PM&E of individual departments for the President; this includes
assisting the President with the management of performance agreements of
individual Ministers
11
………….the Role of DPME
3. General building of capacity for M&E across spheres, and use of M&E
(M&E of both policy and implementation)
 Custodian of Government-Wide Monitoring Evaluation System (GWMES)
 Build capacity of M&E units across government
 Ensure that capacity of line functions to carry out M&E is developed across
government
 Build management culture of using M&E information to effect change
 Use of all types of M&E information to identify and implement key service
delivery interventions
 Formulate service delivery interventions based on results of all types of M&E
 Ensure that interventions are implemented
12
……..the Role of DPME
Other
MANDATES
 A. Participating in the development of an African M&E network
 Share experiences with other African countries
 Build the network
B. Advisory support to President, Deputy President and Minister (M&E
related issues)
C. Adhoc support to President, Deputy President and Minister (M&E related
Issues)
13
The outcomes approach
(Results Based Management)
14
Monitoring
Traditional monitoring
Focuses on implementation monitoring:

Tracking inputs (resources, strategies),
activities (what actually took place),
and outputs (products or services
produced)

Monitoring how well a programme,
project or policy is being implemented

Assessing compliance with workplace
and budget
Intent: to inform the efficiency of
implementation and to incorporate lessons
learned into decision-making
15

RBM monitoring adds a
new emphasis:
monitoring the
outcomes, i.e. the
consequence of having
delivered outputs
RBM monitoring
Evaluation
Formative evaluations are
conducted during the
implementation of an
initiative and focus on the
efficiency thereof
Are things being
done right?
Summative evaluations
are conducted after the
implementation of a
developmental initiative and
focus on the effectiveness
thereof
Are the right things
being done?
16
….the Outcomes Approach
Inputs
Activities
Outputs
Outcomes
Input -outcome model
Step 4
How much do
we need to
achieve the best
mix of desired
outcomes
•Determine
optimal allocation
of inputs
•Reallocate
Inputs if
Secret
necessary
17
Step3
Step2
Which priority
outputs should
we measure to
see if we are
achieving each
outcome
M o n i t o r i n g and E v a l u a t i o n
Where should
the system focus
in order to
achieve the
outputs
Measure Delivery
of key activities
Measure outputs
to test whether
we are making
progress
Step1
What are the key
outcomes that
Government
wants to achieve
•Derived from
Election
Manifesto
•MTSF
Outcomes-Logic Model
Manage/
influence to
achieve
these
Ultimate
Outcomes
Intermediate
Outcomes
Immediate
Outcomes
Responsibility for
execution
Manage
& budget
for these
Outputs
Activities
Inputs
18
18
….the Outcomes Approach
Four guiding principles
1. Problem Analysis: a clear understanding of the problem
 assumptions about causes and their relationships to effects are explicit
 ensure that the problem is understood from the point of view of the needs and
concerns of the intended beneficiaries
 analytical tools such as problem trees can assist with problem analysis
2. Theory of Change: a clear understanding of key levers of change
 where do we place our focus, what will have the biggest impact?
 ‘theory of change’ based on the best available knowledge about causes and
effects
 state assumptions clearly stated so that we can use evidence from M&E to test it
through experience and
 build reliable knowledge about what works in what circumstances
19
….the Outcomes Approach
Four guiding principles (continued)
4. Intervention Logic: chain of logic
• assumptions about what results must be achieved to achieve the outcome,
• how they will be achieved and
• what resources will be necessary.
5. Clear indicators, baselines & targets: clear basis for monitoring progress
and evaluating results.
• indicators for all levels check progress along the whole chain of delivery.
• each indicator should have a clear baseline, and
• targets and timelines should be clearly defined
• indicators must be measurable
20
….the Outcomes Approach
The Government 12 Outcomes
1.
2.
3.
4.
5.
6.
7.
Quality basic education
A long and healthy life for all South Africans
All people in South Africa are and feel safe
Decent employment through inclusive economic growth
Skilled and capable workforce to support an inclusive growth path
An efficient, competitive and responsive economic infrastructure network
Vibrant, equitable, sustainable rural communities contributing towards food
security for all*
8. Sustainable human settlements and improved quality of household life
9. Responsive, accountable, effective and efficient Local Government system
10. Protect and enhance our environmental assets and natural resources
11. Create a better South Africa, a better Africa and a better world
12. An efficient, effective and development oriented public service and an
empowered, fair and inclusive citizenship**
21
Development of high level outcomes,
outputs, activities and metrics
Ruling Party election
Manifesto: 5 priority areas
Establish Implementation Forum
MTSF: 10 strategic priorities
12 strategic outcomes
(based on consultation process)
Performance
Agreements with
Minister(s)
• Based on outcomes
• High level outputs,
indicators, targets and
activities per outcome
• Request to work
together in
Implementation Forum
to produce a Delivery
Agreement per
outcome
22
Develop and implement detailed
inputs, outputs, activities, metrics
and roles and responsibilities
Negotiate detailed inputs, activities,
metrics and roles and responsibilities
Step 1
(Done)
Delivery Agreements
between
stakeholders
Step 3
(July /
August
2010)
Coordinate implementation
Monitor and evaluate
Step 2
(Done)
Feed back loop to annual
revisions of Delivery Agreements
Step 4
ongoing
….the Outcomes Approach
GWM&E System: Data terrain
Conceptual
clarity
Evaluations
Social, Economic &
Demographic Statistics
Dept
Sector
Registers and Admin data
Programme Performance
Information
Derived information system
23
PERSAL
BAS
DPLG
DWAF
…the Outcomes Approach
GWM&E System: Policy platform
GWM&E
framework
Evaluations
Evaluation
policy
Social, Economic &
Demographic Statistics
Guidelines
Registers and Admin data
Programme Performance
Information
PPI
framework
24
SASQAF
Common
definitions
…the Outcomes Approach
Status quo
Policy Platform formalised
Policy platform established 2007
 Pres: GWM&E framework – system goals, describes constituent components
and institutional location, role of all civil servants, principles
 Guidelines for Premiers Offices
 Guideline on annual M&E plans “ policy vision to operational reality”
 Treasury: Programme Performance Information framework – role of M&E in
plannning and budgeting, link to dept performance plan, AR
 StatsSA: SA Statistical Quality Assurance framework – evaluation of statistics
used by dept
 Public Sector Training Agency curriculum - M&E training secure footing
 DPLG(CoGTA): Draft LG framework
 Common definitions document
 Data Forum project
25
….the Outcomes Approach
GWM&E : Products / methodologies
Evaluations
Dept
Social, Economic &
Demographic Statistics
Development
Indicators
Sector
Registers and Admin data
Programme Performance
Information
26
PoA
…the Outcomes Approach
Status quo
GWM&E : Products / methodologies
Cross cutting M&E products
 Development Indicators – annual 76 Indicators progress of society
 PoA –priority programmes progress reported to cabinet bi-monthly
 Public Mang Watch – 13x HR indicators, 2 Financial, 1 audit, weighted to provide
composite index
 Annual performance plans and service delivery targets - 500x indicators monitored quarterly
 IGR publication of non financial information
 OPSC reports – Annual State of the Public Service Report
 AG office – audits of performance information
 ‘Functionality report’
 GIS platform (uncoordinated)
27
…the Outcomes Approach
PoA
(Program of Action)
 Annual State of the Nation – priorities
are announced
 Web based system 300 users
 Bi-monthly report to Cabinet
 Established a culture of reporting
28
….the Outcomes Approach
29
29
….the Outcomes Approach
30
30
….the Outcomes Approach
31
31
….the Outcomes Approach
Conceptualised POA Workflow
PROVINCIAL
Health
Concurs
&
updates
POA
Education
Human
Settlements
NATIONAL
P
R
E
M
I
E
R
Depts, Forums,
Clusters
PME
POA
trigger
updates
POA
validation
POA
updates
trigger
Concurs
32
Publish
….the Outcomes Approach
International Comparison







Political leadership and championing of M&E
Incentives for M&E
Top down and bottom up convergence
Monitoring before evaluation
Information and data constraints
Capacity building
Ownership of M&E system by line ministries
33
….the Outcomes Approach
Fault lines
 Coordination on M&E design between core national departments
 Different conceptualisation of role of state
 Different paradigms of public sector reform
 Decentralised approach vs national direction on policy monitoring
 Sector experts vs overarching direction
 Naïve view that single decision point in government that will decide on all
indicators
 Deep dialogue approach
34
….the Outcomes Approach
Fault Lines
 Frameworks set out requirements and expectations
 … but workplace culture remains stagnant
 Instances of ‘malicious compliance’
 Information sharing
 Negative reaction to incorrect data
 Govt buying data that should be available from sister departments
 Evaluation policy lag
 Focus on M&E – emphasis on management information
 Some evaluation, not institutionalised
35
….the Outcomes Approach
Fault Lines
 Complexity of sub national government
 Policy making separate from budgeting and implementation
 Guidance to Provincial Premier’s Offices
36
…the Outcomes Approach
Lessons Leant

Appropriate balance between design and evolution of system
 Original design not adequate

Coordination essential
 Not all users have same need
 Necessary tension between line ministries specialist view and overarching
strategic discussion
 Create dialogue, discussion, coordination, good practice
37
….the Outcomes Approach
Lessons Leant




Dual implementation strategy
 Good policy platform
 But… also some real products
M&E is a management system not an IT system
Change management
 Resistance inevitable
 Takes about 3 budget cycles to bed reform down
 Target appropriate people
Art & Science
 Pragmatic response that speaks to political, developmental and
government context
38
Way-Forward
 PME will work with Various stakeholders in the Establishment of an M&E
IT Framework
 POA will further be developed into a robust M&E system
that will be a “system ” of “systems”.
 DPME through various structures e.g. Data forums, M&E networks etc,
etc, will work closely with all partners in achieving success to M&E
culture and improved Service delivery
39
Acknowledgements
1.
SITA...........PoA technical Support
2. TAU (Technical Assistance Unit).........Results Based Management support
3. Dr Sean Philips......... Director General DPME
4. Ms Ronette Engela......Deputy Director General DPME
40
41
Ke ya leboga
Ke a leboha
Ke a leboga
Ngiyabonga
Ndiyabulela
Ngiyathokoza
Ngiyabonga
Inkomu
Ndi khou livhuha
Dankie
Thank you
41