Transcript Document 7209415

Seminar on Performance Budgeting and Fiscal Transparency,
Rabat, Morocco, April 21-23, 2009
Session 3. Challenges
to Performance Budgeting:
Observations and Lessons
from Korea
Nowook Park
Center for Performance Evaluation and Management
Korea Institute of Public Finance
Contents
1
Description of the current performance budget system
2
Integration of performance information into the budget
and management process
3
Roles of the MOF, line ministries, parliament, and audit office
4
Performance Information : Indicators and Evaluations
5
How to motivate performance? How to engage politicians?
2
1. Description of the current
performance budgeting
system
Background of Performance-based Budgeting in
Korean Government
Need for enhancing efficiency in public spending
Started 4 major
reform programs in
public finance
Expected budget
problems
Increasing public debts
after 1998’s Asian financial
crisis
• Increasing spending on
social welfare programs
due to aging and economic
polarization problems
•
•
•
•
•
4
Medium-term expenditure
framework: Basis for top
down budgeting
Top down budgeting:
autonomy to line ministries
Performance management
system: accountability
Digital accounting system:
program budgeting &
accrual accounting
Efforts towards Performance-Based Budgeting
2000 ~ 2002
Performance Budgeting (Pilot Project)
- Developed Strategic Goals, Performance Objectives
and Performance Indicators
- Designed after GPRA
2003 ~
Performance Goal Management System
- Expanded “Performance Budgeting” to all ministries and agencies
- Annual performance plan and report are required
2005 ~
Self-Assessment of Budgetary Program (SABP)
- 1/3 of major budgetary programs are evaluated every year
- Designed after PART
2006 ~
In-Depth Evaluation
- Selected programs are subject to program evaluation
- About 10 programs are subject to program evaluation
5
Enactment of “National Financial Act”
 “Public Finance Act” was enacted in December, 2006
 To provide a legal basis for 4 major fiscal reform programs
 Contains articles on performance-based budgeting
 Annual Performance Plan and Report become legal
requirements for line ministries/agencies
 SABP and In-depth Evaluation are stipulated.
 It gives stability and continuity which may be a problem
to performance management system.
 Government has less incentives to maintain and improve
performance management system than to introduce it, because
efforts to improve the system is less visible to the public.
6
Framework for Performance-Based Budgeting
In Korea
 Performance Monitoring
 “Performance Goal Management”
- Monitoring based on the performance indicators
- Monitoring unit is program and sub-program
 Program Review
 “Self-Assessment of Budgetary Program”
- Review based on the checklist
- Review unit is usually sub-program
 Program Evaluation
 “In-depth Evaluation of Budgetary Program”
- Program evaluation for selected programs
- Evaluation unit is usually sub-program, but cross-cut issues are
7
sometimes evaluated
Description of “Self-Assessment of
Budgetary Program”
 Ministry of Strategy and Finance (MOSF) reviews selfassessment of programs done by line ministries/agencies
 Budgetary authority provides a standardized checklist for
reporting self-assessment
 The checklist contains questions on program design,
performance management system, program
implementation, and actual performance
 Entire program will be reviewed in three years.
 About 1/3 programs are reviewed each year
• About 300 sub-programs are reviewed each year
 Evaluation unit is usually sub-programs which are unit for budget
allocation by MOSF
8
Contents of Checklist
Design and
Planning
(30)
• Program purpose
• Rationale for government spending
• Duplication with other programs
• Efficiency of program design
• Relevance of performance objectives and indicators
• Relevance of performance targets
Management
(20)
• Monitoring efforts
• Obstacles of program implementation
• Implementation as planned
• Efficiency improvement or budget saving
Results and
accountability
(50)
• Independent program evaluation
• Results
• Satisfaction of citizens
• Utilization of evaluation results
9
2. Integration of performance
information into the budget
and management process
Use of performance information in budget negotiations
Performance
information
MOSF
Spending ministry
Enforce budget cut to
ineffective programs
Allocate budget based
on evaluation results
vs
Introduce performance
information in their
program management
Provide
recommendations for
performance
improvement
11
Performance information in budget statements
Spending
ministries
MOSF
Assembly
Submitted with budget :
annual performance plan
annual performance report
Submitted with budget :
annual performance plan
annual performance report
12
Use of Performance Information from
Performance Budgeting Systems
 Information from monitoring system (performance plan
and report) has not been systematically utilized so far.
 For internal use, they are useful information
 From the viewpoint of central budget authority, they do not provide
useful information
 Information from review system are actively used in
budget negotiation process.
 Programs rated as ineffective are in danger of suffering budget cut
 Its use is systematically built into the budget process
 Information from program evaluation is usually useful
 Their use in budget process depends on the quality of evaluation
and the will of central budget authority
 It is not systematically utilized in the budget process
13
Observations from Program Review Process
 Evaluation results
 Quality of performance information has not improved much
 Programs are showing better results
 Link between evaluation results and budget
 Evaluation results are utilized at every stage of budget
process
 Moving away from incremental budgeting
 Evaluated programs are subject to bigger budget change
compared to other programs
14
Evaluation Results by Total Score
Frequency
300
250
200
150
100
50
0
0
10
20
30
40
50
60
2005
2006
2007
15
70
80
90
100
Total
Evaluation Results by Ratings
Total
Effective
Moderately
Effective
Adequate
Ineffective
Number
555
28
100
340
87
(%)
(100.0)
(5.0)
(18.0)
(61.3)
(15.7)
Number
577
30
94
388
65
(%)
(100.0)
(5.2)
(16.3)
(67.2)
(11.3)
Number
584
66
139
348
31
(%)
(100.0)
(11.3)
(23.8)
(59.6)
(5.3)
2005
2006
2007
16
Evaluation Results by Section
Planning(30)
Total
Score
(100)
Sub
total
(30)
Design
(15)
Performance
Planning
(15)
Management
(20)
Results
(50)
2005
60.1
23.1
13.8
9.3
15.1
21.9
2006
59.9
22.9
14.3
8.6
14.7
22.3
2007
66.0
23.4
14.2
9.2
15.5
27.1
17
Utilization of Evaluation Results
 MPB encouraged ministries/agencies to use the
results in reshuffling budget allocation
 MPB announced 10% budget-cut would be done to
“ineffective” programs, in principle
 MPB submitted evaluation results to the National
Assembly upon their request
 Evaluation results are open to public since 2006
18
-2
0
2
4
6
Use of Performance Information by Agencies (2005)
0
20
60
40
80
perf
95% CI
agencyuse
19
Fitted values
100
-2
0
2
4
6
Use of Performance Information by MOSF (2005)
0
20
40
60
80
perf
95% CI
mpbuse
20
Fitted values
100
Use of Performance Information by Legislature (2005)
21
Link between Evaluation and Budgeting (2005)
( Unit : 100,000won, % )
Rating
‘06 Budget
‘05
Budget
Difference
Ratio
(A)
Agency
(B)
MPB
(C)
Final
(D)
B-A
C-A
D-A
Ratio_1
Ratio_2
Ratio_3
Effective
15,600
24,948
18,955
22,489
9,348
3,355
6,889
0.52
0.38
0.50
Mod.
Effective
92,994
104,335
107,055
105,762
11,342
14,062
12,768
0.32
0.33
0.28
Adequate
208,066
204,473
195,625
201,214
-3,593
-12,441
-6,852
0.10
0.05
0.05
Ineffective
33,081
28,644
29,007
28,505
-4,437
-4,074
-4,576
-0.15
-0.25
-0.19
Total
349,740
362,400
350,642
357,970
22
Link between Evaluation and Budgeting (2006)
( Unit : 100,000won, % )
Rating
‘06
Budget
(A)
‘07 Budget
Difference
Ratio
Agency
(B)
MPB
(C)
Final
(D)
B-A
C-A
D-A
Ratio_1
Ratio_2
Ratio_3
Effective
8,891
9,467
9,337
8,872
575
446
-19
0.11
0.10
0.06
Mod.
Effective
33,156
35,701
35,364
35,654
2,545
2,208
2,498
0.12
0.08
0.09
Adequate
297,180
296,769
290,481
289,969
-411
-6,699
-7,211
0.10
0.04
0.03
Ineffective
11,431
6,039
5,400
5,380
-5,392
-6,031
-6,051
-0.15
-0.24
-0.25
Total
350,658
347,975
340,582
339,875
23
Link between Evaluation and Budgeting (2007)
( Unit : 100,000won, % )
Rating
‘07
Budget
(A)
‘08 Budget
Difference
Ratio
Agency
(B)
MPB
(C)
B-A
C-A
Ratio_1
Ratio_2
Effective
17,112
18,211
17,503
1,099
391
0.18
0.12
Mod.
Effective
266,051
295,121
291,319
29,070
25,268
0.20
0.13
Adequate
146,034
153,026
142,044
6,992
-3,990
0.28
0.15
Ineffective
3,870
3,457
3,066
-414
-805
-0.001
-0.15
Total
433,067
469,815
453,932
24
Moving Away from Incremental Budgeting
Programs have been subject to larger budget change
after evaluation
Coefficient of Variation in Funding Change
(Excluding Programs of which funding change is greater than 200%)
CV
Year
2005
2006
(B04-B03)/B03
(B05-B04)/B04
(B06-B05)/B05
3.1
2.7
9.2
2.7
2.7
-14.3
2.5
3.1
2007
25
(B07-B06)/B06
(B08-B07)/B07
3.9
Cultural Changes among Line Ministries
Budget allocation based on performance is
spreading among line ministries
 Utilization of SABP results in budget requests
 Changing practices of program management
Evaluation activities become active among line
ministries
 Need for program evaluations are recognized and
accepted among line ministries
26
3. Roles of the MOF,
line ministries, parliament, and
audit office
Ministry of Strategy and Finance (MOSF)
 Evaluate line ministries’ program
 Issuing guidelines on performance-based budgeting
to line ministries/agencies
 Evaluating their program’s performance
 Use performance information in budget
formulation
 Encouraging line ministries/agencies to use
performance evaluation results in preparing their
budget requests
 Incorporating the performance information into its
decisions during budget formulation
28
Line ministries/agencies
 Producing performance information in
compliance with central budget authority’s
initiative
 Submitting strategic plans, annual performance plans
and performance reports
 Evaluate their programs based on SABP checklist
 Conduct program evaluation
 Use performance information in budget request
29
The National Assembly
Before the enactment of National Finance Act,
there had not been official role of the National
Assembly
 They had requested evaluation results for
budget deliberation on an ad hoc basis
The National Assembly receives annual
performance plan (from 2008) and report (from
2009) with budget documents
 The National Assembly Budget Office intends to
analyze budget in connection with performance
information
30
The National Audit Office
 Before the enactment of National Finance Act,
there had not been official role of the National
Audit Office
 As an independent audit office within the
Administration, it has not played any official role
 However, they examined the operation of
performance budgeting system and produced report
The National Audit Office is assigned with the
role of verifying annual performance report from
2009
31
4. Performance Information :
Indicators and Evaluations
Performance Indicators
Status
 Slight increase in outcome measures but there are
big room for further improvement
 Outcome performance measures increase by 3.2 percentage
points, while output performance measures decrease by 4.1
percentage points
 More than 40% of programs do not have relevant performance
indicators in SABP
Type of indicators
Total
Input
Process
Output
Outcome
FY 2007
2,016
73
(3.6)
140
(7.0)
837
(41.5)
966
(47.9)
FY 2008
2,037
111
(5.4)
125
(6.1)
761
(37.4)
1,040
(51.1)
33
Performance Indicators
Examples
Good
Bad
Support the IT business
to go abroad
- Customer satisfaction
- Increase the amount of
export($)
Ugly
Student Loans Program Percentage of attainment
- rate of planned goal
- The amount of loans
achievement about
- The number of loans
employment (x)
-rate of employment (o)
Recommendation :
Using outcome measure
ex) Enrollment rates: The
postsecondary enrollment gap
between low- and high-income
high school graduates will
decrease each year.
34
Recommendation :
Using ‘actual value of
attainment’ instead of
‘percentage of achievement’.
(FY 2008 manual of KPART)
Performance Indicators
Problems
 Difficulties to develop outcome measures in some programs
 Program producing results after long period
• Using milestone indicators may help
• However, using different evaluation cycle needs to be
considered
 Successful management of program may not change outcome
measures
• Reduce scope of outcome measures
• Particular program may not worth evaluating
 Data do not exist
• Investment in data is necessary
• Compare cost and benefit of producing new data
35
Performance Indicators
Lessons
It is desirable to develop outcome measures,
but it is not possible for every program
Be aware of some exceptions and apply it
flexibly according to program type
36
Program Evaluations
Status
 Self-Assessment of Budgetary Program
 Review based on the checklist
 Among the checklist, there is a question whether program
evaluation is conducted or not
 It encourage line ministries to conduct program evaluation at
least once in three years
 Budgetary Program Evaluation
 In-depth evaluation for selected programs by the central budget
authority
37
Program Evaluations
Status - KPART
Q. 3-1. Did you conduct comprehensive program evaluation objectively?
• Has your program been evaluated in depth using
verifiable data ?
• Has your program evaluation been conducted by
independent organization ?
ex) external institution, audit office,
internal evaluation expert, etc.
• Does evaluation cover important issues of program ?
cf) admit ‘yes’ in case that evaluation is in progress
or conducted within 3 years latest
38
YES !
Program Evaluations
Problems
 Little experience with program evaluation among
program managers
 Lack of data to prohibits meaningful program
evaluation
 Lack of fund to conduct program evaluation
 Difficulties in maintaining independence of
evaluators
39
Program Evaluations
Example : Bad
Promotion of the intellectual property (IP)
Purpose
• Measuring the impact of program performance
• Realign the way of investment
• Monitoring the performance management system
Does the number of IP center affect the number of industrial
property ?
Issue
Methodology
Estimation based on the fixed effect model
Result :
Couldn’t find the evidence for the regional impact of the IP center !
Obstacle
• Insufficient Data
- absence of data which is needed to conduct the evaluation
- using customer satisfaction (or number of counseling)
instead of the number of industrial property
• Lack of comprehension about program evaluation
40
Program Evaluations
Example : Good
Development of rural area
Purpose
Issue
Methodology
• Evaluate the impact of the program
• Monitoring the program in progress
Does the amount of fund improve the condition of the
undeveloped-area ?
Data analysis and the survey
Result of Effectiveness
Find the evidence for the regional impact !
- There is positive relation between the program and the infrastructure
- In addition, the infrastructure creates value-added business
Result of Management
Find the problems of the management system!
- The management system is not developed and the each manager’s autonomy
is limited
41
Program Evaluations
Lessons
 Plan ahead for data collection
 Data should go hand in hand with program management
 Data management is a part of program management
 Be creative in securing fund for evaluation
 Evaluation should be part of program management
 In budget negotiation, evaluation plan should be presented
42
5. How to motivate
performance? How to
engage politicians?
Institutionalized Incentives for line ministries
 How performance information will be used in budget
preparation is stipulated in guidelines for budget
preparation
 In principle, 10% budget cut for ineffective program is
encouraged to line ministries
 It draws attention from program managers but not from high
ranking decision makers
 Results from SABP is used as a part of evaluation
information for the whole department
 Now high ranking decision makers pay attention
 But their attention is focused more on getting good scores than
on improving programs’ performance
Forms of Information does Matter
 Performance information (performance indicator) at
monitoring level has not been able to draw attention from
decision makers
 They are internally useful information, but not utilized by central
budget authority
 Performance indicators are good starting point for
communication rather than decision making
 Performance information (program ratings) from program
reviews are actively used by central budget authority
 It summarizes various information into simple ratings
 Power of Simplicity!
Politician’s Engagement
 Although there is no formal mechanism for politician’s
engagement, politicians show interest in performance
information
 They requests performance information on an ad hoc basis
 There is not much suspicion on the evaluation results from the
Administration
 It remains to be seen whether politicians will use
performance information more systematically with the
submission of annual performance plan and report to the
National Assembly,
 Publicizing performance information may help to force
politicians to pay attention to PI