Transcript Slide 1

Distinguishing Inputs, Outputs,
Outcomes and Impacts
www.summerinstitute.eu
Important points to start with
■ Decision making
■ Policy analysis
■ Limited rationality
■ Incentives
■ Process and learning
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Outputs - definition
■ Services or goods which are provided by ministries
and other state organizations to external stakeholders /
beneficiaries (directly or through other organizations)
■ External stakeholders include: general public,
citizens, businesses, NGOs, media and other state
bodies including the minister, other ministries and state
organizations, and the parliament
Examples
■ Provision of prosthetic devices to disabled
■ Provision of schooling
■ Health centre services
■ Analysis and policy advice
■ Laws
■ Inspections
■ Training
Outputs should not be
■ Activities
■ Processes
■ Inter-mediate output
■ Capacity building initiatives
Example: output – schooling
■ Number of graduates from school
■ Activity ■ Process -
teaching
recruitment of teachers
■ IMO – Number of enrolled chidlren
■ Capacity building – increasing
professional qualification of teachers
Consumption or provision?
■ Economic theory view: the delivery of service takes
place between producer and consumer, and therefore
transaction is complete, when service is used (output
consumed)
■ Theatre plays
■ Patients checked
■ Public administration view: for many outputs it does
not mater whether they are consumed or not – the
service should be there in case it is needed
■ Emergency services
■ Prison services
Output characteristics
■ Volume or quantity of provision
■ Timeliness in provision
■ Quality of provision
■ Satisfaction
- surveys and evaluation
■ Comparison – to others or standard
■ Coverage, meting of demand and accessibility
■ Equity
Example: output – provision of prosthetic devices
■ Volume – number of disabled people served
■ Timeliness – service provided in specific time – for
example, one week after request
■ Quality – satisfaction of users of devices
■ Coverage – % of people in need of devices served
■ Equity – people with access difficulties served at
home or transported to the service point
Selectivity
■ Not all output characteristics can/should be used for
all outputs – be selective
■ Focus on key outputs for external users
■ For all other outputs – have at least the volume
Aspects to consider
■ Control of provision
■ Other government interventions
■ Measurability
■ Attribution
■ Users
■ Incentives & gaming
Control of provision
■ Provision of a single output is controlled (directly or
indirectly) by government organization
■ But not its effectiveness (achieving desired policy
objective). Many government outputs make sense only
when used together with other outputs
■ Road safety and reduction of fatal accidents
depends on a group of outputs – quality of road, speed
control, technical state of cars, use of seat belts.
Control of provision (contd.)
Many organizations
Policy
Budgeting
One or more
organizations
Delivery
One organization
Other government interventions
■ To implement policies governments provide not only
goods and services, but also:
■ regulation
■ capital investments
■ social and various other benefits & compensations
■ subsidies
■ If these are not outputs, how do they relate to them?
Other government interventions (contd.)
■ It can be helpful to think of three broad categories
■ goods and services = outputs
■ administered items
■ capital investment
■ mandatory benefits & compensations
■ subsidies and other payments
■ general regulation
■ But remember – most of administred items and
general regulation require outputs
Example – road safety
Outputs
Current
services
Investment
Mandatory
benefits
Subsidies
and other
payments
Road
rehabilitation
Policing &
speed control
Driver
education
■ In budgets, some countries separate outputs / investment /
financing in separate programs while others do not (but
separate using economic classification)
Measurability
■ Some outputs are harder to measure than others due to
Measurement difficulty
(output)
Ambiguity
the nature of organizations providing them
Embassies
Army
Road police
Residential care
Job Counselling
Policy making
ministries
Routine
(process)
Training
institutions
Unemployment
agencies
Attribution
■ Outputs mean something if they can be attributed to
some objective (outcome) – they contribute to achieving
of objective, e.g. Change in outcome
■ Generalized attribution: re-training of unemployed
contributing to reduction of unemployment
■ Specific: re-training of unemployed contributing to
employability of the trained person during the next
year after training
Difficulties in establishing attribution
Clear attribution:
– Broken arm
– Medical service
– Fixed arm
– Services
– Survival
Less clear attribution:
– Cancer
– Earlier diagnosis?
– Better treatment?
– Healthier life style?
– Beneficial effects of
affluence?
Users
■ Different PI users and decision making stages might
have different information needs (and therefore require
different formats of presentation of PI)
Parliament
Government
Ministry of Finance
Minister
Permanent Secretary
Unit Manager
Employee
Output focus
■ Using of PI means analysis and learning rather than
money and / or promotion
Principal – Agent Problem
■ Principals are “owners” and “bosses”, but they lack time
and expertise to do everything
■ Therefore principals hire agents, who have time and
expertise, but are self interested
Medieval
Business
Modern
Principal
King
Shareholder
People
Agent
Lords
CEO
Bureaucrats
Delegated authority
Tax collection
Profit making Service delivery
Principal – Agent Problem (contd.)
■ Agents have one strong advantage over principals information
■ This creates problem called “gaming”, which is based on
two factors:
■ There is information asymmetry between the
principal and agent; and
■ There are wrong incentives for disclosing or
hiding information
Gaming & incentives
■ Bevan and Hood: "reactive subversion such as 'hitting
the target and missing the point' and / or reducing
performance where targets do not apply"
■ Gaming = manipulation with information that might affect
service delivery
■ Incentives = factors (financial or non-financial) that provide
motives for a particular course of action, or count as a reason
for preferring one choice to the alternatives
■ Targets are the most common causes for perverse
incentives
Gaming & incentives (contd.)
Gaming potential
Information asymmetry
High
Low
Loose
Relationship with
decision-making
Tight
Impact of gaming
TARGET
Strategy A
Arrival on
time at final
destination
Strategy B
“False
“Honest Fool”
Champion” or
or no
service
manipulation
manipulation
Strategy C
Strategy D
“Smart Liar”
or
manipulation
of data
“Saboteur”
manipulation
of service
and data
Impact of gaming (contd.)
■ Gaming can result in:
■ Cherry picking – handling of easy cases, patients
etc. at the expense of difficult ones
■ Ratchet effect – slowing down today to be on time
tomorrow
■ Distortions – improving areas where
measurement is dome while neglecting other areas
■ Threshold effect – focusing on 2nd best and
forgetting about the best and worst, as best will be
achieved anyway while worst require too much of an
effort
Dealing with gaming
■ Learning and data improvements
■ Avoiding strong negative incentives
■ Clarify use of PI in advance
■ “Loose” rather than “tight” connection to decisio
making
■ Provide feedback on data relevance and quality
■ Reward those with good PI
■ Do not overload with data – use the key
measures
Dealing with gaming (contd.)
■ Dealing with principal – agent problem at all levels
■ Use of “auditors”
■ National Audit Office
■ Internal audit
■ Encourage ownership at production level
Summarizing about outputs
■ Goods and services
■ Basic element in performance information system
■ Can be measured in different ways, but there are
difficulties
■ Thee are three most common difficulties:
■ Measurability
■ Attribution to objectives (outcomes)
■ Scope for manipulation
■ Loose link between output data and decision making
generally is a better incentive to avoid gaming
Inputs - definition
■ Inputs are resources used in the delivery of goods
and services
■ Labour and other means of service
provision (facilities, computers, teachers,
books etc.)
■ Money is not an input. It s a cost of input
Inputs – output combination
More outputs
V
?
Less inputs
More inputs
?
X
Less outputs
Improving performance
More outputs
Less inputs
Production
possibility
frontier
More inputs
Less outputs
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Outcomes - definition
■ The effects which government is trying to achieve for
the public. They are often expressed as improvement in
the living conditions of people or favourable changes
that contribute to these changes
■ Outcomes are changes in the economic, physical,
social and cultural environments which the state
agency(ies) is trying to influence through provision of
goods and services, general regulation and financing
■ Outcomes are those events, occurrences, or
conditions that are the intended or unintended results of
government actions (OECD)
Examples
■ reduction in maternal / child mortality
■ increase in export
■ decrease of unemployment
■ reduced crime levels
■ improved literacy rate of the population
■ reduction in fatalities from road accidents
■ land productivity improvement
Health
Examples (contd.)
Life expectancy at birth/healthy life
expectancy
Incidence of all cancer/skin cancer
Infant mortality rate
Expenditure on prevention and health
promotion
Education & training
Public expenditure on education as a %
GDP
Education levels
Educational attainment of the adult
population
Adult literacy
Work
Unemployment rate
Employment rate
National income
Examples (contd.)
Real net national disposable income per
capita
Real GDP per capita
Investment as % of GDP, by institutional
sector
Inflation rate
Financial hardship
Inequality of income distribution
Productivity
Labor productivity
Expenditure on R&D
The natural landscape
Threatened species trend
Fish stocks
Forest resources
Biodiversity
Examples (contd.)
Family community and social cohesion
Suicide and drug-induced death rates
Crime
Violent and drug crime
Killed and injured persons in road traffic
Governance, democracy and
citizenship
Voter turnout and informal votes cast
Women in Federal Parliament
Cultural identity
Attendance at cultural events
State Measures and Change
LATVIA
Number of road accidents
Number of injured
Number of fatal injuries
Number of registered cars
2004
2005
change %
48912
47353
-3.2
6416
5600
-12.7
516
442
-14.3
898145
966242
-7.6
State Measures and Change (contd.)
■ the same data can be called state measure or
performance measure
■ it depends whether you attribute policy (outputs,
regulation and finances) to it or not
■ if you do attribute – you are interested in change
influenced by your policy (the question – whether
different policy generates different result, e.g. marginal
changes in result depending on inputs)
■+
-
0
State Measures and Change (contd.)
LATVIA
Number of road accidents
Number of injured
Number of fatal injuries
Number of registered cars
2004
2005
change %
48912
47353
-3.2
6416
5600
-12.7
516
442
-14.3
898145
966242
7.6
■ outputs
■ financing
■ regulation
attribution of policy to change
Different outcomes – results & impacts
Impact or longer term
effects (FO)
Global
objectives
Results or direct &
immediate effects (SO)
Specific
objectives
Outputs
Operational
objectives
Activities
Inputs
Results & impacts defined
■ Results relate to the direct and immediate effect brought
about by a policy or program. They provide information on
changes to, for example, the behaviour, capacity or performance
of direct beneficiaries. Such indicators can be of a physical
(reduction in journey times, number of successful trainees,
number of roads accidents, etc.) or financial (leverage of private
sector resources, decrease in transportation cost) nature
■ Impacts refer to the consequences of the program beyond
the immediate effects on its direct beneficiaries. Two concepts
of impact can be defined. Specific impacts are those effects
occurring after a certain lapse of time but which are,
nonetheless, directly linked to the action taken. Global impacts
are longer-term effects affecting a wider population
Examples
Tax collection
Economy
Justice
inputs
Inputs
Inputs
Inputs
outputs
Tax audits
Regulatory
burden studies
Prisoner
rehabilitation
results or SO
Reduced tax
evasion
Better regulation
Re-integration
into community
impacts or O
Reduced shadow
economy
Enhanced
competitiveness
Reduced repeat
crimes
Results & impacts defined
(contd.)
■ Results = short to medium term outcomes = specific
outcomes
■ Impacts = medium to long term or final outcomes = overall
outcomes
■ Sometimes immediate, intermediate and final outcomes
Single outcomes and indexes
■ Sometimes indexes are used to describe and compare
performance
■ Outcomes from several areas and sometimes even subareas are merged into one index
■ These are of particular use for policy planners
■ For example: the Enabling Trade Index
Example: The Enabling Trade Index 2007/8
MARKET
ACCESS
 Tariff & non-tariff
barriers
 Proclivity or
openness to trade
BOARDER
ADMIN.
T&C
INFRASTR.
 Efficiency of
customs
administration
 Availability &
quality of transport
infrastructure
 Efficiency of
import – export
procedures
 Availability & use
of ICTs
 Transparency of
boarder
administration
BUSINESS
ENVIRONM.
 Regulatory
environment
 Physical security
Example: The Enabling Trade Index 2007/8 (contd.)
Finland
7 / 5.61
Netherlands
11/ 5.51
United States
14 / 5.42
France & Ireland
19 / 5.20
Estonia
25 / 4.89
Lithuania
35 / 4.63
Turkey
38 / 4.53
Latvia
43 / 4.45
Poland
45 / 4.35
Romania
57 / 4.04
Moldova
62 / 3.88
Ukraine
68 / 3.77
Macedonia
81 / 3.58
BiH
89 / 3.47
Kyrgyzstan
109 / 3.03
Example: Tourism Competitiveness Index 2008
REGULATORY
FRAMEWORK
BUSINESS
ENVIRON. &
INFRASTR.
 Policy &
regulations
 Air transport
infrastructure
 Environmental
regulations
 Ground transport
infrastructure
 Safety & security
 Tourism
infrastructure
 Health and
hygiene
 Prioritization of
T&T
 Price
competitiveness
HUMAN,
CULTURAL &
NATURAL
RESOURCES
 Human resources
 National tourism
perception
 Natural & cultural
resources
Example: The Tourism Competitiveness Index 2008(contd.)
United States
5
33
1
12
France
12
13
5
28
Finland
16
7
18
33
Netherlands
19
22
15
25
Ireland
27
14
26
46
Estonia
28
32
25
34
Lithuania
51
57
43
61
Turkey
52
53
63
48
Latvia
53
60
41
77
Poland
63
63
62
60
Georgia
66
55
98
31
Romania
76
87
74
71
Ukraine
78
76
73
89
Macedonia
83
114
82
44
Moldova
95
99
100
83
Kyrgyzstan
102
111
104
84
BiH
104
101
94
108
Problems with outcome measurement
■ Timing
■ Data availability for measurement and transaction costs
■ Outcome (state) measures remain important even in the
absence of policies and programs directed at them
■ Impact of other influences
■ Attribution
Other influences
Influence of other factors
Degree of outcome attribution to outputs
Attribution to outputs
Attribution
Immediate – intermediate - final
Attribution modelling
Outcomes
IMMEDIATE
Knowledge &
understanding
INTER
MEDIATE
Behavioural
changes
FINAL
Changes in
environment
Attribution modelling (cont.)
Outputs
Inputs
Outputs
Target group
Outcomes
Short term
Improved
knowledge
about impact of
farming on
environment
Educational
events
Staff
Finances
Research
Materials
Control
system
Polluting
farmers
Improved
understanding
about the use
of pollutants
Medium term
Systematic
pollution
monitoring in
farms
Long term
Less
polluting
farming
etc.
Site visits /
inspections
Improved
understanding
about the ways
to control
pollution
Changes to
farming
practices
Improved
environment
Summarizing about outcomes
■ Outcomes are changes in the economic, physical, social
and cultural environments which the state agency(ies) is
trying to influence
■ Focus on change (+/-/0) in outcomes is what makes
attribution between outputs and outcomes)
■ Outcomes can be of immediate, medium term and long
term nature. More immediate they are – stronger the
attribution to outputs
■ Outcomes can range from single to more compplex ones
based on some kind of a model
■ Outcomes are particularly useful for policy planning and
evaluation
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Indicators - definition
■ An indicator can be defined as the measurement of
an objective to be met, a resource mobilized, an effect
obtained, a gauge of quality or a context variable
■ Indicators demonstrate how to measure elements of
program that you are interested in or program in
general
Different levels of indicators
Single indicators
Indicators on input
What goes into the system? Which resources
are used?
Indicators on output
Which products and services are delivered?
What is the quality of these products and
services?
Indicators on
intermediate outcomes
What are the direct consequences of the
output?
Indicators on final
outcomes
What are the outcomes achieved that are
significantly attributable to the output?
Indicators on the
environment
What are the contextual factors that influence
the output?
Examples
Description
Indicators
Output
•Construction of a road
Implementation:
•financial: cost, state of progress
•physical: km constructed, level of
progress
Result
•Reduced journey time and •Accessibility (ESS)1
transport costs
•Time savings (in min)
•Cost savings (%)
Specific
impact
•Increased safety
•Increased flows of
persons and goods
•Traffic flows
Global
impact
•Increase in socioeconomic activity
•diversification of production
•net job creation
•Increased regional GDP per capita
•and per occupied person.
Important aspects of use of indicators
■ Output indicators
■ State indicators and baselines
■ Targets
■ Single and ratio indicators
Output indicators
■ Volume or quantity of provision is the most often used
and most relevant to budgeting. However, other output
dimensions can have indicators, too:
■ Timeliness in provision
■ Quality of provision: satisfaction and/or
comparison
■ Coverage, meting of demand and accessibility
■ Equity
Example: technical certification of cars service
■ Volume: X number of cars per years
■ Coverage & demand: 100% every day
■ Timeliness in provision: less than ½
hour waiting, service within 1 hour
■ Quality: satisfaction of car users; quality in
weakest areas improved; online registration etc.
■ Equity: special service for disabled
State indicators and baselines
■ State indicators describe the state of affairs in a given
area of policy concern – unemployment, GDP, mortality,
lack of labour mobility, demand for a service exceeding
supply etc.
■ Baseline indicators refer to the initial value against
which inputs, outputs or outcomes are subsequently
measured.
■ State indicator = photograph of situation; Baseline
indicator = starting point of measuring impact of policy
Targets
■ Show desired level of performance
■ Make sense only if time bound and have baseline
■
Customer satisfaction from a service
increased from 25 to 75% by June 2009
■ Every input, outputs and outcome can have indicators,
but should it?
■ UK PSAs: from 300+ targets in 1998 to 30 targets in 2007
■ Often, if you want to have loose connectionto
accountability and control, simple expected performance
projections can work better than targets
Static and dynamic targets
STATIC
CONCEPT
DYNAMIC
CONCEPT
Value
Needs
Value
Target
Needs
Target
Baseline
Baseline
Time
Single and ratio indicators
Ratio indicators
Efficiency
Costs/Output
Productivity
Output/Input
Effectiveness
Outcome/Objective
(intermediate or final)
Cost-effectiveness
Input/Outcome (intermediate
or final)
These
measures are
valid only to
the extent
that there is a
clear causal
relationship
Single and ratio indicators (cont.)
Needs
Objective
Input
Effectiveness
Productivity
Costs
Cost-effectiveness
Efficiency
Output
Outcomes
Summarizing about indicators
■ Indicators are measures of change
■ Baseline and target data is important for good use of
indicators
■ But be careful – not too many targets. When targets are
set, make sure gaming potential is minimized
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Problems and
needs
Other influences
Policy
input
Structure,
institutional
and
managerial
arrangements
Objectives
Program level
Policy level
Model for structuring performance information
input
Activities
and
processes
Outputs
Short to
medium
term
outcomes
Final
outcomes
Context
Value to the nation
Challenges
Measurement
challenges
Measurement
costs
Relative magnitude
Time delays
Linkages to our
actions
Outcomes
OUTPUTS
IMMEDIATE
INTER
MEDIATE
FINAL
Bottom-up vs. top-down approach in modelling and planning
■ Outputs to outcomes (bottom-up) approach is most
common to budgeting
OUTPUTS
Outcomes
INTER
MEDIATE
IMMEDIATE
FINAL
■ While outcomes to outputs (top-down) – in policy
Policy objective
Options
Instruments
■ However, several OECD countries use the later in
budgeting, focusing on outcomes and leaving connection to
outputs loose
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Budgeting and performance information
■ Performance budgeting has been THE THEME for
several decades now
■ It has not succeeded and no one does it right
■ There are two most discussed topics in this theme:
■ Outputs or outcomes focus?
■ Can everyone do it?
Challenges
■ Informational challenges, e.g. inserting output and
outcome information into budget documentation
■ Decision making challenges, e.g. how much
difference PI makes
Informational challenges
■ Information overload
■ Difficulties to delegate
■ Cost of data
■ Information and knowledge gaps
■ Principal agent problems
Decisional challenges
■ Better information = better decisions – only assumption
■ Hard to attribute action to outcome change. Requires
high quality policy analysis
■ Fixed budgets are not performance budgets
■ But non-fixed budgets can lead to gaming
■ Information asymmetry and principal agent problems
■ Time delays
Outputs or outcomes?
outcome
s
outputs
Program
budgeting
Strategic planning
Evaluation and
spending reviews
Attribution
Need for strong policy analysis
Outputs or outcomes? (cont.)
■ Performance budgeting is about using the budget to
promote performance rather than a specific technique such
as program or output budgeting. What all attempts at
performance budgeting have in common is the attempt to
increase the influence of policy analysis, strategic decision
support, performance information in the budget system
(Graham Scott)
■ Appropriate use of performance information by key actors
at each stage of the budget cycle to inform their decisions
concerning resource allocation and to improve efficiency of
resources (World Bank)
Can everyone do it?
■ Get the basic parts of the budget to perform first
■ Willingness and ability to execute voted money, to
control expenditure
■ Sound financial accountability, respect for financial
management procedures, and effective internal
controls
■ Functioning government
■ Implanting of performance information in the budget
process requires conductive environment – values,
minimal degree of informality, less corruption, leadership
etc.
Where to start?
1. Aggregate Fiscal Discipline
How much money you can spend or should
save for the whole of Government?
Macroeconomic policy as in Budget
Memorandum
2. Allocative Efficiency
2.1 Inter-sectoral
2.2 Intra-sectoral
In what areas we want / need to spend the money on –
health, education, transport?
Strategic policy documents of the Government
In each area, what are the needs and priorities we want
to spend money on – preventive health care, treatment
etc.?
Sector level policy documents
3. Operational Efficiency
How well we spend the money we got? Could we spend
it better?
Monitoring and analysis
Where to start? (cont.)
Aggregate
fiscal
discipline
However, any failure to
successfully achieve macro
fiscal goals will seriously
undermine allocative and
operational efficiency goals
Allocative
efficiency
Operational
efficiency
Importance of PI
What preconditions? (cont.)
■ Baseline projections
■ Trend analysis
■ Attribution to change
What preconditions?
■ Get the basics right (execution, control, accounting,
reporting)
■ Sort out budget process (actors, information flows,
incentives and capacity) – little sense to attempt performance
budgeting within environment where short term political
priorities and deal making prevail over rational analysis of
spending options
■ Focus on performance, and not performance information
■ Focus on process, not only targets. Process is as important
as targets
■ Role and capacity of Ministries of Finance and centres of
Government
■ Holistic rather than patchy approach
Contents
■ Outputs & inputs
■ Outcomes
■ Indicators
■ Logic model
■ Budgeting & PI
■ Future
Future - open questions
■ How does your PI dimension affect your budget process in
terms of actors, information flows between them, capacity and
incentives?
■ How to improve usage of PI in budgetary decision making?
■ How to improve measurement?
■ How to improve quality / user friendliness / usability of
information?
■ How to get politicians to use PI in decision making?
www.cpmconsulting.eu