Measuring and understanding public sector productivity

Helen Simpson IFS and CMPO

Outline • Why measure productivity?

• Measuring outputs and inputs • Productivity measurement techniques • Understanding the drivers of productivity

Why measure productivity?

• Productivity: volume of output relative to volume of inputs • Not the only concern from point of view of welfare – quantity and mix of outputs provided by public sector matter • Measures • Changes over time in efficiency with which resources are used • Differences across organisations in efficiency with which resources are used • Use as a comprehensive measure of performance to understand

why

some organisations get more out given what they put in

Measuring outputs

• Want to measure the volume of output • Private sector • Typically have data on

value

of output Value of output = quantity of output x price of output • Prices are serving as weights – in well functioning private sector markets prices reflect marginal benefits and marginal costs • Deflate value of output using a price index • Often far from perfect – e.g. capturing quality change • Note for some private sector services, e.g. banks, problems are akin to those for public sector

Measuring outputs

• Public sector • Want to capture the full range of outputs valued by society • E.g. doctor’s surgery: • Amount and quality of treatment • Equality of distribution of treatment across different types of individuals and illnesses • Ease of registering and getting an appointment • Insurance services • Direct versus indirect or collective provision • Good measure of output of fire service would capture fires extinguished and fire prevention • Collective services outputs = inputs

Measuring outputs

• Public sector • Measures of activities, outputs and outcomes may all provide useful information • Quality • Activities or outputs, number of lessons provided or pupils taught won’t capture quality, but qualifications or the impact of education on employment outcomes and earnings might help • Isolating effects of providers on outcomes • Even if all relevant outputs can be measured how to add up? • In principle weights should reflect marginal social benefits associated with each output

Measuring outputs

• How to weight different outputs together?

• Relative costs? • Often used in practice, e.g. education sector – primary and secondary schools pupil attendance measures weighted together by costs • Attribution of costs • Surveys, implicit valuations?

• Private sector prices?

• Different characteristics of services provided • Different characteristics of users

Measuring outputs

• What if preferences and valuations change over time?

• Measuring performance of an organisation over time • Changes in marginal social valuations captured in increase in volume output to extent that reflect quality improvement, but not changes in valuations that are not driven by the actions of the service provider • ONS consultation • Increased value of public service provision in an economy with rising real incomes?

Measuring inputs

• For many inputs, measurement no greater problem than in private sector • Full-time equivalent employees, value of capital stock (capital services provided),intermediate inputs • Price indices • Some public services can be thought of as involving joint production • Characteristics of individuals using the service can affect measure of

gross

output • Effort expended by an individual may also affect output • Latter can be influenced by provider – a characteristic of provision that will be valued by society • Value-added measures • Or only compare like with like

Measuring outputs and inputs

• Measuring output of public sector services very tricky • Important to capture as much of the output that is valued as possible if productivity measurement is to be informative • Accurate measurement of inputs important too • Taking account of characteristics of individuals using the service could be crucial when comparing across organisations • But measurement of outputs and inputs for private sector not perfect either

Techniques for measuring productivity

• Index measures • Estimating a production or cost function • Measuring productivity relative to an efficiency frontier • Stochastic Frontier Analysis • Data Envelopment Analysis • Partial productivity measures

Index measures

• Relate volume of outputs to volume of inputs • Require weights to add up • Productivity growth: A i t,t  1  M  i  1 p i t y i t  1 N  i  1 w i t x i t  1 M  i  1 p i t y i t N  i  1 w i t x i t • Straightforward to interpret • Adding in more inputs to ‘explain’ away differences in labour productivity • Comparing across organisations: A it  ln(Y it )  β 1it ln(L it )  β 2it ln(M it )  β 3it ln(K it ) • Total factor productivity as a ‘measure of our ignorance’ • Need for accurate data

Estimating a production function

• Uses aggregate measure of output • Comparing across organisations: lnY it  lnA  β 1 lnL it  β 2 lnM it  β 3 lnK it  ε it • Relative total factor productivity as a residual • Add in more characteristics to try and ‘explain’ this residual • Estimation issues – how to accurately estimate parameters • Can also estimate a cost function – doesn’t require aggregation of outputs

Measuring efficiency relative to the frontier

• Techniques have been used to measure efficiency in public sector • Avoid the need to aggregate outputs • Stochastic Frontier Analysis (SFA) • Data Envelopment Analysis (DEA)

Stochastic Frontier Analysis Output

SFA frontier E B A C D F

Input

Stochastic Frontier Analysis

Output

SFA frontier B A C noise { E } noise D F

Input

Stochastic Frontier Analysis

Output

SFA frontier B A C noise { E

}

} noise inefficiency D F

Input

Data Envelopment Analysis

Output DEA frontier E B A C D F Input

Data Envelopment Analysis

Output DEA frontier E B A C inefficiency D F Input

SFA and DEA

• How robust are these techniques?

• SFA – productivity rankings may turn on assumptions made about shape of frontier, and the distributions of noise and inefficiency • DEA – sensitivity of results to measurement error in the data • DEA – possibility of comparing to hypothetical organisations • DEA – the data decide the weights, but how appropriate will these be?

• Empirical studies, e.g. for hospitals, have shown sensitivity of productivity rankings to technique used

Partial productivity measures

• Measure productivity for a single output or outcome, e.g. survival rates following treatment for a particular illness • Might provide very accurate measure of output • But might be concerns about accurate attribution of inputs • Unlikely to be representative of productivity of organisation as a whole • Potentially even more so if output is targeted • Main message: ensure robustness of any productivity scores or rankings to different measurement techniques

Understanding the drivers of productivity

• Approach in private sector studies • Try and ‘explain’ the total factor productivity residual by adding in further information • E.g. firms’ R&D stocks, ownership characteristics • Two issues that have received attention in both private and public sectors: • Effects of competition on productivity • Conditional on inputs (input growth), is output (growth) higher when product market competition is more intense?

• Effects of performance incentives on productivity • Conditional on inputs, do firms that use performance incentives produce more output?

Effects of competition

• Evidence for private and public sectors differs • Not necessarily surprising given differences in incentives and constraints • But are differences in measures used in studies • Key issue is picking up causal effects – some studies exploit reforms that affect degree of competition • Private sector • Positive effect of competition on productivity growth (innovation), e.g. Nickell, (1996) • Positive effect of competition on level and growth rate of productivity, e.g. Disney et al. (2003) • Role of restructuring and exit of relatively poor performers

Effects of competition

• Public sector • Evidence more mixed - education and health (Burgess et al., 2005) • Some for US healthcare sector (Kessler and McClellan, 2000) finds some positive effects • Some for UK healthcare sector finds small negative effects on outcomes (Propper et al., 2004) • Single outcome measures, or look at outcomes and costs separately • Can capture single aspect of provision accurately, but difficult to make direct comparisons to private sector evidence • Expansion, contraction and exit in the public sector?

Effects of performance incentives

• Private sector • Studies of performance-based remuneration (employee or executive share ownership), typically find positive relationship with productivity, (e.g. Conyon and Freeman, 2004) • Tricky to establish causality • Some evidence on incentive schemes such as piece rates - lead to increased productivity, (Lazear, 2000) • Public sector evidence • More in line with that for the private sector • Two studies that find positive effects on outputs, (Burgess et al., 2004 and Atkinson et al., 2004) • Look at multiple output measures • But important in this case to try and use organisation-level measure – diversion of effort away from non-targeted activities

Understanding the drivers of productivity

• Evidence on competition not very consistent across two sectors • Evidence on performance incentives pointing in same direction • But, studies for the two sectors use different measures • Good to try and use (robust!) organisation-level measures for public sector • Especially as public sector presents opportunities for identifying effects • Constraints on behaviour • Reforms

Conclusions

• Accurate measurement of output of public service providers problematic • Capturing all elements of output • Aggregating • Measurement difficulties arising from individuals as ‘inputs’ • Deriving organisation-level measures of productivity could be valuable – if can be demonstrated to be robust • Help to understand what underlies productivity improvements • Reforms to public sector services and use of pilot schemes present opportunities to pick up driving factors