Transcript The perils and pitfalls of productivity measurement for English local authorities This work is funded by the ESRC Public Services Programme, grant no.

The perils and pitfalls of
productivity measurement
for English local authorities
This work is funded by the ESRC Public Services Programme, grant no. RES-153-25-00 60
Prof Iain McLean
Dr Dirk Haubrich
University of Oxford
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Outline of talk

Productivity in the context of our project

The CPA framework to assess English local authorities

Pitfall 1: Vicious triangles

Pitfall 2: The limits of self-assessment

Pitfall 3: Inferring productivity from authorities’
expenditure & CPA scores
2
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Productivity in the context of our project
Productivity is only one of many issues addressed by our 12-months research
project, which asks 3 questions about the assessment of local authorities:
1. Is CPA a reliable and valid measure to assess service quality in local
authorities? Or is service quality influenced by external constraints, such
as deprivation, discretionary expenditure, or political control?
2. Assuming that CPA is (or can be made to be) a reliable & valid measure,
can it be used to capture productivity?
3. What, if anything, can England learn from Wales and Scotland?
3
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
The CPA Assessment Framework*
* as applied between 2002 and 2004;
minor changes introduced in 2005.
ABILITY TO IMPROVE
Self-Assessment
and
County Councils’
Corporate Assessment
Total Score
category
score
Council Score Councils’ ‘core services’
Category
Score
Category
Single
2 Tier Councils’
3 1
4 Category
Less than 24 points1
Score
Total Score
Score
Housing
Education
Use of resources
Libraries & Leisure
Environment
Social Care
Benefits
Councils’ 1
Poor
24 to 31
‘Ability
Poor
Less than 24 points
1 2
to
32
to 39OVERALL
Improve’
24 to 29
2 3 Weak
ASSESSMENT
category
- Excellent
ASSESSMENT
More
40
Service
score than
4
N/A
30 to 36
3 - Good
FRAMEWORK
- Fair
More than 36
4 - Weak
Education - Poor
Social Services
Poor
Weak
N/A
2
Weak
Good
Less
than 30Fair
points
3 Excellent
GoodPUBLIC
30Fair
to 37
REPORTING
4 Excellent
Minimum
Maximum
38Good
to 45 Excellent
Score
Score
IMPROVEMENT
More than 45 PLANNING
4
16
4
16
Housing*
2
8
Environment
2
8
Benefits*
Libraries and Leisure
Resources
CURRENT PERFORMANCE ON SERVICES
Inspection Judgements, auditor judgements,
TOTAL SCORE
performance indicators, and governmental
Assessments of councils’ plans
1
2
3
4
4 (using hundreds of PIs
► With such1an elaborate scheme
and auditor/inspectorate judgements), CPA may be said to
1
4
present a rounded assessment of an authority’s performance,
potentially1providing a basis4for productivity measurement.
15
60
Audit Commission 2002
4
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 1 – The vicious triangle (I)
Indicator used in CPA and IMD/SSA:
“% of pupils doing well in external assessment“
Performance on
Indicator increases
CPA
CPA score increases
► greater SSA autonomy
IMD/SSA
IMD ranking decreases
► SSA grant decreases
5
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 1 – The vicious triangle (II)

Implication for local authorities:
If you improve your GCSE results, your CPA score goes up but your SSA
share goes down. Conversely, if GCSE results worsen, SSA share goes up
but your CPA score goes down. Either way you gain a (partly) financial
bonus and suffer a (partly) financial penalty. You should do whichever
carries the larger net bonus.

Implication for central government
There are two contradictory regimes in place, at least one of which must be
abandoned or modified.
6
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 2 – The limits of self-assessment (I)
The CPA approach relies on authorities self-assessing their ability to improve
service delivery in the future. Yet, the first three CPA rounds show that their selfassessment in year t is a bad predictor for their actual performance in t+1 or t+2.
7
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 2 – The limits of self-assessment (II)
 CPA ratings and scores provide policy makers with information about an
authority’s performance in delivering public services.
 These might constitute a suitable output/outcome basis for productivity
measurements, but the partial reliance on self-assessment to produce these
scores and ratings raises doubts about their reliability and validity.
 This is a problem, because the Audit Commission is currently in the process of
re-designing CPA for the rounds 2008 onwards and seeks, with support from
the Local Government Association (LGA), to incorporate additional elements of
self-assessment.
 Our analysis holds even when issues regarding ‘ceilings’ and ‘moving of the
goal posts’ are considered and the model is rerun.
8
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 3 – Inferring productivity from CPA & spend (I)
93.75 +
LAs with high
|
*
|
*
productivity? * *
c
|
*
*
* *
**
*
Y
p
| * X
* ** * ** * *
* * *
*
*
Z **
3
Outliers
a
|
*
***
*** ***
* *
*
*
|
* * * ***** * ****** * ** *
*
p
|
*
* * **** ***
** * ****
* * * *
*
e
|
* * * ******* ********* **** *
*
r
|
*
**** * ****** **** * ** *
* * * *
*
c
|
* *********** ** *
***** * **
e
|
**** **************** ****
*** ** ** * *
n
|
* *** ************ * * ** * ** *
t
|
*
*** *
** *** ** *** *
*
a
|
* * *
*** **
**
*
*
g
|
*
* *
* *
e
|
*
**** *** *
*
*
*
s
|
*
*
*
|
*
LAs with low
|
productivity?
36.6667 +
*
+----------------------------------------------------------------+
.915216
discretionary expenditure 2002-2004
1.33661
9
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Pitfall 3 – Inferring productivity from CPA & spend (II)

With such an elaborate scheme (which uses hundreds of PIs and
auditor/inspectorate judgements), CPA may be said to present a rounded
assessment of an authority’s performance.

Potentially, then, CPA provides a suitable basis for productivity
measurement and allows identifying productive (green) and unproductive
(red) authorities.

However, this is only true
 if CPA is a valid and reliable measure of public service in local
authorities (for doubts, see pitfalls 1 and 2)
 if the formula by which central government calculates the SSA/FSS
accurately assesses needs and (!) resources (see pitfall 3)

But what if not?
10
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
End of Presentation
11
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Backup Slides
12
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
Circularity between CPA & IMD: Preliminary Findings


In the statistical model, the 100+ PIs used in the seven CPA service blocks (as
dependent variable) need to be checked for potential circularity with the 27 indicators
used in the IMD domains (as explanatory variable).
So far, no circularity issue was found for Culture, Environment, & Benefits. Education
is still being checked; Social Serv. & Housing has 4 PIs with moderate/low circularity:
CPA Indicator
IMD Indicator
Behaviour of indicators
Concern for statistical model
1 Soc. Services Adults, A0/A5:
“ Emergency Admissions (Interface)
percentage change on previous year in
total emergency admissions to hospital
per 1,000 population”
Health Deprivation:
“Measures of
emergency
admissions to
hospital”
CPA: ↑ (rise) of indicator (emergency admissions) MODERATE, because
leads to ↑ of CPA score, which leads to ↑ in (a) very similar measures are used on both sides,
financial freedom
and both indicators move into the same
IMD: ↑ of indicator (emergency admissions) leads
direction
to ↑ of level of deprivation, which leads to ↑ in
financial resources.
2 Soc. Services Children, CF/A2:
“Percentage of young children leaving
care aged 16 or over with at least 1
GSCE at grade A*-G”
Education
CPA: ↑ of indicator (educational attainment of care LOW, because
Deprivation:
leavers) leads to ↑ of CPA score, which leads (a) CPA indicator (with a denominator of children
“Average Points
to ↑ in financial freedom
leaving care) is only a small subset of the IMD
Score of pupils at
IMD: ↑ of indicator (educational attainment) leads
indicator (with a denominator of all children);
key stage 4 (GSCE)”
to ↓ of level of deprivation, which leads to ↓ in (b) IMD indicator measures average score, while
financial resources.
CPA indicator has threshold of 1 GSCE.
3 Soc. Services Adults A0/C29,30:
“Adults with physical/mental/learning
disabilities helped to live at home per
1,000 population”
Health Deprivation: CPA: ↑ of indicator (percentage of adults cared at
“Comparative Illness
home) leads to ↑ of CPA score, which leads
and Disability Ratio”
to ↑ in financial freedom
IMD: ↑ of indicator (higher ratio of ill or disabled
individuals) leads to ↑ of level of deprivation,
which leads to ↑ in financial resources.
4 Housing BV 184b:
"The percentage change in proportion
of non-decent Local Authority homes
between 1 April x and 1 April x+1." and
definitions of “decent”.
Living Environment:
“Social and private
housing in poor
condition”
LOW, because
(a) the two indicators do not measure exactly the
same phenomenon.
(b) CPA Indicator may have to be “per 1,000
population” in the denominator, rather than “all
disabled adults”?
CPA: ↑ of indicator (percentage change in nonMODERATE, because
decent homes) leads to ↓ of CPA score,
(a) the two indicators do not measure exactly the
which leads to ↓ in financial freedom
same phenomenon, but
IMD: ↑ of indicator (housing in poor condition)
(b) on the IMD side, the indicator has a
leads to ↑ of level of deprivation, which leads
considerable weight on the IMD composite
13
to ↑ in financial resources.
McLean & Haubrich: The perils and pitfalls of productivity measurement for English local authorities
The CPA Assessment Framework
ABILITY TO IMPROVE
Self-Assessment and
Corporate Assessment
OVERALL
ASSESSMENT
- Excellent
- Good
- Fair
- Weak
- Poor
PUBLIC
REPORTING
IMPROVEMENT
PLANNING
Housing
Education
Use of resources
Libraries & Leisure
Environment
Social Care
ASSESSMENT
FRAMEWORK
Benefits
The CPA Assessment
Framework
CURRENT PERFORMANCE ON SERVICES
Inspection Judgements, auditor judgements,
performance indicators, and governmental
Assessments of councils’ plans
Audit Commission 2002
14