Transcript Variants and Accuracy: results from the Subnational Projection Consultation for England.

Centre for
Demography
Subnational Population Projections
Accuracy and Uncertainty
Presentation to BSPS conference
10 -12 September 2008
2006 based Projections
• Subnational Population Projections (SNPPs)
were published in June
‘Southampton won 4 – 2,
this means that they got exactly the same score
as they did against the same opponents last
year,
when they won 3 – 1.’
A major television sports presenter, on live TV.
Scale of revisions to projections
• SNPPs are scaled to national projections, and
the 06-based national projection for 2011 was
revised upwards by 1.4% (since the previous
2004-based projection)
• Nearly 90% of LAs had SNPP revisions within
3% of the national revision (so between 1.6%
down and 4.4% up)
• 20 LAs on either side of the 3% cut-off,
indicating significant revision to their
projection that was not due to the revised
national constraint
Revisions made to LA projections for 2011
between 04-based and 06-based SNPPs
10.0%
8.0%
6.0%
4.0%
2.0%
0.0%
0
-2.0%
-4.0%
-6.0%
-8.0%
50
100
150
200
250
300
350
Largest upward revisions
• Top five biggest upward revisions, 2011:
–
–
–
–
–
Exeter (9.2%)
Durham (8.2%)
Forest Heath (7.1%)
Charnwood (7.0%)
Southwark (6.9%)
• In all cases the mid-year estimates (MYEs) for 2005
and 2006 were above previous trend
• Noticeable that top three areas include two
universities and an area with a major US military
base, all of which make accurate projections more
difficult
Largest downward revisions
• Top five biggest downward revisions, 2011:
–
–
–
–
–
Kensington & Chelsea (6.6%)
Westminster (4.3%)
Newham (4.1%)
West Wiltshire (3.1%)
Fylde (2.7%)
• In each LA a previously strong trend of growth
tapered off in the 2005 and 2006 MYEs
• Top three areas are all in Inner London, where high
population mobility makes accurate projections more
difficult
Accuracy of Projections:
Comparing projections to reality
"As far as the laws of mathematics
refer to reality, they are not certain;
and as far as they are certain, they
do not refer to reality.“
Albert Einstein
Accuracy of Projections
Key findings
• Assessing accuracy is complicated
– Changes to base (revisions etc.)
– National Projections change
• For most areas projections provide a good
general purpose tool to indicate likely future
population
– Projections based on current trends
• Smaller geographies less accurate
• Further ahead the projection the less
accurate
Assessing Accuracy and RMSE
• Accuracy assessed by comparing projections
to annual mid-year estimates
– Mid-year estimates are also subject to statistical
error
• Root Mean Square Error (RMSE) provides a
useful summary measure of difference
• RMSE gives a single summary statistic
• Gives a figure for the average difference a
‘geography’ will have
• Allows approximation of a confidence interval
Historical Accuracy
Root Mean Square Error compared to 2006 MYEs
6.0%
5.0%
4.0%
GOR RMSE
3.0%
County RMSE
LA RMSE
2.0%
1.0%
0.0%
revised 04based
04-based
03-based
00-based
Year of projection
98-based
96-based
Accuracy at Local Authority level
• Accuracy three years ahead useful indicator
• RMSE at LA district level is 1.4%
– So expect 95% of LAs to be within +/- 2.8%
• Areas of high migration most difficult
• Student areas can be difficult
• And some armed forces areas (but not all)
• Full report, including supporting data
available on website
– www.statistics.gov.uk/snpp
Uncertainty
In this world nothing can be said
to be certain,
except death and taxes.
(Benjamin Franklin, 1789, after Defoe)
Projection Variants
• Consulted on possibility of producing
projection variants in the spring
• 48 separate responses
• Prompted a range of opinions
Two Consultation Response Extracts
• Variant projections would be very useful in getting
policy-makers to more fully appreciate the importance
of considering potentially different future scenarios.
These would hopefully help to enable better planning
of future services.
• Publication of variant data sets to a lower than
national / regional level could be misleading; whilst
valid in their own right, their publication away from
the principal projection has the potential to ‘muddy’
the water and generate misleading interpretations.
These were comments to a consultation and do not represent the views of ONS.
Projection Variants
• Consulted on possibility of producing
projection variants in the spring
• 48 separate responses
• Prompted a range of opinions
• Will intend to publish a full report on 29
September
• Reporting here some headline conclusions
Use and Demand
• Main reasons quoted for needing variants
–
–
–
–
Geospatial planning
Planning for service provision
To illustrate uncertainty
To allow scenario planning
• Large majority of respondents indicated that
producing variants would be valuable
• Conclusion: there is a strong case for producing
subnational variants, subject to resources
A Housing Variant
• Single most cited option
• Very strong demand …
but not universal.
Two Consultation Response Extracts:
Housing Variant
• I do not believe that a proper housing capacity variant
can be produced without local assumptions about
vacancy, sharing, representative rates, etc, etc. If an
independent attempt is made to project population
using migration levels that are believed to be
‘consistent’ with planned development it is almost
certainly a waste of time.
• The housing scenario variant is of vital importance to
Barnet because the council is actively supporting the
increase of the borough’s residential stock by 25 % in
the next 20 years. We would compare ONS figures
with the GLAs products to better understand the
effect of different assumptions.
A Housing Variant
• Single most cited option
• Very strong demand …
but not universal.
• Some experience (especially Chelmer,
POPGROUP/HOUSEGROUP, and GLA)
• General agreement that regional spatial strategies provide
sufficient data
but some concerns.
• Difficulties acknowledged by some
but a strong desire to see these overcome.
• Conclusion: very strong demand and a strong case made, so
ONS intend to examine feasibility further
but due to complexity work is likely to be on a slower path than
standard variants.
Standard Variants
• Strong Demand
• Variants most prioritised
– High and Low Population
– High and Low Migration
– Zero migration (or similar)
• Many others mentioned
– But no consensus as to which is next important
• Conclusion: ONS intend to pursue production of five
variants listed above, resources permitting
Other Issues
• Clear that migration research has to remain
overall priority for ONSCD
• Very strong guidance needed on variants
–
–
–
–
Usage guidance
Status
Implications
Methods and Assumption
And Finally …
• Projections provide a useful planning tool,
but due account needs to be taken of
uncertainty.
If you fail to plan,
then you plan to fail.
(Source unknown)
Contacts
• Subnational Population Projections
Room 2300
ONS
Segensworth Road
Fareham
Hants.
PO15 5RR
• 01329 444669
• [email protected]
• www.statistics.gov.uk/snpp