Transcript Slide 1
Measuring the quality of regional estimates from the ABS Jennie Davies and Daniel Ayoubkhani Overview • • • • • • • ABS User needs ABS estimation Methods Results Recommendations Outcomes Annual Business Survey (ABS) • ONS’s largest business survey • Sample of ~73,000 UK businesses • Covers agriculture (part), production, construction, distribution and service (part) sectors • Variables (collected and derived) include: Turnover Purchases of goods and services Approximate Gross Value Added Net capital expenditure Annual Business Survey (ABS) • National publication (November and June) 4-digit SIC breakdown • Regional publication (July) 12 UK regions, 2-digit SIC breakdown • Special Analysis system Ad-hoc requests for lower level estimates (eg local authority, 3-digit SIC by employment sizeband etc) • Standard errors and CVs provided with national estimates but not regional User needs • Large range of ABS users Central government, local government and devolved administrations, Eurostat, National Accounts, academia, consultancy firms, media, public • Quality measures therefore important for sound decision making • Lack of standard errors for regional estimates makes it difficult for users to assess accuracy User needs • Recent UKSA Assessment of ABS: “there is insufficient information about methods and quality” “there is no information about the resulting quality of the statistics and no caveats around their use” • QIF project therefore undertaken to develop methodology for calculating standard errors of published regional ABS estimates • Aim to provide users with information about the quality of regional estimates ABS estimation Reporting unit data Ratio estimation National estimates GES Regional estimates ??? National standard errors Regional apportionment model Modelled local unit data Ratio estimation Regional standard errors Small area estimation for small domains 7 Standard error estimation (1) • Assume that the apportioned values are “true” returns • Single stage cluster sampling • Use GES to calculate standard errors Standard error estimation (2) • The regional apportionment model parameters depend on the sample data so are variable • Use bootstrapping to capture the use of the regional apportionment model Bootstrapping • Standard errors capture sampling variability ie how much estimates vary under different possible samples • Bootstrapping re-samples from the original sample to create a new sample • Carry out estimation on new sample • Repeat lots of times • Calculate standard error of the resulting estimates Bootstrapping • Fix the model parameters based on the full sample data quality assure method against GES • Re-fit the regional apportionment model in each iteration include (possible) additional variance from the model Results • Compared the methods in terms of: Differences in standard errors Practical considerations • Results for turnover presented Other variables produced similar results GES vs bootstrap without re-fitting the model • Bootstrapping without re-fitting the regional apportionment model in each iteration should be comparable with estimates from GES Bootstrapping with and without refitting the model • Comparing these to see if the regional apportionment model increases variances • Compared to bootstrap rather than GES to remove additional differences seen before Practical implications • Bootstrapping took ~30 hours • Relied on exact replication of the regional apportionment model Problem for derived variables such as aGVA Recommendations • Use GES to produce standard errors • With caveats that the regional apportionment model is assumed to be fixed Outcomes • Report published on ONS website (Feb 2014) • Methodology approved by ABS Survey Management Board • Standard errors of regional estimates to be published for first time in July 2014 • Another QIF bid submitted to investigate standard errors of small area ABS estimates Would allow for quality measures to be published alongside majority of Special Analysis requests Summary • • • • • • • ABS User needs ABS estimation Methods Results Recommendations Outcomes