Transcript PPT

Big Data and
Macroeconomic
Accounts
Michael Davies,
Division Head,
Macroeconomic Statistics Division,
Australian Bureau of Statistics
September 2014
1
Background
• There is a lot of hype around big data.
• Many are very excited - it seems to solve
many of the problems of getting good data
from households and businesses, saving
a lot of trouble and expense.
2
First Reaction: Methodology
• Initial reaction from methodologists who
have spent their lives ensuring high
quality survey and census data is to
proceed carefully.
• Elegant solutions are proposed to bring
the data up to the quality of survey and
census data.
3
ABS History
• The ABS has a lot of experience with
administrative data - Customs trade data; tax data
for businesses and people; and financial data from
the banking regulator.
• The ABS is doing a lot of work on the combination
of Government datasets e.g. health, welfare,
employment.
• Big demand for social micro data
4
Now
• The new territory is transactional data
from private businesses e.g. supermarket
scanner data.
• This is at a time when companies are
being told that their data are their most
valuable asset.
5
ABS Recent Experience
• In the case of one big respondent - the boss saw
benefits and said "make it happen" then lawyers got
involved - took forever and we had to accept
significant restrictions on what we can do with the
data.
• Can we make them hand over detailed transaction
data under existing laws? Open to debate and
interpretation of laws intended to cover paper forms
collecting a limited amount of aggregate data.
6
The Way Ahead.
• Lots of things to address in parallel
- legal right to data/ contracts if buying
- continuity of supply
- interpretation of data - don't get hung up on
methodology but give priority to measurement aspects.
- consistency over time
• Haste to save money does not help.
• Take care stopping traditional data sources
7
The Way Ahead Cont.
• Need thoughtful and considered approach
balanced with meeting expectations and
cost reductions.
8
Risks
• More noise than signal
• Disrupted supply (depending on
commercial arrangements)
• Bias (data manipulated by supplier)
• High cost to provider
9
For you may palm upon us new for old:
All, as they say, that glitters, is not gold.
John Dryden, 1687, The Hind and the
Panther.
10