Transcript The Use of Administrative Sources for Statistical Purposes

The Use of Administrative
Sources for Statistical Purposes
Frameworks for the
Access and Use of
Administrative Data
Legal
Frameworks
Data Sources
There are two main sources of data
1. Statistical surveys
2. Administrative records
Legal frameworks are necessary to
enable and control the flow of data
from both sources.
National Legal Frameworks
• Most statistical institutions have
legislation defining their roles and
responsibilities, e.g. a ‘Statistical Law’
• Many new statistical laws introduced in
the last 10 - 15 years
• Some existing statistical laws have
been revised to include provisions
for the use of administrative data
National Legal Frameworks
• Some national legal frameworks give
more powers than others for access
to administrative data
• Historical, political and cultural
factors will have an impact on
national frameworks, therefore these
are not harmonised between
countries
International Frameworks
• The European Union is developing a
legal framework for official statistics,
including references to the use of
administrative data
Case Study - Ireland
Statistics Act 1993:
30. - (1) For the purpose of assisting the
Office in the exercise of its functions under
this Act, the Director General may …
request any public authority to—
(a) allow officers of statistics … to have
access to … any records in its charge, and
(b) provide the Office … with copies or
extracts from any such record,
and the public authority shall … comply with
any such request free of charge.
31. - (2) If any public authority proposes to
introduce, revise or extend any system for
the storage and retrieval of information … it
shall consult with the Director General and
accept any recommendations that he may
reasonably make …
(3) The Director General or any public
authority may request the National Statistics
Board to arbitrate on and, when agreement
cannot be reached, to make
recommendations to the [Prime Minister] for
his decision on proposals made by the
Director General …
Stages of Development
The Eurostat “Handbook of Good
Practices” identifies three key stages
in the development of a legal
framework:
• Foundation
• Consolidation
• Evolution
Stages of Development
• Foundation stage - The statistical
institute starts to get access to
administrative sources
• Consolidation stage – Administrative
data are used to build registers and
produce statistics
Stages of Development
• Evolution stage - The legal
framework encourages greater
co-operation between administrative
data suppliers and the statistical
institute concerning the collection of
data for multiple purposes
How Data Collection
is Changing
80
70
60
50
Administrative
Postal
Interview
40
30
20
10
0
1950
1975
2000
2025
Timing
• National statistical institutes are at
different stages in this process
• Different areas of statistics are at
different stages in this process
• It takes at least 30 years to move
from a population census based on
paper questionnaires to one fully
based on administrative data
Why is This Relevant?
• Legal frameworks should reflect the
desired future position rather than
the current reality, otherwise they
will quickly become a barrier
• Start planning now for the next
revision to the legal framework
Legislation on the Internet
• Austria - www.statistik.at/_institution/bstg2003.pdf
• Ireland - www.irishstatutebook.ie/ZZA21Y1993.html
– See sections 30 and 31
• Norway - www.ssb.no/english/about_ssb/statlaw/statlov_en.html
– See chapter 3-2
• Germany - www.destatis.de/download/ueber/bstag03_e.pdf
– See article 8
Policy
Frameworks
Policy Frameworks
• Government policy on data sharing
– How can the national statistical
institute influence this?
• Codes of practice
– International
– National
International Codes of Practice
• United Nations Fundamental
Principles of Official Statistics
– Principle 5: Data for statistical purposes
may be drawn from all types of sources,
be they statistical surveys or
administrative records. Statistical
agencies are to choose the source with
regard to quality, timeliness, costs and
the burden on respondents
International Codes of Practice
• European Statistical System Code of
Practice
– Principle 2: Statistical authorities must
have a clear legal mandate to collect
information for European statistical
purposes. Administrations, enterprises and
households, and the public at large may be
compelled by law to allow access to or
deliver data for European statistical
purposes at the request of statistical
authorities
National Codes of Practice
• Can reassure the public that data
will only be used for specific and
reasonable purposes
• Should be made available to the
public, e.g. via the internet site of
the national statistical institute.
Organisational
Frameworks
Using Administrative
Sources in Practice
The use of administrative sources is
enabled by a legal framework, in the
context of a policy framework
But - these are not usually detailed
enough to cover all the administrative
arrangements for access and use
What Else is Needed?
• Some sort of agreement:
- Service Level Agreement (SLA)
- Administrative Protocol
- Contract
- Informal or verbal agreement
- Other type of agreement according
to national customs and practices
Formal Agreements
• May not be legally enforceable, but
are seen as a formal commitment
• Set out terms for the provision and
use of data
• Signed by all parties
• Can cover one or more data flows
(e.g. Government contracts)
Informal Agreements
• In a recent inventory, the British
statistical system found it used over
1600 administrative data sets
• Many agreements were informal
• This increases the risk of problems,
particularly if staff in either
organisation move
Group Exercise
What information
should be included in
a formal agreement?
Agreement Contents 1
• Legal basis
• Names of persons transferring and
receiving data
• Detailed description of data covered
• Frequency of data supply
• Quality standards
• Confidentiality rules
Agreement Contents 2
•
•
•
•
•
Technical standards
Provision of metadata
Provisions for payment for supply data
Period of agreement
Contingencies for changes in
circumstances
• Procedure for resolving disputes
Potential Challenges
• Financial agreements need to be
negotiated
• Statistical institute may lack funds for
data
• Suppliers may see data as having a
commercial value
• Suppliers see no benefit to them
Possible Solutions
• Statistical law to give the statistical
institute the right to obtain data
• Statisticians could offer expertise,
analysis of data, or the use of
statistical tools in exchange
• Coordination with other customers in
the statistical institute or government
could help to spread the cost
Quality Issues
• Data need to be ‘fit for purpose’
• Priorities of administrative sources
and statistical offices differ
• Administrative bodies may spend
little or no time on validation work,
particularly on data items that are
not key to their purpose
Quality Conditions
• Have to be acceptable to both sides
• Could include:
– Content of records (e.g. number of
items incorrectly classified should be
less than 2%)
– Frequency and timeliness
– Provision of metadata
• Machinery for dealing with disputes
Technical
Frameworks
Technical Frameworks
• Mechanisms for data transfer
– Paper
– Magnetic media
– Secure on-line connection
• File formats
• Data / metadata standards
– XML
– SDMX
– GESMES
Case Study
Administrative Data for
the British Statistical
Business Register
Legal Framework
• No Statistics Act
• Access to administrative data
controlled by source-specific acts
• Subject to European Union legislation
• Legislation for the independence of
the statistical office may increase
access
Policy Framework
• Government policy favours more
data sharing
• Requirement to cut costs and the
burden on businesses
• Strong data protection legislation
• Civil liberties groups suspicious of
government data sharing schemes
Policy Framework
• Codes of Practice
– Britain applies international standards
– National Code of Practice
– Supported by protocols, e.g. Protocol on
Managing Respondent Load
• Statistical business register is the
gateway for all administrative microdata on businesses
Organisational Framework
• Service Level Agreements with other
government departments
– reviewed annually
– signed at director level
• Statistical analyses / tools provided in
return (e.g. automatic coding software)
• Contracts with commercial data
suppliers
Organisational Framework
• Data suppliers represented on
business register management
committee
• Improving working relationships
across government - ‘Business Data
Sharing Group’
Technical Framework
• Mostly quarterly copies of
administrative registers on CD-ROM
• Government Secure Intranet link for
daily updates from Value Added Tax
• Flat text files
• Metadata as separate tables
Data Sources
Four main administrative sources:
• Value Added Tax (VAT) data
• PAYE (income tax) data
• Company registration data
• Company ownership data (from a private
sector source - Dun & Bradstreet)
Value Added Tax Data
• Framework: VAT Act / Formal agreement
• Frequency: Daily, weekly, monthly
• Timeliness: Delivery within 5 days
• Quality:
Good
• Delivery:
Electronic via secure intranet
• Key content: Name, address, NACE, turnover
Income Tax (PAYE) Data
• Framework: Finance Act / Formal agreement
• Frequency: Quarterly
• Timeliness: Within 1-4 months
• Quality:
Employment - good
Address - moderate
Classification - poor
• Delivery:
CD-ROM
• Key content: Address, NACE, employment
Company Registration Data
• Framework: Contract
• Frequency: Quarterly updates, continuous
on-line access
• Timeliness: Good
• Quality:
Good
• Delivery:
CD-ROM / Internet
• Key content: Legal name, company number
Dun & Bradstreet Data
• Framework: Contract
• Frequency: Annual update, quarterly
supplement, continuous
on-line access
• Timeliness: Good
• Quality:
Variable
• Delivery:
CD-ROM / Internet
• Key content: Enterprise group links, foreign
ownership codes
Group Discussion
What national frameworks
do you have in place?
How effective are they?