Transcript Statistics Estonia on its way to improving efficiency UN ECE Seminar on New Frontiers for Statistical Data Collection Geneva, 31.10‒02.11.2012 Tuulikki Sillajõe.

Statistics Estonia on its
way to improving
efficiency
UN ECE Seminar on
New Frontiers for Statistical Data Collection
Geneva, 31.10‒02.11.2012
Tuulikki Sillajõe
Outline
 Organization of the data collection
 Production system as a whole
 Conclusions and future plans
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Central Data Collection Department
since 2004
Data Collection
Department
Manager
Data Collection
Development
Manager
Data
Collection
Service
02.11.2012
Fieldwork
Organisation
Service
Data
Entry
Service
Seminar on New Frontiers for Statistical Data Collection
Collection of administrative data (I)
 Data Collection Department is not responsible for
collection of administrative data
 Data Processing Systems Department
 is responsible for a single entry point for
administrative data
 runs pre-agreed data processing
 makes the data available for statistical domains
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Collection of administrative data (II)
 Methodology Department
 consolidates the needs of statistical domains
 conducts negotiations with the holders of
administrative registers
 organises the conclusion of agreements with
these holders
 is in charge of the description of data in a
central metadata system
 Statistics Estonia used about 100 different
administrative registers (2012)
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Functionalities of ADAM (data collecting
system for administrative data)
 Automatic extraction of detailed personalized data
from administrative sources using
 X-road (data exchange layer)
 ftp, etc.
 Storing data in raw data databases
 Data processing
 coding
 duplicate removal
 Making data available for in-house applications
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Milestones of Data Collection in
Statistics Estonia, 2000‒2012
H
O
U
S
E
H
O
L
D
S
E
N
T
E
R
P
R
I
S
E
S
CAPI
PHC 2011
PAPERLESS
FAILURE
WITH 1st
CAWI
1st MIXED
MODE
CAPI+ CATI
OTHER
CAWI
PILOTS
PILOT PHC
2011
CAWI+CAPI
AGRICENSUS
CAWI+CAPI
CENTRAL
DEPARTMENT
CALL
CENTER FOR
ENTERPRISES
ABOLISHM
ENT OF
REGIONAL
BUREAUS
WEB-BASED
COLLECTION
(eSTAT)
TERMINATI
ON OF
SENDING
Q.-S
PRE-FILLING
WITH ADMINDATA
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Functionality of eSTAT for external
users (I)
 to view the list of statistical questionnaires, which a
particular economic entity has to present to Statistics
Estonia during the current year
 to view deadlines for presenting these statistical
questionnaires
 to order reminders, which notify by e-mail about
upcoming deadlines
 to compile statistical questionnaires, i.e. to fulfil cells on
the web with data or download and upload CSV-files
 to run controls, i.e. check whether they have compiled
the statistical questionnaires required
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Functionality of eSTAT for external
users (II)
 to correct statistical questionnaires immediately
upon compilation thereof
 to submit statistical questionnaires
 to look at all earlier statistical questionnaires
submitted to Statistics Estonia via eSTAT by a
respondent concerned
 to print out a paper copy of a compiled statistical
questionnaire
 to administer users, i.e. to create, change and cancel
rights and access
 to accept or correct one’s contact information
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Functionality of eSTAT for internal
users (I)
 to define statistical questionnaires in the system, i.e.
to describe and add them
 to define controls for statistical questionnaires
described in the system
 to follow the inflow of statistical questionnaires and
send reminders
 to register contacts with respondents
 to see the time and the content of contacts with
respondents
 to see the same information as external user and
help them online
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Functionality of eSTAT for internal
users (II)
 to see and correct contact information of economic
entity who can deliver questionnaires via the system
 to compile statistical questionnaires (e.g. when
receiving them by phone)
 to view and correct statistical questionnaires
compiled by respondents
 to administer external main users and internal users
 to create empty statistical questionnaires in pdfformat for printing out or saving them as a file for
different administrative purposes
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Questionnaires received from economic
entities by channel, 2008–2012
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Pre-filling of questionnaires (I)
 Annual statistical questionnaires for the year 2012,
i.e. for the reference year 2011 data collection, are
prefilled using administrative data
 structural business statistics (EKOMAR)
 agriculture, forestry and fishing
 financial intermediation and activities auxiliary
to financial services and insurance activities
 non-profit institutions
 Data providers have to fill in only the gaps, i.e.
information not available from annual bookkeeping
report
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Pre-filling of questionnaires (II)
 The information from annual reports is preloaded to
eSTAT every hour
 First results
 52% of the questionnaires for structural business
statistics had been pre-filled from annual reports
 80% of the fields were pre-filled, and respondents
had to fill in the remaining 20% (only fields filled in
with a number other than zero are taken into
account)
 Average compiling time of a statistical questionnaire
has been reduced twice (from 3 hours to 1,5 hour)
compared to previous year
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Mixed mode was used for PHC 2011
 Modes of questioning
 e-Census on the web (CAWI) → 66%
 Interview (CAPI)
 Institutions filled in a special questionnaire
 Different data sources
 Population and Housing Census 2000
 Administrative registers
 Data collection from persons
 New software was developed next to eSTAT
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
New software for data collection (VVIS)
 Supports data collection process for various surveys
and census, dividing it into three sub-processes
 Preparation work for data collection
 Data collection on the web (CAWI) and
fieldwork (CAPI)
 Support and management of the whole process
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Applications of VVIS





Questionnaire definition application
Interviewers’ (enumerators’) application
Public web application
Management application
Interfaces with external systems
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Questionnaire definition application
 A tool for preparing questionnaires
 Built on top of Eclipse
 Functionality
 Every survey can use a different, custom data model
 Several questionnaires can be built on top of data
model (e.g. different questionnaires for CAWI and
CAPI)
 Advanced navigation and validation rules within
questionnaires
 Use of common classifications in different surveys
(e.g. occupations: ISCO, education: ISCED, etc.)
 Data model and questionnaires saved in open XML
format
 Multi-language support for questionnaires
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Interviewers’ application
 Stand-alone desktop application can be used in laptop or desktop
computer (Delphi)
 Synchronises all data for offline work over encrypted channel (HTTPS)
 Functionality
 Advanced validation and navigation rules within questionnaire
 Navigation between different questionnaires
 Questionnaires in multiple languages
 Planning / scheduling of work
 Reminders
 Communication with direct supervisor
 Map info for location of subjects, GPS positioning, work
planning
 Overview of general fieldwork progress for interviewers
 Help information for interviewers
 Automatic software updates (distributed from central server)
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Public web application
 Used by survey subjects independently
 Public system, accessible over the web (Java,
Weblogic)
 Functionality:
 Authentication with ID card or through bank link
 Choice between questionnaires assigned to the
subject
 Support for all rules used in questionnaires, the
same questionnaires can be used that are
defined for interviewers’ application
 Background information and help texts about the
surveys and questionnaires
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Management application
 Used by fieldwork management, statisticians, help desk
 Internal web application (Java, Weblogic), accessible from within
internal network
 Functionality:
 Authentication using LDAP or any other authentication method
 Creation of survey object, configuration of methodology,
fieldwork hierarchy and other characteristics of survey
 Role management
 Possibility to work simultaneously with several surveys
 Overview of fieldwork progress, task management
 Definition of milestones (deadlines) for fieldwork organisation
 Help desk functionality
 Data processing tools (e.g. for classification of data)
 Communication (messages from one user to another)
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Interfaces with external systems
 Information database
 Meta-info about surveys
 Background survey information (displayed on the web)
 Classifications used in questionnaires
 Authentication system (e.g. Active Directory)
 Statistical register
 Import of sample
 Pre-filling of questionnaires
 Export of sample (changes in subjects’ information)
 GIS database
 Maps
 Location info of buildings
 Hierarchy of district division
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Generic Statistical
Business Process Model
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Architecture of the information system
Metadata
iMETA
system
KUNDE
Economic
entities
eSTAT
Data
VVIS
collection
Persons
ADAM
Administrative
registers
02.11.2012
Statistical
SRS
registers
VAIS
Processing
eGeostat
Statistical
Analyse
analysis
PX-Web
Dissemination
Users
Census-HUB
Data
Warehouse
Seminar on New Frontiers for Statistical Data Collection
Generic Statistical Business Process Model
1994
Statistical
Business
Register
2002
Statistical
Farm
Register
project started
2011 system for
statistical registers
2001
metadata
management
2011
iMeta
2006
economic entities
project started
2011 persons
1993
2004
project
started
2012
2004
economic
entities
1994
economic entities
project
started
2011
planned
1993
2004
persons
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Sources for efficiency gains of
Statistics Estonia
 Central data collection department, i.e.
standardisation of processes
 Generic office-wide software
 Administrative data instead of survey data if
applicable
 Pre-filling of statistical questionnaires with
administrative data
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Small developments, big efforts
 Reminders sent before the deadline instead of after
the deadline
 Informing economic entities simultaneously about all
the questionnaires they have to fill in next year
 Centralisation of the preparation of questionnaires
from statistical departments to IT Department and
within a few years to Methodology Department
 Standardisation and simplification of instructions
about questionnaires for economic entities
 Creation of a list of input variables
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Next practical steps
 Introduction of CAWI as the main data collection
method for surveys on individuals (2013)
 Introduction of CATI for data collection from both
types of respondents: economic entities and
individuals (step by step, starting from 2012)
 Implementation of generic office-wide software for
other functions than data collection
 Reuse of data within the statistical office
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Strategic directions
 Training of data suppliers (economic entities,
individuals, registers, etc.), incl. about their personal
and public gains
 Closer cooperation between the data collection
function and dissemination function within the
organisation, for better communication with data
suppliers (based on the experience of PHC 2011)
 Simplification of statistical reports (harmonisation of
concepts, deadlines, practices, etc.)
 Development of infrastructure for selling data
collection services
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Further challenges
 Wider use of administrative and commercial data
 Do we need two data collection tools?
 Centralization of data processing function?
02.11.2012
Seminar on New Frontiers for Statistical Data Collection
Thank you for your attention!
[email protected]
02.11.2012
Seminar on New Frontiers for Statistical Data Collection