Transcript PowerPoint pateikties šablonas (anglų k.)

QUALITY REPORTS
EXPERIENCE OF STATISTICS LITHUANIA
Nadiežda Alejeva
Head, Price Statistics Division
Statistics Lithuania
13 November, 2014
Seminar on Quality Reports
www.stat.gov.lt
CONTENT OF THE PRESENTATION
 Basis for quality reports.
 The role of quality reporting
 Types of the quality reports
 Experience of Statistics Lithuania
 Quality Report structure
13 November, 2014
Seminar on Quality Reports
2
www.stat.gov.lt
BASIS FOR QUALITY REPORTS
 Provisions of the Regulation (EC) No 223/2009 of the European Parliament and of the Council
of 11 March 2009 on the European Statistics laid down in Article 12 regarding statistical
quality:
 To guarantee the quality of results, European statistics shall be developed, produced and
disseminated on the basis of uniform standards and of harmonised methods. In this
respect, the following quality criteria shall apply:
 Relevance,
 Accuracy,
 Timeliness,
 Punctuality,
 Accessibility,
 Comparability,
 Coherence.
 European Statistics Code of Practice adopted by the European Statistical System Committee
on 28th September 2011
 sets the standards for developing, producing and disseminating European statistics.
 ESS Handbook for Quality Reports.
 Technical Manual of the Single Integrated Metadata Structure (SIMS).
 ESS Guidelines for the Implementation of the ESS Quality and Performance Indicators (QPI).
13 November, 2014
Seminar on Quality Reports
3
www.stat.gov.lt
THE ROLE OF QUALITY REPORTING
 Quality is a multi-dimensional concept and include all aspects of how well statistics are fit for
their purpose.
 Quality is of primary importance producing official statistical information.
 Producers of official statistics have to guarantee that statistics are developed, produced and
disseminated on the basis of uniform standards and of harmonised methods.
 Users of statistics are guaranteed assess of appropriate metadata describing the quality of
statistical outputs, so that they are able to interpret and use the statistics correctly.
 In order to meet the standards of the European Statistics Code of Practice, comprehensive
quality reports for the full range of statistical processes and their outputs have to be
prepared.
13 November, 2014
Seminar on Quality Reports
4
www.stat.gov.lt
THE ROLE OF QUALITY REPORTING
 The quality report is one type of documentation for statistical processes and enable the
process quality monitoring.
 Quality report should include the information which enable to identify the statistical
process and outputs quality problems as well as potential improvements.
 Recommendations for quality improvement should be identified and later implemented
based on the quality report.
 It is recommended that quality reports should be updated annually.
13 November, 2014
Seminar on Quality Reports
5
www.stat.gov.lt
STATISTICAL PROCESSES
 Six types of statistical processes:
 Sample survey,
 Census (for example, population, agricultural),
 Using of administrative source(s),
 Statistical process involving multiple data sources (for example, structural business
survey),
 Production of price or other economic index,
 Statistical compilation (for example, national accounts, balance of payments).
13 November, 2014
Seminar on Quality Reports
6
www.stat.gov.lt
TYPES OF THE QUALITY REPORT
 A quality report may be:
 Producer-oriented
 producer-oriented quality reports are prepared for the own internal assessment of
process;
 User-oriented
 user-oriented quality reporting is in general less detailed and focuses on the output
quality,
 in general, user-oriented quality report is a sub-set of the detailed, produceroriented quality report;
 Both.
 A quality report can range from very short and concise to very detailed;
 A quality report can be prepared: for a specific statistical process (survey); for homogeneous
group of processes; for an individual statistical indicator; it may cover only a few specific
attributes and indicators.
 The European Self Assessment Checklist for Survey Managers (DESAP) –covers all aspects
of the statistical process and its outputs, but not in great detail.
 Technical reports and the like where the statistical methodology is described in detail.
 Quality reports may be prepared for every cycle of the statistical process with the periodicity
that is in line with the type and specificity of the statistical process.
13 November, 2014
Seminar on Quality Reports
7
www.stat.gov.lt
TYPES OF THE QUALITY REPORT
 In line with latest developments, several metadata structures have been used for the purpose
of quality reporting.
 The Euro-SDMX Metadata Structure (ESMS) was recommended as reference metadata
report structure:
 ESMS is considered as the concise, user-oriented format of quality reporting;
 ESMS contains a basic level of quality information which is structured along the quality
criteria as defined in the European Statistics Code of Practice and Regulation (EC)
No 223/2009 – the information focuses more on the statistical output rather than the
underlying process itself.
 A more detailed quality reporting structure ESS Standard for Quality Reports Structure
(ESQRS):
 ESQRS is more addressed to the producers of statistics and focuses more on the
statistical process side.
 All statistical concepts of the 2 existing ESS report structures: ESMS and ESQRS have been
included and streamlined in the Single Integrated Metadata Structure (SIMS).
13 November, 2014
Seminar on Quality Reports
8
www.stat.gov.lt
SINGLE INTEGRATED METADATA STRUCTURE (SIMS)
 Single Integrated Metadata Structure (SIMS) has been created in order to:
 streamline and harmonise metadata and quality reporting in the ESS,
 decrease the reporting burden on the statistical authorities by creating the framework
for “once for all purposes” reporting, where each concept is only reported upon once and
is re-usable for other reporting,
 create an integrated and consistent quality and metadata reporting framework where
the reports are stored in the same database,
 create a flexible and up-to-date system where future extensions are possible by adding
new concepts.
13 November, 2014
Seminar on Quality Reports
9
www.stat.gov.lt
EUROPEAN STATISTICS CODE OF PRACTICE
 Standard ESS quality reports are built around the fifteen principles of the European
Statistics Code of Practice.
 Quality is approached along 3 lines:
 Quality of characteristics of the institutional environment (6 principles);
 Quality of the statistical processes (4 principles);
 Quality of the statistical output (5 principles).
13 November, 2014
Seminar on Quality Reports
10
www.stat.gov.lt
EUROPEAN STATISTICS CODE OF PRACTICE
 INSTITUTIONAL ENVIRONMENT QUALITY CRITERIA:

Professional independence
 Mandate for data collection
 Adequacy of resources
 Commitment to quality
 Statistical confidentiality
 Impartiality and objectivity
 STATISTICAL PROCESSES QUALITY CRITERIA:
 Sound methodology
 Appropriate statistical procedures
 Non-excessive burden on respondents
 Cost effectiveness
 STATISTICAL OUTPUT QUALITY CRITERIA:
 Relevance
 Accuracy and reliability
 Timeliness and punctuality
 Coherence and comparability
 Accessibility and clarity
13 November, 2014
Seminar on Quality Reports
11
www.stat.gov.lt
EXPERIENCE OF STATISTICS LITHUANIA
 COMMITMENT TO QUALITY
 In 2007, a quality management system conforming to the requirements of the ISO 9001
standard was introduced in Statistics Lithuania and in 2013 it has been certified for the third
time.
 Introduced and certified quality management system provided a good framework for the
implementation of the European Statistics Code of Practice:
 management rules, structure, processes and responsibilities are clearly defined and
documented,
 performance results are planned and pursued purposefully,
 quality assessment and improvement methods and tools (audits, self-assessment, quality
indicators, user satisfaction surveys, etc.) are implemented in a clear and systematic way.
 Quality management system enables the total quality control.
13 November, 2014
Seminar on Quality Reports
12
www.stat.gov.lt
EXPERIENCE OF STATISTICS LITHUANIA
 The main trends in activity of Statistics Lithuania aimed at quality management and
continuous development in the institution are established in the Quality Policy and
Quality Objectives of Statistics Lithuania.
 Quality policy reflects the key commitments of Statistics Lithuania, set in the Strategy
of Statistics Lithuania for 2013–2017, comprising the Mission statement, Strategic
Objective and main trends together with the improvement actions (measures) and
expected results.
 The guidelines, comprising the description of each stage of statistical production
process, their documentation and quality monitoring are published in the process map
in which all activities of Statistics Lithuania are detailed using process schemes and
supporting documentation.
13 November, 2014
Seminar on Quality Reports
13
www.stat.gov.lt
EXPERIENCE OF STATISTICS LITHUANIA
 The system for measuring and monitoring of Statistics Lithuania performance indicators has
been introduced; every year a plan for monitoring of performance indicators (internal
document) is elaborated.
 The set of regularly monitored performance indicators covers ESS quality indicators in
relation to the European Statistics Code of Practice and indicators related to different stages
of statistical and other processes.

For all performance indicators, critical values are defined. A regular analysis of performance
indicators is quarterly made at the survey and institutional levels.
 From 2007, inspections of statistical surveys are carried out, the purpose whereof – objective
estimation of the quality of preparation of statistics, exchange of experience, indication of
good examples.
 Starting from 2005 Statistics Lithuania performs several types of regular user satisfaction
surveys, assessing user perception from different perspectives: surveys targeted on different
user groups; a survey exploring the image of Statistics Lithuania; surveys focusing on the
improvement of the services of Statistics Lithuania, etc. From 2007, it has been started to
estimate the user satisfaction level.
13 November, 2014
Seminar on Quality Reports
14
www.stat.gov.lt
EXPERIENCE OF STATISTICS LITHUANIA
 Periodic quality reporting (serving both producer and user perspectives) according
to ESS recommendations for Euro SDMX Metadata Structure are prepared and sent
to Eurostat.
 Every year, assessment of statistical processes are performed by survey managers
using a checklist for quality assessment of surveys (DESAP). Every statistical survey
is assessed once per 5 years. Self-assessment exercise is carried out on the initiative
of survey managers.
 Methodologies of all statistical surveys are standardized, described and publicly
available for all users.
 Statistical questionnaires are regularly checked and tested in order to reduce
measurement errors.
 Statistics Lithuania annually prepares the Official statistics releases calendar, which
covers all statistics prepared according to the Official Statistics Work Programme.
13 November, 2014
Seminar on Quality Reports
15
www.stat.gov.lt
EXPERIENCE OF STATISTICS LITHUANIA
 User-oriented Quality reports are prepared by Statistics Lithuania and made available to the
public.
 Metadata (Quality report) with integrated quality indicators are documented according to
the standardized scheme, consistent with the SIMS.
 Metadata (Quality report) includes some background information on the statistical process
and outputs that are the subject of the report.
 An overview of all outputs associated with the process are given.
 References and links are given to other documentation on the methodology of the process and
quality of the outputs.
13 November, 2014
Seminar on Quality Reports
16
www.stat.gov.lt
QUALITY REPORT STRUCTURE
 NAME OF THE STATISTICAL SURVEY (INDICATOR)
 1 CONTACT
 1.1 Contact organisation
 1.2 Contact organisation unit
 1.3 Contact person
 1.4 Contact person, position, area of responsibility
 1.5 Contact person's postal address
 1.6 Contact person's email address
 1.7 Contact person's phone number
 1.8 Contact person's fax number
 2 METADATA UPDATE
 2.1 Metadata last certified
 2.2 Metadata last posted
 2.3 Metadata last update (version, check)
13 November, 2014
Seminar on Quality Reports
17
www.stat.gov.lt
QUALITY REPORT STRUCTURE
 3 STATISTICAL PRESENTATION
 3.1 Description of statistical information (main characteristics, purpose)
 3.2 Classification(s), classification system
 3.3 Sector coverage
 3.4 Definition(s)
 3.5 Statistical unit
 3.6 Statistical population
 3.7 Geographical coverage
 3.8 Time coverage
 3.9 Base period
 4 MEASUREMENT UNIT(S)
 5 REFERENCE (REPORTING) PERIOD
 6 INSTITUTIONAL MANDATE
 6.1 Legal acts and other agreements
 6.2 Statistical data exchange
 7 CONFIDENTIALITY (LEGAL ACTS PROVIDING FOR STATISTICAL DATA
CONFIDENTIALITY)
 7.1 Confidentiality policy
 7.2 Data confidentiality regulations
13 November, 2014
Seminar on Quality Reports
18
www.stat.gov.lt
QUALITY REPORT STRUCTURE
 8 RELEASE POLICY
 8.1 Release calendar
 8.2 Link to the release calendar
 8.3 Release procedure
 9 FREQUENCY OF DISSEMINATION
 10 DISSEMINATION FORMAT
 10.1 Press releases
 10.2 Publications
 10.3 Databases
 10.4 Access to micro-data
 10.5 Other
 11 METHODOLOGICAL DOCUMENTATION
 12 QUALITY MANAGEMENT
 12.1 Quality assurance
 12.2 Quality assessment
13 November, 2014
Seminar on Quality Reports
19
www.stat.gov.lt
QUALITY REPORT STRUCTURE
 13 RELEVANCE
 13.1 User needs
 13.2 User satisfaction
 13.3 Completeness of statistical information
 13.3.1 Degree of completeness of required information
 14 ACCURACY AND RELIABILITY
 14.1 Overall accuracy
 14.2 Sampling error
 14.3 Non-sampling error
 14.3.1. Non-response error
 14.3.1.1 Unit non-response rate
 14.3.1.2 Item non-response rate
 15 TIMELINESS AND PUNCTUALITY
 15.1 Timeliness
 15.2 Punctuality
 15.2.1 Percentage of statistical information released on time
 16 COMPARABILITY
 16.1 Geographical comparability
 16.2 Comparability over time
13 November, 2014
 16.2.1 Length of comparable time series
Seminar on Quality Reports
20
www.stat.gov.lt
QUALITY REPORT STRUCTURE
 17 COHERENCE
 17.1 Cross domain coherence
 17.2 Internal Coherence
 18 ADMINISTRATIVE BURDEN FOR RESPONDENTS (TIME USED BY ONE
RESPONDENT TO FILL IN A STATISTICAL QUESTIONNAIRE)
 19 REVISION OF STATISTICAL INFORMATION
 19.1 Revision policy
 19.2 Revision practice
 20 STATISTICAL DATA PROCESSING
 20.1 Statistical data source
 20.2 Periodicity of statistical data collection
 20.3 Statistical data collection
 20.4 Statistical data validation
 20.5 Production of statistical information
 20.6 Adjustment
 21 COMMENTS, LINKS TO RELATED METADATA
13 November, 2014
Seminar on Quality Reports
21
www.stat.gov.lt
QUALITY INDICATORS
 Indicators useful as user-oriented quality indicators and are included in the Quality
report:
 Data completeness – rate,
 Sampling errors – indicators:
 coefficient of variation
 confidence interval,
 Unit non-response – rate,
 Item non-response rate,
 Time lag – final results,
 Punctuality – delivery and publication,
 Length of comparable time series,
 Data revision – average size.
 Quality indicators are published in the Metadata (Quality report), documented
according to SIMS and publicly available for all users.
13 November, 2014
Seminar on Quality Reports
22
www.stat.gov.lt
QUALITY REPORT
 Survey managers of Statistics Lithuania have to prepare Metadata in line with the Single
Integrated Metadata Structure (Quality report)
 167 Metadata (Quality reports) have been prepared and are available to users.
 Annually, till 1 February, Metadata (Quality reports) have to be revised and updated when
methods and sources have changed.
 Metadata (Quality reports) including quality indicators is updated according to the approved
procedures regulating the preparation, updating and dissemination of Metadata (Quality
reports).
 Information on quality indicators is updated during 3 days after new statistical information
are published. The time of the last update is indicated in the text of the Metadata (Quality
reports).
13 November, 2014
Seminar on Quality Reports
23
www.stat.gov.lt
RESPONSIBILITY
 Methodology and Quality Division is responsible for setting general requirements for the
statistical production, coordination of the introduction of common methodologies and
monitoring their implementation.
 Strategic Planning and Official Statistics Programming Division is responsible for the
organization of definition of performance criteria of Statistics Lithuania and monitoring
their implementation.
 Information Systems Maintenance Division provides technical support for implementing
process quality monitoring. It is responsible for the maintenance of the system for recording
quality characteristics of statistical surveys introduced in Statistics Lithuania.
13 November, 2014
Seminar on Quality Reports
24
www.stat.gov.lt
THANK YOU FOR YOUR ATTENTION
QUESTIONS?
13 November, 2014
Seminar on Quality Reports
25
www.stat.gov.lt