Slide 1

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 2

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 3

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 4

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 5

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 6

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 7

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 8

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 9

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 10

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11


Slide 11

XML-publication in Finnish
Labour Force Survey (LFS)

ESTP training course on “Data Dissemination and
Publication of Statistics” Madrid, 3.-5.3.2008
Kalle Sinivuori ([email protected])

Finnish Labour Force Survey
It is a continuous panel survey based on a sample of about
12,000 persons per month.
 Sampling units : individuals
 Field interviewers collect the LFS data using computer aided
telephone interviews (Blaise)
 The data is available approximately three to four weeks from
the reference time period.
 Periodicity of the results: monthly, quarterly and annually
 Monthly press release, www-tables and -figures, pdf- and
paper publication.


Kalle Sinivuori

5.3.2008

2

Revision of the publication process (LFS) background


Revision of the statistical production system in 2002-2006
=> Shift from the oldest technology to the latest technology
=> From mainframe to open environment



Need to change the publication process as well



Old publication process was ‘clumsy’ : troublesome to
update and based on mainframe.

Kalle Sinivuori

5.3.2008

3

Revision of the publication process in context of
Statistics Finland


Production model project in years 2003-2006
=> New production model of Statistics Finland



New XML-based publication process was defined and tools
for new publication process were selected (=> but not
implemented).



CoSSI: Common Structure of Statistical Information
=> Based on fact: statistical information has a certain simplifiable and
acceptable universal structure ; www.stat.fi/cossi

Kalle Sinivuori

5.3.2008

4

Reasons to use XML
=> from Final Report of Production model -project
Possibility to add distribution channels without changing the
publication process
 Independence of (certain) software's ~


XML is suitable format for archiving statistical information
 XML makes possible wide-ranging metadata to describe
statistical information and statistical publications
 XML creates good possibilities for integration between
different applications


Kalle Sinivuori

5.3.2008

5

LFS - old dissemination process
Automatical
publishing

Excel

-manually

and
- automaticly
Statistical
application

-Timer
controlled

(Monthly & quarterly publ,
publication tables...)

and
- automaticly
Publication
editor

Word,
Excel,

StatFin
Database

Web-site

Publication production

Excel
-manually
Mainframe

Stat Build

Database
services

www.stat.fi

Word:

- Conversion
to HTML

FastWeb

HTML

-Timer
controlled
Paper

Conversion
to PDF

Publication

Kalle Sinivuori

5.3.2008

6

What we need(ed)
More and better metadata
 Language versions
 All information in a single file
 Archiving
 Automatical conversion to different dissemination channels
 Structured searches
 To add new dissemination channels


Kalle Sinivuori

5.3.2008

7

/ XML based dissemination process –
XML and PC-Axis
.PX
.PX
.PX

PX-Web:
PC-Axis tables

.PX

FastWeb-XML

PX-Edit -> PX&CoSSI

Conversion

SuperStar -> PX&CoSSI

Metadata:
eXist,
XMLdatabase

Database
services

PX-Web

Statistical
application

SAS -> PX&CoSSI

Publishing
and
preview

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

8



LFS New publication process

started from may 2007

Kalle Sinivuori

5.3.2008

9

/ XML based dissemination process –
integration completed
FastWeb-XML

Statistical
application
PX-Edit -> PX&CoSSI

PX-Web:
.xml
matrices
(PXML)

.xml

Metadata:
eXist,
XMLdatabase

Publication
editor

Arbortext
Monthly & quarterly publ,
publication tables...)

Database
services

PX-Web

Conversion

SuperStar -> PX&CoSSI
SAS -> PX&CoSSI

Publishing
and
preview

Dissemination
database
eXist,
XMLdatabase

- statistical metadata
- classifications
- processing
metadata

HTML

HTML

PDF

PDF

RSS,
SDMX

RSS,
SDMX

Web-site

www.stat.fi

Printing
house
PDF

Kalle Sinivuori

5.3.2008

10

Future of XML-publishing in Statistics Finland
So far 39 statistics have implemented SAS to XML publishing process, which was developed in LFS.
 During 2008 most of the statistics (about 200 in Statistics
Finland) are implementing xml-publishing.




For technical details:
ask [email protected] (Head of IT-development/ Dissemination)
[email protected] (Technical expert / )
[email protected] (Head of the Data Dissemination sect.)

Kalle Sinivuori

5.3.2008

11