Transcript PIAAC: delivering the survey in Ireland CSO

PIAAC: Delivering the Survey in
Ireland
Kevin McCormack
Senior Statistician - Social Analysis
www.PIAAC.ie
[email protected]
Overview
•
•
•
•
•
•
Background
Development
Survey Process
Content
Results
Dissemination
Handouts
PIAAC Brochure
Background Questionnaire
PIAAC Factsheet
Outline of international report
Proposed thematic reports
Background
• SLA 2010 - 2013
– DES (and DETE originally)
– CSO
• Statistics Act 1993
– Data collection
– Dissemination
• Timeline
– 2009 – 2013
– Publication Oct 2nd 2013
PIAAC Timeline
October 2013
June 2010 – July 2011
Publication
August 2011 – March
2012
May
2012 – May2013
Main Study preparation:
International
Main Study Data processing,
May 2010
Testing
National
5,983 completed interviews
verification, cleaning
Field
Trial
Translation
March
and
2010
Localisation
Developing IT systems
&
72% Overall responseTable
rate generation
Interviewer
training
tasks
SLA
began
signed
September
2009 Commissioning
laptops
Databases
70% Computer 30% Paper
National report
Sampling,
maps,
addresses
CSO,
DES, DETE
CSO commenced
formal
Training
interviewers
Questionnaire testing
engagement
2009
2010
2011
2012
2013
Survey Development
• Instrument development
• Translation and Localisation
• Scoring testing and re-testing
• IT development
• Virtual Machine
• Screener and Case Management System development
• Field Test
Survey Development
• OECD oversight
– “The Consortium”
•
•
•
•
•
Sampling plans
Interviewer training
Quality Control
Weighting - Westat
Data Adjudication
Survey Process
• Main Study Aug 2011 – March 2012
• Sampling
• Interviewer training
• Data collection
• Data processing
700 Blocks
15
Households
1 Person
Survey Process
• Main Study
• Sampling
• Interviewer training
• Data collection
• Data processing
• Mix of new and experienced interviewers
• 8 day training programme
• OECD specification
Focus on :
•Gaining co-operation
•Use of survey instrument
•Other important elements:
•Nature of survey
• Quality control
•Confidentiality
Survey Process
• Main Study
• Sampling
• Interviewer training
• Data collection
• Data processing
•Encrypted data - file transfer & laptops
•Screener - random adult (16-65)
•Targets and bonus payments
•Incentive payment
•Management information system
•Unlimited call –backs
•Strict weekly monitoring
Quality control
•Taped interviews
•10% of all interviews verified
•Telephone calls, field visits,
Survey Process
• Main Study
• Sampling
• Interviewer training
• Data collection
• Data processing
Industry and occupation coding
Manual
Automatic
Verification against LFS
Paper booklet scoring
Inter-coder reliability design
Anchor booklets (international)
Data cleaning
Screener vs PIAAC questionnaire
Survey Content
• Background Questionnaire
• ‘Core’
– IT competency
– Basic Skills assessment
• Direct Assessment
– Computer Based Assessment
– Paper Based Assessment
Background Questionnaire
A. General Information – Age & Gender
B. Education and Training – Formal Education & Training
C. Current status and work history
D. Current work – Occupation & Earnings
E. Last job - Occupation & Earnings
F. Skills used at work – Frequency of use of workplace skills
G. Literacy, numeracy and ICT skills used at work
H. Literacy, numeracy and ICT skills used in everyday life
I. About yourself – Respondent attitudes and activities
J. Background information – including parents’ occupations
PIAAC Assessment Design
Everyone takes the
BQ
BQ
Aged 16-65
Basic IT Skills
Basic Skills
(Computer)
Assessment
(Computer)
6 tasks: 4 of 6 to PASS
Scrolling, clicking,
Basic Skills
highlighting,
(Paper)
FAIL
•All
those
who
fail the simple
FAIL
4
literacy
tasks
3
literacy
tasks
•Literacy
and
Numeracy
literacy
orOR
numeracy
task
on
<4
of
8
Literacy
Numeracy
4
numeracy
tasks
of 6and
3<3
numeracy
tasks
•Numeracy
Literacy
computer
or
paper
are
routed
booklet
(same as computer
tasks)
•Numeracy
andto
Problem-solving
automatically
the Reading
4
of
8
to
PASS
3 of 6 to PASSbooklet
Components
Selected at random
Reading
Components
Assessment
(Paper)
PIAAC Assessment Design
BQ
Basic IT Skills
Basic Skills
(Computer)
Basic Skills
(Paper)
Assessment
(Computer)
Assessment
(Paper)
Reading
Components
Results
Background Questionnaire
• Economic
• Education
• Social
• Age
• Gender
• Skills use (work & home)
Assessments
• Literacy
• Numeracy
• Problem-solving in
Technology Rich
Environments (PS-TRE)
PIAAC Proficiency Scales
• Proficiency
– Item difficulty
– Item Response Theory & scale scores
– Cut-points (levels 1 to 5)
• IRELAND: Trends from IALS to PIAAC (linked
items)
Trends (IALS vs PIAAC)
IALS
ALL
PIAAC
Prose Literacy
Prose Literacy
Document Literacy
Document Literacy
Literacy (rescaled for trend)
Literacy (rescaled for trend)
Literacy
Numeracy
Numeracy
Quantitative Literacy
Problem solving
Problem solving in TRE
PIAAC Proficiency Scales
Level 1
Level 2
Level 3
Level 4
Level 5
0-225
226-275
276-325
326-375
376-500
Key findings of IALS
• 25% of Irish Adults at lowest levels of ‘literacy’
• Gender: Males more likely to be at higher levels in
Quantitative Literacy – otherwise little difference
between males and females
• Age: Older people more likely to be at lower levels of
ability – related in IALS to educational attainment
• Education: Higher performance related to higher levels
of educational achievement.
Document
Literacy
(IALS
1997)
Quantitative
Literacy
(IALS 1997)
Prose Literacy
(IALS
1997)
Scales
40%
35%
30%
30%
25%
20%
23%
34%
32%
32% 31%
28%
25% 25%
15%
14%
10%
16%
12%
5%
0%
Level 1
IALS 1997
Level 2
Level 3
Levels 4/5
International comparison
IALS 1997
Documents scale
100%
90%
25.1
18.9
11.5
5.8
20
16
16.1
16.2
18.1
19
35.5
80%
70%
60%
50%
40%
30%
45.4
20%
10%
0%
18.2
25.3
9
10.1
6.2
23.7
International comparison
Quantitative scale
IALS 1997
Some data!
Percentage of those 20-24 with Leaving Certificate
or equivalent
Both sexes
Male
Female
87.6
82.8
85.3
77.3
72.0
1996
Source: CSO (Census 1996 & QNHS Q2 2011)
2011
89.8
Source: DES (Education Statistics 2010/2011)
Highest level of Education Attained (Aged 25-64)
31%
28%
25%
1996
2011
24%
23%
22%
15%
12%
10%
10%
Primary (incl. no formal
education)
Lower secondary
Upper secondary
Source: CSO (Census 1996 & QNHS Q2 2011)
Third level: Non-degree
Third level: Degree or higher
Population change 1996 to 2011
Age profile of Population 1996 vs 2011
450000
400000
350000
300000
250000
200000
150000
100000
50000
0
0-4
years
5 - 9 10 - 14 15 - 19 20 - 24 25 - 29 30 - 34 35 - 39 40 - 44 45 - 49 50 - 54 55 - 59 60 - 64 65 - 69 70 - 74 75 - 79 80 - 84 85
years years years years years years years years years years years years years years years years years
and
over
2011
1996
Country of Birth
1996
2011
Outside
Ireland, 7%
Outside
Ireland, 20%
Ireland, 93%
Source: CSO (Census 1996 & 2011)
Ireland, 80%
PIAAC results
International report
• 2nd October 2013
• Two volumes
– Results (I) and Technical (II)
• Distribution of skills (mean
scores)
• International comparisons
• Main focus on LITERACY
• Other domains on the web
• Problem-Solving in TRE for
later thematic releases
National report
• Focus on Irish
performance
• Trends from IALS
• Relevant international
comparisons
Profile 9 – Census 2011
A Study of Education & Skills in Ireland
• November 22nd 2012
Thank you
Questions?