Transcript PIAAC Presentation (November 2013)

PIAAC
=
Programme for the
International Assessment
of Adult Competencies
• Through the assessments and background
questionnaire, PIAAC situated
respondents within a continuum of life-long
learning
From Whence We Came
2011
Programme for the
International Assessment of
Adult Competencies
(PIAAC)
1994
International Adult Literacy
Survey (IALS)
2003
Adult Literacy and Life Skills
Survey (ALL) – in Canada
IALSS
.
4
PISA
• The OECD also coordinates the Programme
for International Student Assessment (PISA)
 Every 3 years, starting 2000
 15 year old students from randomly selected schools
worldwide
 Assessments in reading, mathematics and science
 Students and principals complete background
questionnaires looking at family background, approaches
to learning and school administration
 In some cases, parents also complete a questionnaire
What PIAAC Assessed
 Literacy
• Single scale measuring reading of prose,
document, digital and mixed format texts
• IALSS reported literacy as two separate domains
(prose and document comprehension on two
separate scales)
What PIAAC Assessed
• Numeracy
 Essentially the same concept of numeracy as
used in IALSS
 PIAAC, however, gathered significantly more
data for the construction of its numeracy scale
What PIAAC Assessed
• Problem Solving in a Technology Rich
Environment (PS-TRE)
PS-TRE
• Newly introduced domain
• PIAAC broadens the concept of literacy by including digital
technology, communication tools and networks to acquire and
evaluate information, communicate with others, and perform
practical tasks
• At a basic level, it represents the intersection between
computer literacy skills and the cognitive skills required to
solve problems
• PIAAC assessed adults capacity to use computer tools
(keyboards and mice, file management tools, applications,
and graphic interfaces) to assess the capacity of adults to
access, process, evaluate and analyze information
effectively
• PIAAC employed adaptive testing – a
feature of computer-based testing that
adjusts the difficulty of questions as
participants take the test
• Provides a much more nuanced distinction
in scores than were provided by paperbased predecessors
• If respondents could not (or would not) use
the computer based instrument, they were
provided with a paper-based instrument
that measured literacy and numeracy
• Respondents could score at levels 4/5 in
literacy and numeracy using the paperbased instrument
Main Elements of PIAAC
Direct assessment (skills measured by the survey)
•
Literacy
•
Numeracy
•
Problem solving in technology-rich environments
Background questionnaire (information about respondents)
•
Demographic characteristics
•
Educational attainment and training
•
Employment status and income
•
Social and linguistic background
Module on skills use (information on how skills are used every day)
•
cognitive skills (e.g., engagement in reading)
•
non-cognitive skills (e.g., capacity to work in a team) and communications, organization, and
influencing skills
•
workplace skills (e.g., autonomy over key aspects of work)
The Background
• 157,000 adults 16-65 in 24 countries
 Second round with nine additional countries
 Third round proposed for 2014
• Language of assessment was the official
language(s) of the countries participating
• Sample sizes were determined nationally
• Canada chose a larger sample size to
ensure PIAAC data could be compared to
IALSS
PIAAC Sample By Country
PIAAC Sample in Canada
General
sample
British Columbia
Alberta
Saskatchewan
Manitoba
Ontario
Quebec
New Brunswick
Nova Scotia
Prince Edward Island
Newfoundland and Labrador
Provinces only
Nunavut
Northwest Territories
Yukon
Territories only
TOTAL
Aboriginal
populations
974
947
994
990
2,758
5,311
1,230
1,441
929
1,609
17,183
302
100
607
624
1,168
60
185
463
471
1,119
594
454
359
1,407
18,302
4,568
3,161
Minoritylanguage
populations
Recent
immigrants
649
177
698
647
314
456
740
226
2,115
1,792
Youth
508
508
Total
2,733
1,224
1,601
2,312
5,313
5,911
1,686
1,441
929
1,609
24,759
779
917
830
2,526
2,115
1,792
508
27,285
15
PIAAC Sample in Canada
• Aboriginal status was self-declared, as
determined through other Stats Canada reporting
• Hours and hours of pan-Canadian conference
calls and meetings were required to achieve this
level of partipation
• Conference calls and meetings continue as we start to
work with the data
•
Confidentiality assured. The Ministry of Education
will have access to the full data set and this was
communicated to respondents
The Results …
Computer Based Assessment
Computer Based Assessment
• At 86%, Saskatchewan had the third
highest proportion of the population (in the
world) that engaged in computer based
survey – after Sweden and the
Netherlands.
• The next highest proportion in Canada
was Nova Scotia at 84%
Computer Based Assessment
• Japan had a relatively low rate of
completion via computer, BUT scored
highly in literacy and numeracy
 PS-TRE could not be measured by the PIAAC
paper-based assessment instrument
Saskatchewan is …
• At the OECD average in literacy
• At the OECD average in PS-TRE
• Below the OECD average in numeracy
Saskatchewan …
• Saskatchewan’s overall performance
mirrors Canada’s performance
• Matches the OECD average for people at
the highest proficiency levels
• Has a higher proportion of the population
at the lowest proficiency levels in
numeracy and PS-TRE compared to the
OECD average
Saskatchewan …
• Had 10% of respondents whose first
language was not English and French
 Matches the OECD average, but is 13% below
the Canadian average
• Youth (16-24) is performing below the
OECD average in all three domains
 In PS-TRE, Saskatchewan youth at levels
2/3 is 38% vs. the OECD average of 51%
• Nationally, educational achievement for both
Aboriginal and non-Aboriginal people affects their
overall literacy attainment in similar ways
 Saskatchewan’s results do not show the same correlation,
and we are starting to work with Manitoba and other panCanadian partners to try and understand why the data
shows this
• The gap between Aboriginal and non-Aboriginal
populations in Saskatchewan (at a 26 and 35 score
point difference in literacy and numeracy
respectively), is larger than in Canada overall at 14
and 22 points.
Proficiency Levels - Literacy
 Below Level 1
• 3% OECD, 4% Canada, 3% Saskatchewan
 Level 1
• 12%, 13%, 14%
 Level 2
• 34%, 32%, 33%
 Level 3
• 39%, 38%, 39%
 Level 4/5
• 12%, 14%, 11%
Proficiency Levels - Numeracy
 Below Level 1
• 5% OECD, 6% Canada, 6% Saskatchewan
 Level 1
• 14%, 17%, 18%
 Level 2
• 33%, 32%, 33%
 Level 3
• 35%, 33%, 33%
 Level 4/5
• 13%, 12%, 11%
Proficiency Levels - PS-TRE
 Non-respondents
• 24% OECD, 19% Canada, 15% Saskatchewan
 Below Level 1
• 12%, 15%, 18%
 Level 1
• 29%, 30%, 35%
 Level 2
• 28%, 29%, 28%
 Level 3
• 6%, 7%, 5%
• So Brett, how does our performance
compare to IALSS?
 (and, wink, wink, I know we’re doing worse)
PIAAC and IALSS
• It is not possible to make a direct comparison of the results
from IALSS and PIAAC; the scales used are not the same
• The data from the prose, document and numeracy
measures of IALSS have been rescaled at the national level
to fit with the literacy measure used in PIAAC – these can
be compared, but are still do not represent a 1 for 1
correlation
• Level 3 in PIAAC is not equivalent to Level 3 in IALSS. In
IALSS the OECD identified Level 3 as a minimum threshold
of the skills needed to manage the demands of work and
life. IN PIAAC, Level 3 was not defined the same way.
PIAAC and IALSS
• The OECD designed the PIAAC survey to help jurisdictions
determine the degree to which their skills address their
needs. As a result, the OECD proposes that results for each
jurisdiction should be assessed in terms of the association
between skill levels and other characteristics, such as job,
level of education, and income specific to the populations in
each jurisdiction.
• Does that mean all of our existing benchmark work is now
thrown out? No – at this stage, the results from PIAAC have
limited bearing on existing benchmarks. Much further
analysis and research will be needed before the results
can be applied to literacy programs and the measures
we all use.
• Overall, the difference between the
performance of people from different
backgrounds within jurisdictions is much
larger than the difference between
jurisdictions themselves
• e.g., The gap between Canadians (and
Saskatchewanians) at the 95th and 5th
percentiles is larger than the gap between
OECD countries on average
Skills in the labour force
Literacy ― Proficiency of population aged 16 to 65 by employment status, OECD average and countries, 2012
Percent of workers with literacy proficiency of
level 4 or 5
Japan
Australia
Norway
Canada
United States
Estonia
Denmark
Ireland
Austria
Slovak Republic
Korea
Italy
4
0
5
6
8
8
9
9
12
12
11
11
10
9
14
13
13
16
16
16
15
19
19
21
24
Percent of workers with literacy proficiency of
level 1 or below
25
5
Finland
Netherlands
Australia
Czech Republic
Flanders (Belgium)
Cyprus
OECD average
Austria
Canada
United States
France
7
8
9
9
10
10
10
11
12
12
12
13
13
13
13
13
14
15
16
16
19
Italy
10
15
Percent
20
25
30
0
5
10
15
Percent
20
22
25
26
30
Skills in the Labour Force
• Contrary to what might be expected,
Canada and the United States have an
almost equal proportion of working people
at levels 4/5 and level 1 and below
 Italy’s and Korea’s performance better fits
expectations
Immigrants
Literacy — Average scores and score point differences of population aged 16 to 65 by immigrant status,
OECD average and countries, 2012
Literacy mean proficiency scores of
foreign-born population
Slovak Republic
Ireland
Estonia
England/N. Ireland (UK)
OECD Average
Norway
Germany
United States
Korea
Spain
Italy
248
247
247
245
242
241
240
239
238
235
235
232
229
228
271
Finland
268
268
Norway
263
260
Denmark
256
256
Flanders (Belgium)
255
200 210 220 230 240 250 260 270 280
Score
Mean score differences on the literacy scale
between native- and foreign-born adults
Germany
Austria
26
25
24
23
23
21
Canada
Spain
Australia
Czech Republic
6
6
5
Ireland
0
10
13
10
20
29
30
38
38
38
37
37
36
34
40
Score point difference
54
51
43
50
60
Immigrants
• Canada and Australia have very diverse
populations with a range of educational
backgrounds, but immigration status does
not factor as highly in determining overall
literacy attainment in our two countries as
in others
The skills of young adults
All Adults Compared with Young Adults
Difference in literacy (points) between young
and all adults
Korea
20
Poland
15
France
13
Spain
12
Estonia
11
Netherlands
11
Italy
10
Flanders (Belgium)
10
Finland
9
Germany
9
Austria
8
OECD Average
7
Czech Republic
7
Denmark
5
Ireland
4
Australia
4
Sweden
4
Japan
3
Canada
2
Slovak Republic
2
United States
2
Cyprus
-2
Norway
-3
England/N. Ireland (UK)
-7
Difference in numeracy (points) between young
and all adults
Korea
18
Spain
9
France
9
Poland
9
Estonia
5
Netherlands
5
Austria
4
Italy
4
Germany
3
Canada
3
OECD Average
3
Finland
3
Flanders (Belgium)
2
Australia
2
Ireland
2
Czech Republic
2
Slovak Republic
2
Cyprus
0
Sweden
-1
United States
-3
Japan
-5
Denmark
-5
England/N. Ireland (UK)
-5
Norway
-7
Difference in PS-TRE (Level 2 and above, %)
between young and all adults
Korea
33
Estonia
23
Flanders (Belgium)
23
Czech Republic
22
Finland
20
Poland
19
Austria
18
Germany
18
Sweden
18
Netherlands
17
OECD Average
17
Ireland
15
Slovak Republic
15
Canada
14
Norway
14
Australia
13
Denmark
12
Japan
11
England/N. Ireland (UK)
8
United States
6
Cyprus
–
France
–
Italy
–
Spain
–
The skills of young adults
PS-TRE ― Proficiency of population aged 16 to 65 at Level 2 or 3 by age groups, OECD average and
Canada, 2012
OECD average
Canada
60
51
51
50
49
49
Percent
42
38
40
28
30
23
20
17
11
10
0
16 to 24
25 to 34
35 to 44
Age group
45 to 54
55 to 65
Youth
• Do we need to look at new ways of
knowledge transfer between older and
younger workers?
• How do employment opportunities
(primarily p/t vs. primarily f/t affect
attainment)?
• Other questions will be forthcoming …
Where Does Canada Go Next?
• A series of thematic reports
 Education
 Official Language Minority Communities
 Labour Market
 Social Outcomes
 Aboriginal Populations
 New Canadians
• PIAAC Online
Where do WE Go Next?
Together
• WE can look at building richer programs
and services for individuals, families,
employees and communities
• WE can support individuals in a life long
learning continuum that is reflective of their
changing circumstances
• WE can honour the holistic teachings of
the communities we have all worked with
www.piaac.ca
http://www.oecd.org/site/piaac/
http://piaacdataexplorer.oecd.org/ide/idepiaac/
Questions?
Brett Waytuck
Provincial Librarian / Executive Director
Provincial Library & Literacy Office
[email protected]