Transcript Highlights from the Program for the International

Findings from the Program for the
International Assessment of Adult
Competencies (PIAAC), 2012
November 12, 2014
Sondra Stein & Katie Landeros
American Institutes for Research
1
About
PIAAC
PIAAC is an international large-scale assessment
administered in 2011-12 in 23 countries
It assessed 16 - to 65-year-olds, non-institutionalized, residing
in each country, irrespective of nationality, citizenship, or
language status
Laptop computer In the U.S., 80% took the computer tests and
or paper-and15% took the paper-and-pencil tests.
pencil:
Assessment
subjects:
Literacy
Numeracy
Problem Solving in Technology-Rich
Environments (digital problem solving)
The background survey was conducted in English
Assessment
or Spanish. About 4% could not complete the BQ
was conducted
because of language difficulties or learning or
only in English
mental disabilities, and 1% could not complete it for
in the U.S.:
other reasons.
2
Participating Countries
2012
Australia
Austria
Belgium
Canada
Cyprus
Czech Republic
Denmark
Estonia
Finland
France
Germany
Ireland
Italy
Japan
Korea, Rep of
Netherlands
Norway
Poland
Slovak Republic
Spain
Sweden
United Kingdom
United States
2015
Chile
Greece
Indonesia
Israel
Lithuania
New Zealand
Singapore
Slovenia
Turkey
3
The PIAAC Assessment was delivered to a nationally
representative sample of households in every country.
 In the U.S. the household sample was selected through a 4stage stratified area sample:
• Counties (PSUs)
• Blocks
• Housing units with households
• Eligible persons within households
 Resulted in 5,010 respondents
 A U.S. supplement will add 3,600 more adults (report due late
2015) that represent key populations (young adults 16-34 yrs,
older adults 66-74 yrs, unemployed adults,16-65 yrs)
 A representative prison sample will include 1,200 inmates, 16-74
yrs, in state, federal and private prisons (report due 2016).
4
PIAAC provides a rich source of data
that tells us:
 What skills adults actually have and can use rather
than just the number of years of education they have
completed or the degrees they have.
 How adults acquire those skills, and what factors are
related to skill acquisition and decline.
 What the level and distribution of skills is within and
across various subgroups within the population.
 As a result. PIAAC data enables us to target our
efforts to focus on raising the skills of adults with the
greatest needs.
5
PIAAC collects data through it’s background
questionnaire and module on skill use as well as
through direct assessment of skills.
Background
questionnaire
Direct
assessment
of key
informationprocessing
skills
Module on
skill use
6
PIAAC Background Questionnaire
Focused on identifying:
 Skills that are critical to functioning successfully in today’s
society,
 How participants acquire those skills, and
 How those skills are distributed.
Areas of BQ include:
 Education and training, present and past,
 Work experience,
 Literacy, numeracy and ICT skill use at work and at home,
 Other 21st century skills used at work,
 Personal traits, and background information.
7
Every country could add up to 5 minutes of unique
questions and also make country-specific adaptations.
U.S. changes include:
Background questionnaire
5 min
 Additions
• Basic skills training
• Political Efficacy Information
• Health
• Race/Ethnicity
• Language
 Adaptations:
• Formal Education, Informal
Training
• Occupation, Economic
Sector, Earnings
88
The Direct Assessment focuses on
four domains:
 Literacy: both paper & pencil and computer versions
 Numeracy: both paper & pencil and computer versions
 Problem solving in technology-rich environments: only
on computer
 Reading components: only paper & pencil
 All countries were required to administer literacy and
numeracy assessments
 The U.S. assessed all four domains
9
Definitions of PIAAC direct assessment subjects
 Literacy is understanding, evaluating, using and engaging
with written texts
• to participate in society,
• to achieve one’s goals, and
• to develop one’s knowledge and potential.
 Numeracy is the ability to access, use, interpret, and
communicate mathematical information and ideas, in order to
engage in and manage the mathematical demands of a range
of situations in adult life.
 Problem solving in technology-rich environments involves
using digital technology, communication tools and networks to
acquire and evaluate information, communicate with others
and perform practical tasks.
10
PIAAC results are reported in two ways:
 Average Scores: reported on a scale of 0-500 for all
domains.
 Proficiency Levels: reported as the percentages of adults
scoring at six performance levels ( from below level 1 to
level 5) in literacy and numeracy and at four performance
levels in problem solving in technology-rich environments
(from below level 1 to level 3).
11
These descriptions of the PIAAC Proficiency Levels
for Literacy define what adults can do at each level.
Below
Level 1
(0-175)
Locate
single piece
of
information
in familiar
texts.
Level 1
(176-225)
Read
relatively
short
digital,
print or
mixed
texts to
locate
single text.
Level 2
(226-275)
Make matches
between text
and
information
that may
require low
level paraphrasing and
drawing lowlevel
inferences.
Level 3
(276-325)
Level 4
(326-375)
Perform
multiple-step
Identify,
interpret, or operations to
integrate,
evaluate
one or more interpret, or
synthesize
pieces of
information information
from
and often
complex
require
texts, and
varying
may require
levels of
complex
inference.
inferences.
12
Level 5
(376-500)
Integrate
information
across
multiple,
dense texts;
construct
syntheses,
ideas or
points of
view; or
evaluate
evidence
based
arguments.
Below level 1 literacy item
Election results
Stimulus is a short report on the results of a union election. The
report contains several brief paragraphs and a simple table
identifying the three candidates in the election and the number of
votes they received.
• The test-taker is asked to identify which candidate received the
fewest votes.
• He or she needs to compare the number of votes that the three
candidates received and identify the name of the candidate who
received the fewest votes.
• The word “votes” appears in both the question and in the table
and nowhere else in the text.
13
Literacy example – low difficulty
Literacy item
Level = 1
14
Literacy example – moderate difficulty
Literacy item
Level = 4 (low)
15
Overview of U.S. results
PIAAC results tell a story about the systemic nature of
the skills deficit among U.S. adults. PIAAC raises the
question:
What are we going to do to make sure that
• The U.S. has the workforce it needs to succeed in
the global economy?
• U.S. citizens have the skills necessary to support a
thriving democracy?
16
How did we do compared to other countries?
Literacy Numeracy PS-TRE
Japan
Finland
Netherlands
Australia
Sweden
Norway
Estonia
Flanders-Belgium
Czech Rep.
Slovak Rep.
Canada
Korea, Rep. of
U.K.
Denmark
Germany
United States
Austria
Cyprus
Poland
Ireland
France
Spain
Italy
Japan
Finland
Flanders-Belgium
Netherlands
Sweden
Norway
Denmark
Slovak Rep.
Czech Rep.
Austria
Estonia
Germany
Australia
Canada
Cyprus
Korea, Rep. of
U.K.
Poland
Ireland
France
United States
Italy
Spain
Japan
Finland
Australia
Sweden
Norway
Netherlands
Austria
Denmark
Czech Rep.
Korea, Rep. of
Germany
Canada
Slovak Rep.
Flanders-Belgium
U.K.
Estonia
United States
Ireland
Poland
Italy
Spain
Cyprus
France
 The U.S. ranked
lower than most
other countries in all
three domains.
 The US ranked
better in Literacy
than in Numeracy or
Problem Solving in
technology-rich
environments.
17
The U.S. average literacy score (270) was
lower
than
the
international
average
(273).
Literacy
Japan
Finland
Netherlands
Australia
Sweden
Norway
Estonia
Flanders-Belgium
Czech Rep.
Slovak Rep.
Canada
Korea, Rep. of
U.K.
Denmark
Germany
United States
Austria
Cyprus
Poland
Ireland
France
Spain
Italy
 Scores on literacy ranged from
296 (Japan) to 250 (Italy)
 U.S. scores were:
• Lower than in 12 countries
• Not significantly different
than in 5 countries
• Higher than in 5 countries
18
The U.S. average is low because a higher proportion of
U.S. adults are at the lowest levels (level 1 and below level
1) of literacy.
19
Who are the low-skilled adults in the U.S. in
literacy?
Educational attainment
Percentage of population
45
40
35
30
25
20
15
10
5
0
Grad or prof.
degree (10%)
Bachelor’s
degree (16%)
Below Level 1
Associate’s
degree (9%)
Level 1
High school
credential (50%)
Level 2
20
Below high
school (15%)
Who are the low-skilled adults in the U.S. in
literacy?
Race/ethnicity
Percentage of subpopulation
45
40
35
30
25
20
15
10
5
0
White, non-Hispanic (65%) Black, non-Hispanic (13%)
Below Level 1
Hispanic (14%)
Level 1
Level 2
21
Other (8%)
Who are the low-skilled adults in the U.S. in
literacy?
First language
Percent of subpopulation
40
35
30
25
20
15
10
5
0
English as first language (85%)
Below Level 1
Other as first language (15%)
Level 1
Level 2
22
Who are the low-skilled adults in the U.S. in
literacy?
Immigration status
Percentage of subpopulation
40
35
30
25
20
15
10
5
0
Born in U.S. (85%)
Below Level 1
Not Born in U.S. (15%)
Level 1
Level 2
23
Percentage of subpopulation
Who are the low-skilled adults in the U.S. in
literacy?
Employment
50
40
30
20
10
0
All (100%)
Employed
full-time
(52%)
Employed Unemployed
part-time
(8%)
(12%)
Below Level 1
In school
(10%)
Level 1
Retired (4%) Permanently Looking after Other (3%)
disabled
family (6%)
(5%)
Level 2
24
Who are the low-skilled adults in the U.S. in
literacy?
Occupation
Percent of subpopulation
50
45
40
35
30
25
20
15
10
5
0
Elementary
occupations
(8%)
Machine
operators
(6%)
Crafts and
Service
Clerks (8%)
trades
workers/sales
workers (9%)
(21%)
Below Level 1
Level 1
Technicians Legislators, Professionals
and
officials, and
(21%)
associates
managers
(16%)
(10%)
Level 2
25
Percentage of subpopulation
Who are the low-skilled adults in the U.S. in
literacy?
Industry
50
45
40
35
30
25
20
15
10
5
0
Below Level 1
Level 1
Level 2
26
Mean task input in percentiles of 1960 distribution
Since 1970, there has been a shift in the U.S. economy away
from routine and manual tasks and towards more analytic and
interpersonal tasks that require higher skills.
70.0
65.0
60.0
Routine manual
55.0
Non-routine manual
Routine cognitive
50.0
Non-routine analytic
Non-routine interpersonal
45.0
40.0
35.0
1960
1970
1980
1990
2000
2006
2009
Source: Autor, D. H. and B.M. Price (2013), "The Changing Task Composition of the US Labor Market:
An Update of Autor, Levy, and Murnane (2003)", MIT Mimeograph, June.
27
The percentage change in earnings since the 1970s mirrors that
shift in occupations
Percentage
Change in
Earnings
Since 1961
Tabulations of annual March Current Population Survey Data, by David Ellwood, Harvard University.
Slide prepared by ETS.
28
Literacy skills in younger and older
generations
29
Time for the U.S. to Reskill? What the Survey
of Adult Skills Says (an OECD report)
Approximately 36
million U.S. adults have low skills.
Roughly the same population of Minnesota, New York, and
Michigan states combined.
MN
+
NY
+
MI
30
Workforce Innovation and Opportunity
Act (WIOA) and Ready to Work
 Signed into law July 2014
 Previously 1998 Workforce Investment Act
 Ready to Work: Job-Drive Training and American
Opportunity report also released
 Both initiatives aim to “train Americans with the skills
employers need, and match them to good jobs that need to
be filled now.”
31
Three Essential Features of Career Pathways
2. Multiple entry points –
for both well-prepared
students and targeted
populations
1. Well-connected
and transparent
education, training,
credentials, and
support services
3. Multiple exit points
32
Who are the low-skilled adults in the U.S.
workforce?
 A higher percentage of Blacks and Hispanics
 A higher percentage in manual and blue and
white-collar semiskilled occupations, including the
following sectors:
•
•
•
•
•
•
Construction
Administrative services
Transportation
Hospitality
Retail
Health
33
33
Barriers to participation in Non-Formal Education (NFE)
Something unexpected
came up that prevented me
from taking education or
training
6%
I did not have the
prerequisites
2%
Other
10%
I did not have time
because of child care
or family
responsibilities
16%
The course or
programme was
offered at an
inconvenient
time or place
13%
Education or training
was too expensive/I
could not afford it
24%
Lack of employer's support
3%
I was too busy at work
26%
3434
What can we do to raise the skills of working
age adults?
In the long run, the best policies we have [to
combat inequality] involve investing in our citizenry.
… Higher education, and public education, is
America’s best idea.
-David Autor
35
35
What does the data suggest about
what we need to do differently?
 Do a better job of assuring every American has a strong
foundation of basic skills.
 Do a better job of providing skills upgrading for immigrants
to America.
 Do a better job of providing skills upgrading to working
adults at every level so they can keep pace with change.
 Understand that improving the skills of our population is
everybody’s business: we can educate our way to a better
economy.
36 36
How PIAAC can help
 Use PIAAC data to advocate for more investment in
adult lifelong learning.
 Use PIAAC data to strengthen program approaches to
building lifelong career pathways.
 Use PIAAC data to create and renew local, state, and
national approaches to lifelong learning where
learning is everybody’s business.
 Use PIAAC data and tools to document successes.
37 37
What else PIAAC offers at:
www.piaacgateway.com
 A test of Education and Skills Online – an online
version of PIAAC that will be available in 2015.
 Outreach Toolkits (coming soon) so you can share
this data with others.
 Access to the Data itself, through the NCES or
OECD International Data Explorer.
 Research reports that analyze the data.
 Regular updates through the PIAAC Buzz.
38
38
PIAAC Outreach Toolkits
These Outreach Toolkits include all documents and power point
modules you will need to share information about PIAAC with others.
*Simply download the pieces you want to use!*
Toolkit
Documents
•
PIAAC
Overview
•
Key Results
•
National
Supplement
•
Education &
Skills Online
(ESO)
•
Videos
•
Infographics
Toolkit Slides for Your Presentation
Instructions: Start with “What is PIAAC” and “Results Overview”.
Then add other modules to build a presentation that suits your
audience.
What is
Key U.S. Issues
Results
Gateway
PIAAC
• Low-skilled Workers
Overview
& Other
•The Future Workforce
Resources
•Health Status & Skills
These can be
Sample
downloaded
•The Impact of Parent
PIAAC
with a focus on
Education
National
Tasks
literacy,
Supplement
(Each of these can be
numeracy, or
digital problemsolving
downloaded with a focus
on literacy, numeracy or
digital problem-solving)
ESO
39
39
Visit the PIAAC Gateway at:
www.piaacgateway.com
40
Questions and Discussion
41
For more information about PIAAC
 Visit our website at https://piaacgateway.com
 Or contact us at [email protected]
 Sign up for our regular newsletter, the PIAAC Buzz at
https://piaacgateway.com
42
What you can find on the PIAAC Gateway
www.piaacgateway.com
Links to everything you want to know about
PIAAC, including:
 The latest PIAAC reports and presentations
 A calendar of PIAAC-related events
 Infographics, brochures, and videos to share
 Data tools and training resources
 Press coverage from the U.S. and around the world
 Links to release events, reports, and presentations
43
How to Access PIAAC Data
 The PIAAC Results Portal enables you to produce figures and tables
that take an in-depth look at the U.S. results as well as compare the
performance of U.S. adults to adults in other participating countries.
http://nces.ed.gov/surveys/piaac/results/makeselections.aspx
 The US PIAAC International Data Explorer (IDE) is a web-based tool
that produces customizable tables and graphs using data from PIAAC.
The US IDE includes US national variables as well as data from all
other participating countries from the administration of PIIAC in 2012.
It is free of charge and does not require any advanced statistical
software or knowledge. http://nces.ed.gov/surveys/international/ide/
 PIAAC IDE Training offers online training for researchers,
policymakers, and practitioners who want to use PIAAC data to answer
questions they have about adult skills. To sign up for a webinar or
arrange one for your group, contact [email protected]
44
44
National Center for Education Statistics’ PIAAC
International Data Explorer (IDE)
45
NCES PIAAC Data Explorer (IDE)
 What is the NCES IDE?
 PIAAC Data Explorer Demonstration
 Answering your research questions
46
What is the NCES IDE?
 A web-based application for accessing PIAAC
data that does not require any advanced statistical
knowledge or software
 A point-and-click interface for:
• Creating statistical tables and charts
• Exploring levels of adult skills and demographics
47
What the NCES IDE can and can’t do
 The PIAAC Data Explorer can:
• Compute simple descriptive statistics such as:
– averages, percentages, and percentiles
– percentages of students at pre-defined proficiency levels
•
•
•
•
Run cross-tabulations
Run significance tests on computed statistics
Collapse response categories within a variable
Create and export charts, graphs, and tables
 It cannot:
• Run regression models or multi-level analysis
• Compute new variables
48
What’s in the NCES IDE?
 Overall scale (e.g. literacy, numeracy, problem solving in
technology-rich environments, reading components)
 Continuous variables from international and U.S. national
background questionnaire (e.g. earnings/hours of work per
week)
 Demographic variables (e.g. age, education background,
employment history)
 U.S. specific background questionnaire variables (e.g. selfreported health status, race/ethnicity)
 Trend variables from PIAAC (2012), ALL (2003), IALS (1994)
 Variables are organized into categories that have shared
characteristics
49
Overall Process
1. Select Criteria:
Choose your
measure(s), year(s),
and jurisdiction(s).
2. Select Variables:
Select at least one
variable from the
selection of
categories and
subcategories.
3. Edit Reports:
Preview how your
data will look, and
edit your report
format options and
statistics options as
desired.
50
4. Build Reports:
Retrieve the data,
make charts and
graphs, save, and
print reports.
Select Criteria
51
Select Variables
52
Edit Reports
53
Edit Reports: Format Options and
Statistics Options
54
Build Reports
55
Chart Options
56
Significance Test Options
57
Output: Data Table
58
Output: Significance Test
59
Output: Chart
60