Transcript Slide 1

PISA Mathematics Assessment
APEC Tokyo Feb 2010
Kaye Stacey
University of Melbourne, Australia
Data and images in this presentation are from OECD websites and official publications and from ACER
publications on PISA in Australia.
The views expressed here are those of the author and do not represent the OECD or associates.
Outline
• What is PISA and what does it test?
• What is mathematical literacy?
• A small sample of results
–
–
–
–
Country comparisons
Levels of proficiency
Performance of subgroups
Social gradient
• Possibilities for Computer-based
Assessment of Mathematics
PISA: Programme for International
Student Assessment
• Test years 2000, 2003, 2006, 2009, 2012, ..
• 15 year olds
• assesses “the knowledge and skills that students have
acquired at school and their ability to use them in everyday
tasks and challenges”
– reading literacy
– scientific literacy
– mathematical literacy
• statistically rigorous, to ensure that the results are as
meaningful as possible, measuring
– student performance
– data on the student, family, school and system factors
Key features of PISA (from OECD)
• policy orientation
– major aim is informing educational policy and practice
– aim to significantly improve understanding of the outcomes of education
• concept of “literacy” (discussed later)
• relevance to lifelong learning
– motivation to learn,
– attitudes towards learning
– learning strategies;
• surveys to explore features associated with educational success
– characteristics of students and schools
– trends monitored every 3 years;
• breadth
– by 2006, around 90% of the world economy
– nearly 400 000 students
• Recent studies tracking young people in the years after age 15 show
PISA measures knowledge and skills relevant to a life success
Participating countries
• PISA 2000: 43
• PISA 2003: 41
• PISA 2006: 57
– nearly 400 000 students
• PISA 2009: 66
– PISA Plus : +9
• PISA 2012: about 90?
Asia-Pacific
2009
•
China
–
–
–
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Hong Kong
Macao
Shanghai
Indonesia
New Zealand
Thailand
Japan
Korea
Australia
Singapore
Chinese Taipei
Chile
Peru
Panama
Argentina
Mexico
USA
Canada
+9 PISA Plus
–
Malaysia, …
Survey methods
• Schools randomly selected by PISA (usually 150+)
• Random sample of 35+ students per school
– between age 15 yrs 3 mths & 16 yrs 2 mths
• Strict sampling criteria to be included in reports
– e.g. Netherlands in 2000 below required so not in trend
data
• Some countries oversample for their own purposes
• Each student does 2 hour test and 30 min
questionnaire
• Items are in rotating booklets (about 13)
– results of individual students not available/meaningful
TIMSS: Trends in International
Mathematics and Science Study
• Independent body, not OECD
• Tests every 4 years, since 1994/5
• Grade based sample (years 4, 8, 12)
• Tests randomly sampled intact classes
– hence teacher survey makes sense
• Aims to test achievement of curriculum goals
– Careful and extensive curriculum comparisons
• More Asian countries have participated in TIMSS
– Singapore’s high results very famous
Schedule of performance measures
• Additional cognitive assessments:
–
–
–
–
–
2003: problem solving
2006: computer based assessment of science
2009: electronic reading
2012: problem solving
2012: computer based assessment of mathematics
Now
preparing
Questionnaire components
• School context and attitudes
– themselves and their homes
– attitudes to learning
• School questionnaire, optional teacher questionnaires
• TRENDS since 2000 (nearly)
2000
2003
2006
2009
2012
Country
rankings
are
always
of
interest
–
statistics
make
comparison
complicated
Statistically better than Australia - Maths
• PISA 2003
– Hong-Kong-China, Finland, Korea, Netherlands,
Liechtenstein, Japan, Canada
• PISA 2006
– Chinese Taipei, Finland, Hong-Kong-China, Korea,
Netherlands, Switzerland, Canada, Macao-China
• Movements: 5 stay above Australia, 2 drop to
Australia’s group, 1 rises from Australia’s group, 2
new entrants
What percent of students in top
performance bands?
Percent of students at levels 5 & 6 in PISA mathematics
35
30
25
20
15
10
5
0
Australia Finland
2003
2003
Korea
2003
Australia Chinese Finland
2006
Taipei
2006
2006
HongKongChina
2006
Korea
2006
What percent of students in lowest
performance bands?
Percentage of students below Level 2 in PISA mathematics
25
20
15
10
5
0
Australia 2003
OECD average
Australia 2006
OECD average
2006
OECD: PISA Proficiency Level 2
“a baseline level of proficiency at which students begin to
demonstrate skills that enable them to actively use mathematics”
What is mathematical literacy?
What is mathematical literacy?
• PISA assesses “the knowledge and skills that
students have acquired at school and their ability
to use them in everyday tasks and challenges”
• Reflect recognition that globalisation and
computerisation are changing labour markets and
societies, and that a different set of skills is needed
• US evidence:
– greatest decline in jobs over the past decade has not
been in manual labour, but in routine cognitive tasks –
those that can easily be done at less cost by computer
(Levy & Murnane, 2006).
Mathematical literacy
• 2003/2006 "an individual’s capacity to identify and
understand the role that mathematics plays in the
world, to make well-founded judgments, and to
engage in mathematics in ways that meet the
needs of that individual’s current and future life as
a constructive, concerned and reflective citizen."
• Strong links to other concepts
– Mathematical modelling (in PISA framework)
– Numeracy (but certainly not just “basic skills”)
– Quantitative literacy
Sample domain items
• PISA: “Take the test”
– Reading – page 13
– Science – page 187
– Mathematics – page 97
• Questionnaires
– Also download from “MyPISA”
Reading
Reading
%
teachers
unaware
Reading
•
Written text
followed by
questions
•
This one:
answer as
graph
A test across countries needs…
• cultural breadth and balance in tests
– bullying, Ministry of Education, ….
– not a question of intersection of school curricula around the world
(TIMSS)
– school curricula (e.g. reading graphs) influence success rate and
hence usefulness of items for constructing a measure
• high quality in translations
–
–
–
–
two “masters” for each item (English and French)
translations from both masters compared
back translation etc
some informal language will not translate
• corner vs vertex
English: corner or vertex
French: vertex
Growing Up
• 6.1 A height of female in 1980 (given increase since then)
• 6.2 Explain how graph shows growth rate of girls slows down after 12
yrs of age
• 6.3 When are females taller than males of same age?
Growing Up (6.2 – growth rate girls)
• Classification
– Scientific; Change and Relationships; Connections
– Difficulty 574 PISA score points.
• The question requires students to:
–
–
–
–
Analyse different growth curves
Evaluate characteristics of data set, represented by graph.
Note and interpret different slopes along graphs.
Reason and communicate the results of this process, within explicit
models of growth.
• OECD average 45%
• Most successful countries: Netherlands (77%), Finland
(68%), Belgium (64%), Canada (64%)
• Large omission rates: Austria (44%) and Greece (43%).
Scoring for 6.2 (growth rate for girls)
•
•
Score (Code) 1 : Response refers to “change” of gradient of female graph, explicitly or
implicitly.
Code 11:Refers to reduced steepness, using daily-life language.
–
•
Code 12: Refers to reduced steepness ,using mathematical language.
–
•
Compares actual growth (comparison can be implicit).
From 10 to 12 the growth is about 15 cm, but from 12 to 20 the growth is only about 17 cm; The
average growth rate from 10 to 12 is about 7.5 cm per year, but about 2 cm per year from 12 to 20
years.
Score (Code) 0
Code 01: Student indicates that female height drops below male height, but does NOT
mention the steepness of the female graph or a comparison of the female growth rate
before and after 12 years.
–
•
You can see the gradient is less; The rate of change of the graph decreases from 12 years on;
(uses words like “gradient”, “slope”, or “rate of change”)
Code 13:
–
•
•
It does no longer go up, it straightens out; The curve levels off; It is more flat after 12; The line of
the girls’ starts to even out and the boys’ line just gets bigger; It straightens out and the boys’ graph
keeps rising.
The female line drops below the male line.
Code 02: Other incorrect responses. For example, the response does not refer to the
characteristics of the graph, as the question clearly asks about how the GRAPH shows….
–
Girls mature early; Girls don’t grow much after 12.
Growing Up (6.2 – growth rate girls)
• Answer type:
– Daily life language: over 70% of correct answers in 24
countries
– Mathematical language : 56% of correct answers in
Korea
– Comparing actual growth: common in Austria (34%);
Mexico (26%), Greece (23%), France and Turkey (19%).
• Common errors
– Most common error: not referring to graph e.g. “girls
don’t grow much after 12”.
– Around 40% of incorrect answers in France, Korea and
Poland refer to graph, only to show the female height
drops below the male height. (concept of gradient??)
Heatbeat (M537) - graph
Heartrate = 200 – age
Heartrate = 208 – 0.7*age
• Newspaper statement in
text alerts student to
phenomenon
•
•
Question 46.1: from which
age does the
recommendation
increase?
Question 46.2: write
formula for most effective
training heartrate (80% of
max)
Possible solution to Q46.1
Meaning of “2 out of 3 in next 20 years”
How additional data affects average –
complex multiple choice item
Interpreting an
unusual
representation
Bookshelves
• Classification
– Quantity; Occupational; Connections
– Difficulty rating: 499 PISA score points (Mean is set to 500)
• The question requires students to:
– Develop a strategy to connect two pieces of information for each
component: how many available, how many needed per set
– Use logical reasoning to link that analysis across the components to
produce the required solution.
– Communicate the mathematical answer as a real-world solution
(not 5.5 bookshelves)
• Most successful:
– Finland and Hong Kong-China (74%),
– Korea, the Czech Republic, Belgium and Denmark (72%).
• OECD average: 61% correct, 29% of students attempted &
incorrect and 10% did not attempt.
CBAM 2012
2012 CBAM: computer based test of
mathematics
• New opportunities for presentation of items to
measure same material better
– More attractive presentation
– Better presentation (e.g. animation)
– Better response formats (e.g. move an animation?)
• Able to test some aspects of doing mathematics by
computer and so extend notion of literacy to better
match world
– What are these aspects?
Mathematical Skills in the Workplace
Final Report to the Science, Technology and Mathematics Council, UK, 2002
C. Hoyles, A. Wolf, S. Molyneux-Hodgson & P. Kent
• Recommendation 1 . Raising Visibility and Awareness of the
Importance of Mathematical Literacy in the Workplace
• The focus should be:
– The nature of mathematical literacy: that it is anchored in real data, in the
context of a particular workplace.
– That maths used in the workplace has economic benefits in the marketplace.
– That mathematics may be present quite implicitly in jobs and tasks, which
are not obviously mathematical.
– Many employees, regardless of their level of employment, are required to
use mathematical literacy.
– That IT and mathematical skills are interdependent.
What IT-maths skills in ML?
Percent of students at levels 5 & 6 in PISA mathematics
35
30
25
20
15
10
5
0
Australia Finland
2003
2003
Korea
2003
Australia Chinese Finland
2006
Taipei
2006
2006
HongKongChina
2006
Korea
2006
What IT-maths skills in ML?
Use 3-D views
See GoogleSketchUp demo
What IT-maths skills in ML?
Percent of students at levels 5 & 6 in PISA mathematics
35
30
25
20
15
10
5
0
Australia Finland
2003
2003
Korea
2003
Australia Chinese Finland
2006
Taipei
2006
2006
HongKongChina
2006
Korea
2006
Plot
graph
What IT-maths skills in ML?
Write
formula?
What tools?
(graphics) calculator?
computer – what software?
mobile phone?
Enter
data?
How well
can you…..
use a
spreadsheet
to plot a
graph?
• “Do well by
myself”
– OECD total:
42.09%
– Macao:
28.49%
– Thailand
25.75%
– Australia
58.42%
MyPISA query
Open Geogebra demo
What IT-maths skills in ML?
Explore
maths
Studying subgroups of students
• Gender
• Ethnic and home background
– Migration
– Language background
– Indigenous students
• Social gradient
Social gradient
Mathematical literacy by socio-economic
background (Australia)
Graphic shows:
mean and confidence interval (white)
5th, 10th, 25th, 75th, 90th, 95th percentiles
Performance against social index (Science 2006)
(Note wide spread)
Social gradient (Science 2006)
(Sci-literacy score against social index)
From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up
Position of line; strength of relationship;
gradient of line; curvature; length of line
From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up
Interpreting the social gradient
• Strength of association (variance explained)
– Hong Kong 6.9% < Australia 11.3% < OECD 14.5%
– Asian countries tend to be low
• Gradient
– Australia 43 > OECD 40 > Hong Kong 26
• Length
– Australia has less variation in social index than OECD
– US has higher top level than OECD; same bottom
• Position
– Australia does better than the OECD average
• Curvature
– low and high groups do not differ on relationship (cf NZ, Canada)
• Australia – maths slightly less affected by social index than reading and
science – relatively more affected by school
• NOTE: For almost all countries, the effect of school average ESCS
outweighs effect of student’s own ESCS.
“Equally prepared for life?” OECD
• Reading PISA2000
– females significantly outscored males in all countries
• Mathematics PISA2003
– males often outscored females
• Science PISA2006
– no significant difference between males and females overall, but
some difference in patterns of strengths
– no significant differences in attitudes to school science
– marked differences in expectations of science career
Gender differences in science
in schools with different social levels
Australia’s indigenous girls and boys
Graphic shows:
mean and confidence interval (white)
5th, 10th, 25th, 75th, 90th, 95th percentiles
Information about PISA
http:// www.oecd.org
https://mypisa.acer.edu.au/
PISA: “Take-the-test”
MyPISA: public database and analysis
MyPISA data request
Thoughts and discussion points
• Immense amount of information and excellent reports
– Available to you through public databases (except secure ‘trend items’)
– But no study answers all questions
• in-depth understanding of thinking e.g. of algebra
• What has caused the results (e.g. Finland’s success; Asia’s success)
• An international study operates under severe constraints
– Create items that work internationally to measure target construct validly
– Anomalies can reveal differences
• e.g. Korea-Australia average, 103 vs 104
• Concepts of mathematical and scientific literacy
– Major contribution to educational aspiration in many countries
– Still developing
• How does computer-based mathematics affect definition of math literacy?
• Has mathematical literacy changed from 2003 to 2012?
• Aim is to find the school systems, schools, teaching and societies that
best prepare all future citizens for living productive and satisfying lives
Thank you
[email protected]
http:// www.oecd.org
https://mypisa.acer.edu.au/
Stacey, K. & Stephens, M. (2008). Performance of Australian School Students in
International Studies in Mathematics. Schooling Issues Digest 2008/1. Canberra
http://www.dest.gov.au/sectors/school_education/publications_resources/
schooling_issues_digest/.