Transcript Document

The PIAAC International Survey of
Adults’ Skills: Policy Context, Aims,
Methods, and Opportunities
Jeff Evans
[email protected]
Radstats, York, 23 Feb. 2013
Aims of presentation
To describe the PIAAC international survey: its
aims, definition of ‘numeracy’, its methods,
and their ‘validity’
To describe features of the relevant educational
policy context, esp. the turn to ‘governance by
numbers’
To sketch some possible developments related to
statistics and social and educational research
2
PIAAC - Project for the International
Assessment of Adult Competencies
• previous international surveys : IALS (1994-98) and ALL
(2004-06), but few countries
• PIAAC: commissioned by OECD
• Domains: Literacy, Numeracy, and Problem solving in
technology rich environments
• design and planning overseen by subject-matter ‘expert
groups’ and by international consortia of consultants
• fieldwork done by national teams, in 24 countries
• first run in 2011-12, full results available in Oct. 2013
3
PIAAC Aims
To “measure the skills & competencies needed for
individuals to participate in society and for
economies to prosper”: ‘inclusive liberalism’
• Comparative: between [groups of] individuals and
across countries in key “competencies” (Literacy,
Numeracy, Problem Solving)
• Correlational: with individual outcomes, e.g. further
learning / earnings; & with aggregate outcomes,
e.g. economic growth, social equity in labour market
• Evaluative: potentially, of e.g. performance of
education / training systems (Schleicher, 2008)
4
PIAAC concepts and measures
Numeracy: the ability to access, use, interpret,
communicate mathematical information & ideas, to
engage in / manage mathematical demands of a range of
situations in adult life.
(PIAAC Numeracy Expert Group, 2009)
Also:
Background Questionnaire: attitudes to, and experience
with, numeracy (literacy, etc.) in various adult practices
Job-Related Assessment: use of skills in workplace
5
PIAAC survey design (1)
• adults 16-65: 5000+ sampled per country; various
incentives to participate
• unlike PISA (& TIMSS): no ‘captive population’ in schools
 combines household survey methodology with
educational testing
• default method of administration is via PC
• ‘adaptive testing’ to find appropriate ‘level’ of
respondent, to guide selection of further test items
 Issues of survey validity & reliability, esp. for
international comparisons of performance
6
PIAAC survey design (2)
Construct validity: extent to which measure represents all facets of
concept and in a theoretically ‘appropriate’ way
Reliability of testing across countries & across interviewers
Ecological validity: whether setting of research is ‘representative’ –
i.e. whether tasks on-screen representative of tasks to which one
wishes to generalise
External validity: overall representativeness of findings for
population sampled and for tasks / skills of interest
… Similar dilemmas for most educational assessment.
Possible approach: Radical Statistics Education Group, Reading
between the Numbers: a critical guide to educational research
(1982), 1st edn.
[needs revision – see Epilogue, below]
http://www.radstats.org.uk/books/ReadingBetweenNumbers.pdf
… BUT also need to discuss how results will likely be interpreted
and used, by politicians, producers, media (as e.g. with PISA?)
7
Educational policy context
Competitive global economic context  countries / regions like EU
must display appropriate ‘levels’ of skill and productivity …
educational values increasingly interpreted through neo-liberal
imperatives (Rizvi & Lingard, 2010)
Lifelong learning (LLL) aims to promote development of knowledge,
& to enable citizens’ active participation in all spheres of social &
economic life
BUT holds individuals responsible for their own education
• stress on the need to acquire and update range of abilities,
attitudes, knowledge and qualifications over the life-course – and
to demonstrate having done this
8
Role of international organisations
OECD & the EU are disseminating ideas & practices that strongly
influence nat’l policy making worldwide (Dale & Robertson, 2009)
• Authoritative definition – and deployment – of concepts like
adult numeracy (cf. Cockcroft, 1982; Crowther, 1959; etc., etc.)
• Promotion of “expertise in creating comparable datasets …
countries can measure relative success of education systems &
shift policy orientations accordingly” (Grek, 2010)
• … and ’governing by data’ (Lingard, 2011)
9
PIAAC survey design (3): further issues
Validity also  how survey’s measure scores interpreted or
reconceptualised in reports of the sponsors, or of the media …
• Can an adult’s performance score be related to putative ‘level of
numeracy’ in a one-dimensional sense?
Many adults have “spiky profiles”, which are different due to
distinctive life experiences: some find type A items (“data &
chance”) more difficult; others find type B items (“dimension &
shape”).
• Is it legitimate for, say, pressure groups to try to stipulate a
‘minimum level of numeracy needed to cope with the demands
of adult life’, for adults within their countries – based on scores
from an international comparative study – and then to count how
many adults have a relevant ‘deficit’?
10
Epilogue: relation to statistics & social research
• Big Data ? … PIAAC data at least ‘moderately big’
• Open Data … OECD undertakes data deposited on website by
October 2013 launch; data for other surveys (PISA) already there
• Clearly space for groups to investigate issues not investigated by
OECD and national partners / in a different way … a role for RS? *
• Raises issues for concerns explored in Rad Stats Education Group
(1982, see above) … However, the context has changed radically:
for one thing, educational policy is much more globalised,
responsive to ideologies of economic competition – and key
studies are international and comparative, including this one *
* Conference participants or others interested in working on either
of these themes are invited to contact the speaker:
Jeff Evans [email protected]
11
References
Cockcroft Report (1982). Mathematics Counts: Report of the Committee of
Enquiry into the Teaching of Mathematics under the Chairmanship of Dr. W H
Cockcroft. London: HMSO.
Dale R. & Robertson S. (2009) Globalisation & Europeanisation in Education.
U.K.: Symposium Books.
Grek, S. (2010) International Organisations and the shared construction of
policy ‘problems’: problematisation and change in education governance in
Europe. European Educational Research Journal, 9(3), 396-406.
Lingard, B. (2011). Policy as numbers: Ac/counting for educational research.
Australian Educational Researcher, 38, 4, 355-382. Online:
http://www.springerlink.com/content/u515384526754608/fulltext.pdf
OECD (2012). Literacy, Numeracy and Problem Solving in Technology-Rich
Environments: Framework for the OECD Survey of Adult Skills. Paris: OECD
Publishing. Online: http://dx.doi.org/10.1787/9789264128859-en
Rizvi, F. & Lingard, B. (2010) Globalizing education policy. London: Routledge.
Schleicher, A. (2008). PIAAC: A New Strategy for Assessing Adult
Competencies. International Review of Education, 54(5-6), 627-650. Online:
http://www.oecd.org/dataoecd/48/5/41529787.pdf