Transcript Slide 1

The European Survey on
Language Competences:
the impact of multilingual language
assessment on language policy and
educational outcomes
Neil Jones
Director, European Survey on Language Competences
Language Sciences Initiative 3/10/2013
© UCLES 2013
The European Survey on Language
Competences
First administration
– Main Study Spring 2011 (England November
2011)
– Results published June 2012
Aims:
– Establish an indicator to measure progress
towards the 2002 Barcelona conclusions, to
“improve the mastery of basic skills, in
particular by teaching at least two foreign
languages from a very early age”
Instruments
– Language tests (English, French, German,
Italian, Spanish)
– Contextual questionnaires (addressing 13
language policy issues; for students,
teachers, principals and countries)
Interpretation: the Common European
Framework of Reference (CEFR)
C2
Proficient
User
C
C1
B2
Independent
User
B
B1
Levels tested in ESLC
A2
Basic User
A
A1
Writing: a B1 task
DE - Familienmitglied
EN - Favourite family member
Von einem deutschen Brieffreund bekommst du
eine E-Mail. Darin schreibt er:
… Bitte schreibe mir in deiner nächsten E-Mail,
wen du in deiner Familie besonders gern magst.
Was macht ihr gemeinsam? Warum versteht ihr
euch gut? …
Schreib eine E-Mail an deinen Freund und
antworte auf seine Fragen.
Schreib 80–100 Wörter
This is part of an email you receive from an
English pen friend:
In your next email, tell me about someone
in your family that you like a lot. What sorts
of things do you do together? Why do you
get on well with each other?
Write an email to your friend, answering
your friend's questions.
Write 80–100 words.
ES - Miembro de la familia
IT – Familiare preferito
Aquí tienes parte de un e-mail que has recibido
de un amigo español.
En tu próximo e-mail, háblame de alguien de tu
familia que te guste mucho. ¿Qué tipo de cosas
hacéis juntos? ¿Por qué os lleváis bien?
Escribe un e-mail a tu amigo en el que
contestes las preguntas que te hace.
Escribe 80–100 palabras.
© UCLES 2013
Questa è una parte di un'email che hai
ricevuto da un amico italiano.
Quando mi scriverai la prossima email,
parlami di una persona della tua famiglia
che ti piace molto. Che tipo di cose fate
insieme? Perché andate così d’accordo?
Scrivi un'email al tuo amico e rispondi alle
sue domande.
Scrivi 80–100 parole.
A performance at CEFR B1
Dear John, Thanks for your email. In my family I like a
lot Marie. It’s my sister. I have 3 sisters but I’m
going to talk you about Sophie. Sometimes we go
shopping together and we kocht a lot of clothes.
Marie is very friendly. We talk a lot together about
our personnal life: about boys friends, school.. It’s
funny. Last week I wend in her flat in Brussel. She’s a
student in chemistery, The day we went shopping for
find a dress for her. We finded it and she’s very
beautiful. See you soon Isabelle
© UCLES 2013
First target language (Skills averaged)
CEFR levels First language (Skills averaged)
Percentage
100%
0%
80%
20%
60%
40%
B2
B1
A2
40%
60%
A1
Pre-A1
20%
80%
0%
100%
UK- FR BE nl PL ES PT BE fr BG BE EL HR
SI
EE NL MT SE
ENG (EN) (FR) (EN) (EN) (EN) (EN) (EN) de (EN) (EN) (EN) (EN) (EN) (EN) (EN)
(FR)
(FR)
Second target language (Skills averaged)
CEFR levels Second language (Skills averaged)
Percentage
100%
0%
80%
20%
60%
40%
B2
B1
A2
40%
60%
A1
Pre-A1
20%
80%
0%
100%
SE
PL UK- EL
PT FR HR BG
SI
EE BE fr ES
(ES) (DE) ENG (FR) (FR) (ES) (DE) (DE) (DE) (DE) (DE) (FR)
(DE)
MT
(IT)
NL
(DE)
BE BE nl
de (EN)
(EN)
Questionnaire: major themes
•
•
•
•
Basis for life-long learning of foreign languages
Language friendly living environment
Language friendly schools
Teacher Initial and In-service Training.
© UCLES 2013
Questionnaire findings – success factors
Looking for a positive message, the questionnaire findings could
be interpreted as offering a simple recipe for success:
A language is learned better where motivation is high, where
learners perceive it to be useful, and where it is indeed used
outside school, for example in communicating over the
internet, for watching TV, or travelling on holiday. Also, the
more teachers and students use the language in class, the
better it is learned.
This ideal situation is approximated only in some countries, and
mainly for English.
Is English a model for other languages to follow?
The Commission thought not.
© UCLES 2013
Implications for language policy
The ESLC shows considerable scope for improvement in
language learning in Europe.
But does it provide evidence to improve the situation?
Evidence based policy: a fashionable concept.
Politicians cherry-pick findings that best serve their position more like policy-based evidence.
“Evidence” in evidence-based policy often concerns headline
outcomes: league tables, performance indices – an agenda
for top-down control and management.
Other sources of evidence (in-country academic studies) are
often neglected – though this is evidence that could inform
change and improvement.
© UCLES 2013
Implications for language policy
The ESLC makes it clear that policies for improvement, and
specific interventions, must be developed country by
country.
This is what Cambridge English is already doing, in Europe
and beyond, working at government and institutional
level.
E.g. the BEDA (Bilingual Education Development and
Asessment) project in Madrid.
© UCLES 2013
FERE’s* view on early impacts of the BEDA
programme
*Federación Española de Religiosos de la
Enseñanza
Learners now use much more English in
schools – less fear of speaking
English.
Teacher motivation has increased.
Teacher confidence in using English has
increased.
Teaching practice has become more
creative,
The contribution of the language
assistants has been very positive.
© UCLES 2013
Early impacts for schools in the BEDA
programme
Extrinsic motivation of students in taking
the examinations:
‘the exams give students a goal to work
towards. Students know the exams will
be good for their future, they are very
worried about the future’.
Increased self-awareness of students:
‘The atmosphere at school is completely
different and now students are much
more self-aware as to their strengths
and weaknesses than they were’.
Increased awareness of exam levels by
parents.
© UCLES 2013
Student proficiency
Biggest change in student
proficiency in oral skills in part due to the language
assistants,
but the Cambridge English
examinations have also
impacted.
© UCLES 2013
Evidence, understanding and action
The point is not merely to understand the world,
but to change it. (Karl Marx)
But then, what kind of evidence and understanding
do you need to do that?
© UCLES 2013
Cambridge English Research and
Validation group
was set up in 1990:
– To implement better operational approaches to assessment.
– To do research and develop new products, e.g. CAT.
– To improve and provide evidence for the quality of our exams.
The nature and the range of evidence we use has
continuously expanded, in response to:
– constant growth in the candidature,
– increasing engagement at state/institutional levels,
– an increasing focus on learning and educational reform,
i.e. a requirement to change the world.
© UCLES 2013
R&V early priorities
..
..
..
Candidate
information
(demographic,
learning history)
Test
analysis
(classical)
Reliability,
dependability
Scaling, item banking
Link to
CEFR
Scoring
validity
..
90
80
B1
70
..
60
A2
50
..
40
30
A1
Candidate
information
(demographic,
learning history)
Test
analysis
(classical)
Reliability,
dependability
Item
Response
Theory
calibration,
common
scale
Item response theory and item-banking
Measurement
scale
Standards
consistently
applied
..
90
80
B1
Test 1
70
..
A2
60
Test 2
50
..
A1
40
Test 3
30
Learners located
on scale
Tests at
appropriate
level
Item bank links all
levels
ESLC: Overlapping tests at three levels
ESLC Test design
Level 1
tasks\booklets
A1-R1-a
A1-R1-b
A1-R2-a
A1-R2-b
A1-R3-a
A1-R3-b
A2-R2-a
A2-R2-b
A2-R3-a
A2-R3-b
A2-R4-a
A2-R4-b
A2-R5-a
A2-R5-b
English
ER111
ER112
ER211
ER212
ER311
ER312
ER221
ER223
ER321
ER323
ER422
ER423
ER522
ER523
time
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
b1
1
English
ER221
ER223
ER321
ER323
ER422
ER423
ER522
ER523
ER532
ER533
ER631
ER633
ER731
ER733
time
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
7.5
English
ER532
ER533
ER631
ER633
ER731
ER733
ER642
ER643
ER741
ER742
ER841
ER843
time
7.5
7.5
7.5
7.5
7.5
7.5
15
15
15
15
15
15
b4
2
b5
b6
b7
b8
2
1
1
2
1
1
1
1
3
1
1
3
4
4
1
2
2
1
3
30
4
3
4
30
Level 2
b14
b15
3
4
4
3
4
4
3
3
4
30
30
3
30
b16
b17
b18
2
30
30
30
4
30
b19
2
b20
b21
b22
2
1
2
1
2
2
2
2
2
3
3
4
3
4
4
3
4
4
3
3
3
30
30
Level 3
b26
b27
2
2
4
3
4
3
30
30
30
3
30
30
4
30
30
3
30
b28
b29
b30
b31
b32
b33
b34
b35
b36
1
2
1
1
2
2
1
1
2
1
2
2
3
3
1
1
2
3
3
2
3
4
4
30
2
1
b24
1
1
1
4
b23
2
1
2
3
30
1
1
2
1
30
1
1
1
3
3
30
b12
2
3
3
b25
1
2
b11
4
4
4
b10
2
3
b13
1
b9
2
2
30
tasks\booklets
B1-R5-a
B1-R5-b
B1-R6-a
B1-R6-b
B1-R7-a
B1-R7-b
B2-R6-a
B2-R6-b
B2-R7-a
B2-R7-b
B2-R8-a
B2-R8-b
b3
1
2
30
tasks\booklets
A2-R2-a
A2-R2-b
A2-R3-a
A2-R3-b
A2-R4-a
A2-R4-b
A2-R5-a
A2-R5-b
B1-R5-a
B1-R5-b
B1-R6-a
B1-R6-b
B1-R7-a
B1-R7-b
b2
2
30
30
30
30
30
30
3
30
3
30
30
2
1
30
30
Cambridge English exams and the CEFR
Historical and empirical links
Flyers
Movers
C2
Starters
Beg-A2
KET
A2
1988
Council of
Europe
Waystage
level
PET
B1
FCE
B2
1939
1981
Council of
Europe
Threshold
level
Council of
Europe
Vantage
level
CAE
C1
1991
Effective
Operational
Proficiency
level
CPE
C2
1913
Mastery
level
Constructs
Criterionrelated
validity
Corpus
data
Construct validity
Scoring
validity
Link to
CEFR
..
90
80
B1
70
..
60
English
Profile
A2
50
..
40
30
A1
Candidate
information
(demographic,
learning history)
Test
analysis
(classical)
Reliability,
dependability
Item
Response
Theory
calibration,
common
scale
Criterion reference and GCSEs
The ESLC illustrates the use of the CEFR to describe
achievements in concrete terms.
How do these equate to GCSE grades?
We made a comparison based on the cohort which took
the ESLC in November 2011 and GCSE in June 2012
© UCLES 2013
GCSE grades and CEFR levels
Studies in Language Testing
© UCLES 2013
A model for
reading
(after Weir
2005)
Remediation where
necessary
Creating a text level structure:
Construct an organised
representation of the text [or
texts]
Building a mental
model
Integrating new
information
Enriching the
proposition
Monitor: goal
checking
Goal setter
Careful reading
Local:
Understand sentence
GlobaI
Comprehend main idea(s)
Comprehend overall text
Comprehend overall texts
Expeditious reading
Local:
Scan for specifics
Global:
Skim for gist
Search for main ideas and
important detail
Metacognitive
mechanisms/
Strategies
General knowledge of the
world
Topic knowledge
Inferencing
Selecting appropriate type
of reading:
Text structure knowledge:
Genre
Rhetorical tasks
Meaning representation of
text(s) so far
Establishing
propositional
meaning at clause
and sentence levels
Parsing
Lexical access
Word recognition
Visual input
Central processing core
Syntactic
knowledge
Lexicon
Lemma:
Meaning
Word class
Lexicon
Form:
Orthography
Phonology
Morphology
Knowledge
© UCLES 2013
First two volumes in EP studies series
© UCLES 2013
Impact
Criterionrelated
validity
Corpus
data
Construct validity
Scoring
validity
Link to
CEFR
..
90
80
B1
70
..
60
English
Profile
A2
50
..
40
30
A1
Candidate
information
(demographic,
learning history)
Test
analysis
(classical)
Reliability,
dependability
Item
Response
Theory
calibration,
common
scale
Educational
context
(impact studies,
LearningOriented
Assessment)
Consequential
validity
Learning Oriented Assessment
Impact studies: essentially
post-hoc evaluation.
LOA: a theory of action for
achieving positive
impact by design
– Implies planned modifications
to classroom practice
© UCLES 2013
External
exams
Learning
Oriented
Assessment
High-level goals
In-school
tests
Classroom
Teaching,
Learning,
Assessment
Setting goals and evaluating
achievement
© UCLES 2013
Outcomes
A complementary relationship with teaching:
Quantitative and qualitative dimensions
C2
..
Quantitative
(measurement)
dimension
..
..
C1
..
..
B2
Skills profile
..
B1
Qualitative (individual)
dimension
..
A1
© UCLES 2013
Individualisation: the primary
domain of the teacher
A2
..
Macro level (setting and
monitoring targets)
Learning objectives
(high-level, detailed)
Micro level (materials,
classroom practice)
LOA syllabus
C2
B1
Check
prior
knowledge
External
exam
Task
Language activity
Summative
monitoring
C1
B2
In
Frame of
reference
(CEFR)
Feedback,
modify learning
objectives
LOA
activity
Teacher observation
A2
Teacher decision making
A1
Interpretation
Record
Interpretation
informal record
structured record
Record of
achievement
LOA: evidence of (and for) learning
In LOA the collection and use of appropriate data
(evidence) is key to:
– implementing individualised adaptive learning
– empowering learners to manage their own learning
– fulfilling necessary accountability and monitoring functions.
The focus is on using evidence to promote
learning,
– not simply to measure it
– not simply to evaluate the delivery of learning.
It requires the support of appropriate technologies.
© UCLES 2013
Conclusions (1)
The ESLC shows the scope that exists to improve
language learning outcomes.
Interventions must be designed on a by-country basis.
Cambridge English is already working with ministries and
institutions to introduce educational reforms in a range of
countries.
Learning Oriented Assessment provides a coherent and
flexible model for such interventions in education.
LOA defines the nature of the evidence necessary for
achieving positive impact.
It sets us technological challenges to address (big data).
© UCLES 2013
Conclusions (2)
What is the relevance of these developments to
the University?
• To confirm our standing as a centre of
excellence in learning and teaching?
• As a vital educational mission that we are best
placed to fulfill?
• As an exciting context for pure and applied
research and development?
• All of the above!
© UCLES 2013
Thank you for your attention
© UCLES 2013