Transcript Criterion Referenced Assessment: Relating Tests to the

The Role of the
Common European Framework
John H.A.L. de Jong
EALTA Conference, Kranjska Gora,
Slovenia, May 14 – 16, 2004
LANGUAGE TESTING S ERVICES
1
John H.A.L. de Jong 2004
LANGUAGE TESTING S ERVICES
2
John H.A.L. de Jong 2004
LANGUAGE TESTING S ERVICES
3
John H.A.L. de Jong 2004
Overview
Language
The Common Framework
Application
LANGUAGE TESTING S ERVICES
4
John H.A.L. de Jong 2004
Bilingual
Brain
LANGUAGE TESTING S ERVICES
5
John H.A.L. de Jong 2004
Speaking: From intention
to articulation

CONCEPTUALIZER

Message structure

FORMULATOR

Utterance structure

ARTICULATOR

Utterance
LANGUAGE TESTING S ERVICES
6
Levelt, 1989
John H.A.L. de Jong 2004
LANGUAGE TESTING S ERVICES
7
John H.A.L. de Jong 2004
Person A
Spoken
turn
Person B
declarative
declarative
social
social
discourse
discourse
Decoding
Encoding
Spoken
turn
language
structures
LANGUAGE TESTING S ERVICES
Decoding
Encoding
Spoken
turn
language
structures
8
John H.A.L. de Jong 2004
Facility in Spoken Language
is not Oral Proficiency
Language
Language use
Culture
World knowledge
–OP includes the ability to use the
language in socially appropriate
ways in a range of complex
functions over a variety of topics.
–High OP implies adequate FSL
–High FSL does not guarantee
high OP
OP(L2)  f (OP(L1), FSL(L2), Culture(C2), Social (S2))
FSL(L2)  OP(L2) - OP(L1)
LANGUAGE TESTING S ERVICES
9
John H.A.L. de Jong 2004
Some Basic Issues in the CEF
Some of which are often forgotten
LANGUAGE TESTING S ERVICES
10
John H.A.L. de Jong 2004
Types of Scores
Discrete
fixed set of fixed values
2
0
Continuous
4
3
1
infinite number of values
(1, 2, 3, etc. But also 1.2, 1.53 1.6 etc.)
LANGUAGE TESTING S ERVICES
11
John H.A.L. de Jong 2004
Defining stages on a continuous scale
Defining stages on a continuum is arbitrary.
We can define different numbers of stages, according to our liking.
As long as we clearly define where a one stages ends and another begins.
Note: There is always a stage below the lowest defined stage
LANGUAGE TESTING S ERVICES
12
John H.A.L. de Jong 2004
Potential Language Development:
follows two basic dimensions
Quality
Quantity
LANGUAGE TESTING S ERVICES
13
John H.A.L. de Jong 2004
The quantity development
is in fact multidimensional
Dimensionality
and quality can develop along
each of the dimensions
LANGUAGE TESTING S ERVICES
14
John H.A.L. de Jong 2004
QUANTITY refers to:
the number of ...
domains
functions
notions
situations
locations
topics
roles
(school, work, home, etc.)
(ask, command, inquire, etc.)
(north, south, table, mother, cat, eat, drink, etc.)
(meeting, e-mail, telephone, etc.)
(market, school, police station, etc.)
(weather, study, holidays, etc.)
(listener in audience, participant in discussion, etc)
... that a language user can deal with.
LANGUAGE TESTING S ERVICES
15
John H.A.L. de Jong 2004
QUALITY refers to:
the degree to which
Language use is effective
leading to degree of precision
1) in understanding what is meant
2) in expressing one’s meaning
Language use is efficient
leading to communication
with least possible effort
LANGUAGE TESTING S ERVICES
16
John H.A.L. de Jong 2004
C1
Note: It is essential to
understand that
in the CEF
Combining
Q&Q
A combination
of
Mastery does NOT mean
Mastery
No
pr blem
B2
‘native speaker’
level. Surely
Quantity
& Quality:
an illiterate native speaker is
B1
not even at Breakthrough for
Fluent and Spontaneous
Operational
Proficiency
reading. And there are big
A
differences
among
defines
I’m afraidFully independent
native speakers with you are
Levels
respect
to the of
below A1
other skills
Language
as well.
Vantage
Proficiency
Note: there is always
a level below the
lowest defined
level
Independent
Threshold
Co-operative
Waystage
Basic
functions
Breakthrough
Below Breakthrough
LANGUAGE TESTING S ERVICES
17
John H.A.L. de Jong 2004
Simplify
A more simple
system of level
names (universal: no
C2
C1
B2
translation problems)
B1
A2
A1
Below A1
LANGUAGE TESTING S ERVICES
18
John H.A.L. de Jong 2004
Broad & Narrow
C2.2
C2
C2.1
Now we can make
level
distinctions
C1.2
C1.1
B2.2
B2.1
B1.2
and
ones
B1
B1.1
A2.2
A2
A2.1
A1.2
A1.1
C1
B2
A1
Below A1
LANGUAGE TESTING S ERVICES
19
C
B
A
John H.A.L. de Jong 2004
Note: a person at the border between level A2 and B1 will probably be
Oral
scale
EXAMPLE:
Oral
Interaction
Scale
able to do 80% of the
tasks at
level A2, but can also
do 50% of
the tasks at level B2, etc, because the underlying scale is
continuous and the CEF descriptors have been IRT scaled.
C2
1%
Conveys finer shades of meaning precisely and
naturally.
C1
5%
Shows fluent, spontaneous expression in clear,
well-structured speech.
B2
15%. Relates information and points of view clearly and
without noticeable strain.
B1
50% Relates comprehensibly main points he/she wants
to make on familiar matters
A2
80% Relates basic information on, e.g. work,
background, family, free time etc. .
A1
95%
LANGUAGE TESTING S ERVICES
Makes simple statements on personal details and
very familiar topics.
20
John H.A.L. de Jong 2004
Hierarchy of Scales
To illustrate the hierarchical
principle one branch of the
hierarchy is worked out in
detail to the right 
Communicative
Strategies
The Global scale is for
Overall Language
Proficiency.
Reception
Global Scale
Overall language Proficiency
Communicative
Language Competencies
Production
Communicative
Activities
Interaction
Spoken
And at each further node in the
hierarchy the CEF offers
descriptive scales.
Note: this branch of the hierarchy
is part of the quantity
dimension: how much a
language learner can do
LANGUAGE TESTING S ERVICES
Mediation
Written
Understanding
a native speaker
Conversation
Informal
Discussion
Formal
Discussion
Obtaining Goods
and Services
Interviewing &
being interviewed
21
John H.A.L. de Jong 2004
The Quality Hierarchy
The same Global Scale
analyzed here into the
constituent elements
of the quality
dimension: how
well a learner can do.
And here too
there is a scale at
each node.
Vocabulary
Range
LANGUAGE TESTING S ERVICES
Global Scale
Overall language Proficiency
Communicative
Strategies
Communicative
Language Competencies
Linguistic
Range
General
Linguistic
Sociolinguistic
Communicative
Activities
Pragmatic
Control
Grammatical
Accuracy
Phonological
Control
Vocabulary
Control
Orthographic
Control
22
John H.A.L. de Jong 2004
Automatic speaking tests
Integrated “listen then speak” items
Real-time processing of spoken linguistic forms
Overall score and diagnostic subscores
• Sentence Mastery
• Vocabulary
• Fluency
• Pronunciation
LANGUAGE TESTING S ERVICES
23
John H.A.L. de Jong 2004
What is tested?
Facility in spoken English:
• ability to track what is said,
• extract meaning in real time,
• formulate responses
• produce responses at a conversational pace, that are
relevant and intelligible
LANGUAGE TESTING S ERVICES
24
John H.A.L. de Jong 2004
Conversational Skill
hear utterance
get lexical items
extract structure
build semantic field
contextualize
infer demand (if any)
articulate response
construct response
select lexical items
build phrase structure
select register
decide on response
(After: Levelt, 1989)
LANGUAGE TESTING S ERVICES
25
John H.A.L. de Jong 2004
Native & Non-Native Overall CDF
Table 1: Overview of main data on human scoring with CEF
SET-10
SST
Number of candidates
587
528
Number of raters
7
4
Total number of ratings
8464
2500
Interrater reliability (ANOVA)
.95
.94
Rater agreement (cumulative)
Exact same level
64%
67%
1 level difference or less
28%
24%
2 levels difference or less
6%
6%
>2 levels difference or less
2%
3%
LANGUAGE TESTING S ERVICES
26
John H.A.L. de Jong 2004
SET 10 ~ CEF
Experiment:PhonePass
Score interpretation
on CEF scale
LANGUAGE TESTING S ERVICES
27
John H.A.L. de Jong 2004
English and Spanish
on CEF
scale
CEF English
vs Spanish
8
6
4
Spanish
2
C2
0
C1
B2
-2
Linear regression
y = 0,5418x - 0,0315
R2 = 0,9949
Polynomial regression
y = -0,0024x3 + 0,0047x2 + 0,6334x - 0,1583
R2 = 1
LANGUAGE TESTING S ERVICES
B1
-4
-6
A2
-8
-8
-6
-4
-2
0
2
4
6
8
English
28
John H.A.L. de Jong 2004
English and Spanish vs. CEF scale
6.0
Diffculty Estimates
4.0
2.0
0.0
-2.0
Spanish
English
-4.0
North, 2000
-6.0
A1
LANGUAGE TESTING S ERVICES
A2
B1
B2
CoE Levels
29
C1
C2
John H.A.L. de Jong 2004
So what is the role of the CEF?
• Transparency - Tool for communication
• Comparability - Common yardstick
• Progress - Stimulates discussion
LANGUAGE TESTING S ERVICES
30
John H.A.L. de Jong 2004