Transcript Manual for Relating Examinations to the Common European

Manual for Relating
Examinations to the
Common European
Framework
Sauli Takala
EALTA Confererence
Slovenia, Granjska Gora, May 14-16,
2004
Overview
• Some background
• Why is the Manual needed?
• What is the approach adopted?
• What is/is NOT the Manual for?
• Procedures proposed in the Manual
• Current situation
• Future prospects and challenges
Some historical notes
The Convention for Cultural Cooperation (CDCC), which provided the mandate for
the Council of Europe´s action within the domain of cultural affairs was established
50 years ago
Modern languages were included in the agenda about 45 years ago
After a number of more ad hoc set of activities, action related to modern languages
was organised into medium-term projects
Work focused on exploring how learning objectives could be determined in a manner
that reflected the social and learner needs
The Threshold specification model of learning objectives is a good example of
outcomes of this exploration and development work
In 1991, at an intergovenmental symposium in Rüschlikon, Switzerland, it was
agreed to launch a programme to develop a framework of reference for languages,
which would provide a coherent conceptual tool for the various needs of language
education. The Portfolio project was also launched. After some ten years of work,
inlvolving broad consultation, the Common European Framework of Reference for
Languages was adopted at an intergovernmental conference in Strasbourg
In recent years both the Council of Europe and European Union have adopted the
CEF as the basic tool for their work on language education
Current CoE Projects
• Common European Framwork of Reference
of Languages (CEF)
• European Language Portfolio
• Manual for Relating Examinations to the
CEF
• Language Policy analyses and reviews
• Other: CEF-sign language; Romani
language curriculum; bank of descriptors;
language-specific reference frameworks
(German, French ...)
What is the CEF – some points to
consider
CEF provides:
•a ‘vertical dimension’ and outlines an ascending series of
common reference levels for describing learner proficiency
(6-point scales)
• a ‘horizontal dimension’ - a set of descriptive categories
map out made up of parameters of communicative activity
and communicative language competence (not all are scaled)
CEF construct of language proficiency: the key to
understanding the CEF (p. 9, English version)
Language use, embracing language learning, comprises the
actions performed by persons who as individuals and as social
agents
•develop a range of competences, both general and in particular
communicative language competences.
•They draw on the competences at their disposal in various
contexts under various conditions and under various constraints
•to engage in language activities involving language processes
•to produce and/or receive texts in relation to themes in specific
domains,
• activating those strategies which seem most appropriate for
•carrying out the tasks to be accomplished.
The monitoring of these actions by the participants leads to the
reinforcement or modification of their competences.
C
2
•Can understand with ease virtually everything heard or read. Can summarise information from
different spoken and written sources, reconstructing arguments and accounts in a coherent
presentation. Can express him/herself spontaneously, very fluently and precisely, differentiating
finer shades of meaning even in more complex situations.
C
1
•Can understand a wide range of demanding, longer texts, and recognise implicit meaning. Can
express him/herself fluently and spontaneously without much obvious searching for expressions.
Can use language flexibly and effectively for social, academic and professional purposes. Can
produce clear, well-structured, detailed text on complex subjects, showing controlled use of
organisational patterns, connectors and cohesive devices.
B
2
•Can understand the main ideas of complex text on both concrete and abstract topics, including
technical discussions in his/her field of specialisation. Can interact with a degree of fluency and
spontaneity that makes regular interaction with native speakers quite possible without train for
either party. Can produce clear, detailed text on a wide range of subjects and explain a viewpoint
on a topical issue giving the advantages and disadvantages of various options.
B
1
•Can understand the main points of clear standard input on familiar matters regularly
encountered in work, school, leisure, etc. Can deal with most situations likely to arise whilst
travelling in an area where the language is spoken. Can produce simple connected text on topics
which are familiar or of personal interest. Can describe experiences and events, dreams, hopes
and ambitions and briefly give reasons and explanations for opinions and plans.
A
2
•Can understand sentences and frequently used expressions related to areas of most immediate
relevance (e.g. very basic personal and family information, shopping, local geography,
employment). Can communicate in simple and routine tasks requiring a simple and direct
exchange of information on familiar and routine matters. Can describe in simple terms aspects of
his/her background, immediate environment and matters in areas of immediate need.
A
1
•Can understand and use familiar everyday expressions and very basic phrases aimed at the
satisfaction of needs of a concrete type. Can introduce him/herself and others and can ask and
answer questions about personal details such as where he/she lives, people he/she knows and
things he/she has. Can interact in a simple way provided the other person talks slowly and clearly
and is prepared to help.
Language learning potential
progresses on two dimensions (Source: John
de Jong,2004)
Quality
Quantity
Scale hierarchy: quantitative dimension
Global scale (T. 1)
Communication
Strategies
Communicative
Language
competences
Reception
(4.4.2)
Communicative
Activities (4.4)
Production
(4.4.1)
Interaction
(4.4.3)
Spoken
Mediation
(4.4.4)
Written
Understanding
Native speaker
Conversation
Informal discussion
Formal discussion &
meeting
Transactions to
Obtain goods &
services
Goal-oriented
Co-operation
Frequently asked questions:
How far do pupils/students progress in L2 in
the school system (no PISA in L2; EU
Barcelona indicator; European item bank)
• What is the currency value of national
school reports/certificates nationally and
internationally? How could it be improved?
• How can we relate our exams to the CEF?
(need of guidance)
• How do I know that my B1 is your B1?
(need of guidance)
• Need of guidance in relating exams to CEF
in a credible manner led to the project of
developing a Manual
Some history related to the Manual:
Invitational Expert Seminar; Helsinki,
summer 2002
• Report: DG IV/EDU/LANG (2002) 15
• Recommended development of
a
Manual as a collaborative endeavour
building on current work in member
states.
– Describe the examination coverage,
administration and analysis procedures;
– Align results reported from the examination
to the CEF common reference levels;
– Provide supporting evidence that reports on
the procedures followed to do so.
Who done it?
• Authoring Group/Phase 1: (aka
”Manual workers”) : Brian North,
Neus Figueras, Piet Van Avermaet,
Sauli Takala, Norman Verhelst
• Sounding Board of consultants: other
Helsinki Seminar participants
• Ad hoc Project Group (new): Charles
Alderson, John de Jong, John Trim...
Participants can vary according to
topic of discussion
Development Timeline
•
•
•
•
July 2002:
September 2002:
November 2002:
Nov-Jan 2003:
• Jan-May 2003:
• May 19-20, 2003
0.5
• Sep 2003:
• Sep 2003:
• April 15-16, 2004:
Helsinki seminar
Project approval
Authoring group set up
Consultation on Outline with
Sounding board (consultants)
Authoring Group: revision of
draft text
Meeting of Authoring Group and
consultants to discuss version
(Strasbourg)
English & French drafts (DG
IV/EDU/LANG (2003) 5
Piloting Feedback invited (guide
provided)
Seminar on Piloting, Strasbourg;
further action discussed
Aims of Manual
• contribute to competence building in the
area of linking assessments to the CEF
• encourage increased transparency on the
part of examination providers
• encourage the development of both formal
and informal national and international
networks of institutions and experts
• Thus: Manual is not just a technical
guidance tool but a rich resource for
thoughtful action
What the Manual NOT?
• Not a guide how to write good tests; presupposes that
exam results are reliable and their interpretation is
valid
• Does not demand any particular type of examination
format nor exclude automatically some types of
examination
• Being CEF-based, it cannot provide resources that are
not in the current CEF-version; however, new
descriptive categories can be added in the Specification
job (->CEF-revision)
• Does not prescribe any one method of linking; other
approaches are acceptable (cf. Reference Supplement)
• Does not present a mechanism which would provide
accreditation of examinations; this requires a specific
rationale of its own
• Is not a recipe; does nor provide rule-of-thumb
shortcuts
• Does not guarantee success in linking examinations to
the CEF; the linking activities must be of high standard
Overview of Linking Scheme
• Chapter 1: Introduction gives a very good overview
(“read me first”)
• Familiarisation: thorough knowledge of the CEF
“philosophy” and its resources (Chapters 2 & 3, pp. 1328)
• Specification: aims, CEF audit of coverage
– Objectives, coverage, sections, weighting, text & item
types, marking criteria, reporting (Chapter 4, pp. 29 – 63)
• Standardisation: understanding CEF levels
– Training with calibrated examples, benchmarking local
examples for standardisation training; standard setting
(Chapter 5, pp. 65-98)
• Validation: Collection and analysis of data
– Internal (test characteristics) External (relation to
calibrated tests and descriptors
– Actually ALL four activities are essential elements in
building and substantiating an argument for a high quality
work with linkage to the CEF
– Chapter 6, pp. 99-122
What do we claim we are testing? (What is our
construct?)
We now need to check what out examination really
looks like. (check the claim; awareness raising)
If we wish to relate our examination to the CEFR, we
need to profile our examination drawing on its
descriptive scheme and scales.
Preliminary Pilot Manual proposes a procedure to
profile examinations vis-a-vis the CEFR: SPECIFICATION (Ch. 4, 29-63)
We need to use the descriptive resources the CEFR
provides as much as is possible, and supplement the
scheme if it is not sufficient for profiling the exam
Specification
• General description of exam (Form
A1): purpose, target population,
domains etc (”this is our product”)
• Test development procedures,
marking, reporting results
• More specific description in terms of
communicative activities: reception,
interaction, production, mediation
• More specific description in terms of
aspects of language competences in:
reception, production, mediation ...
Specification sample
C2
C1
B2
B1
A2
A1
LC
SP
Inter- RC
action
Wr
Specification sample
C2
C1
B2
B1
A2
A1
Personal Public
Occupational Educational
How do our exam results relate to CEFR levels?
How do I know what level our exam results correspond to?
How do I know my B2 corresponds to your B2?
We obviously need to standardise our procedures (in Europe)–
STANDARDISATION (Ch. 5)
We need to set standards, including cut-off scores.
We need rigorous rater training: check intrarater consistency,
interrater consistency
We need to choose scales: CEF scales; validation of local scales
We need benchmarks: European, ”local”
We will choose to apply one of the several possible methods of
standard setting; Manual suggests one approach – Reference
Supplement, Section B, contains more information
External check of standard
setting: verification
• What it the evidential basis of our linking?
• Internal validation
• External validation
• Chapter 6 in the Manual
• Reference Supplement
Link to CEF
Through
another
scale
(Fellianka Kaftandjieva, Belgrade,
2003)
Direct
Standard
Setting
Through
another
test
VISUAL REPRESENTATION OF PROCEDURES TO RELATE EXAMINATIONS TO THE CEF
"BUILDING AN ARGUMENT"
Recommended procedures encourage
alignment of examinations to CEF at
differing degrees of rigour appropriate
to different testing contexts and to the
extent permitted by resources and
expertise available.
THEORETICAL RATIONALE
QUALITATIVE VALIDATION PROCEDURE
SPECIFICATION
QUALITATIVE VALIDATION PROCEDURE
QUANTITATIVE VALIDATION PROCEDURE
STANDARDISATION
EMPIRICAL VALIDATION
OF EXAMINATION CONTENT
THROUGH TRAINING
THROUGH DATA ANALYSIS
Internal familiarisation with CEF
Familiarisation training
Data collection
Training in assessing Training in judging the
performance to CEF difficulty of test items
using standardised to CEF standardised
samples
items
Internal validation:
Benchmarking local
performance samples
to CEF
External validation:
Internal validity: description and analysis of
° General examination content;
° Process of test development;
° Marking, grading, results;
° Test analysis and post-examination review.
External validity: relate
° General examination description to CEFscale;
° Description of communicative activities tested
to CEF scales;
° Description of aspects of communicative
language competence tested to CEF scales.
CLAIM of link to CEF
Item difficulty
Estimation
to CEF
Dissemination and implementation
CLAIM (reinforced
CLAIM (confirmation on basis of
on basis of specification)
specification and standardisation)
Possible outcome:
European chart which shows how
validated examination results are
located on the Framework scales.
Not only ”My B1 is your B1” but also
”My own exam´s grade 6 (or ”Good”)
corresponds to B2”, grade 7 (”Very
good”) is C1.1 ....
Matr
C1.2 Laud
C1.1 Eximia
B2.2 Magna
B2.1 Cum l..
B1.2 Lubent.
B1.1 Approb
A2.2
A2.1
A1.2
A1.1
Improb
Gr 9
VWO/RC VWO/
LC
VWO/
SP
Havo/
RC
Havo/SP
What the Manual is about?
• A tool to promote competence to link
examinations to CEF; a guide
• A tool for raising awareness about the
particular examination (a tool for reflection)
• A tool with a certain approach outlined
• Represents one particular approach; other
approaches may be equally valid (need to
present rationale and evidence)
• Is mainly concerned with standard setting (not
test development or construction)
• Requires thoughtful application
• Presupposes the provision of extensive and
adequate empirical evidence to substantiate
any claims
Contributions welcome:
• FEEDBACK on reading the Manual
• PILOTING on the basis of using the Manual
• CASE STUDY – writing up experience of using the
Manual
• PERFORMANCE SAMPLES – to be collected for
speaking + writing; LC and RC items
These will contribute to a revised and extended
Manual “kit”
www.coe.int/lang
First feedback Seminar: Strasbourg, April 15-16,2004
Final word
• Many claim linkage to the CEF but
few produce evidence, and so far –
arguably - no evidence is completely
adequate (DIALANG probably the
best exception)
• So:
• Quis custodiet ipsos custodes?
• Caveat emptor!
• When there are good reports of
linkage, what happens to them next?