Transcript Applying CEFR principles to Testing and Assessment in practice

Applying CEFR principles
to testing and assessment
in practice
ACERT Conference
15th March 2014, Prague
Mila Angelova
Aims of the session
• To give you a short update on EAQUALS
operation and activities
• To introduce you to the EAQUALS
Certification Scheme
• To (re)-introduce you to the most salient
features of the six global levels according
to CEFR
• To assess two oral performances for
standardisation purposes
The Association
• Founded in 1991
• A company registered in UK as charity
• International NGO with participatory
status at the Council of Europe (2006)
EAQUALS Mission
The mission of EAQUALS is:
• to contribute to the development of
excellence in language education
• to be the internationally recognised
authority for quality standards and quality
assurance in language education and
training
EAQUALS Values
•
•
•
•
Plurilingualism
Intercultural understanding
International cooperation
Lifelong learning
EAQUALS Members
A network connecting 35 countries
Accredited members x 112
25 languages, private and state sector
Associate members x 27
British Council; Instituto Cervantes; CIEP
France; Goethe-Institut; Cambridge
Assessment; Trinity London; g.a.s.t. Germany;
12 national quality associations
Project partners x 5
Centro di Ricerca e di Servizio + Universita per
Stranieri di Perugia
School of Languages, Sabanci, Turkey
Individual membership (2011)
EAQUALS Expertise
• Quality management:
- design of quality accreditation schemes
- training and standardisation
- capacity building at local level
• The Common European Framework of
Reference for Languages (CEFR):
- descriptors, curriculum, assessment,
- portfolio, certification
• Teacher management and professional
development:
- teacher competences (profiling grid)
- teacher training and development
- academic management
EAQUALS Activities
International accreditation of language
education
- standards, inspections; recommendations
Support and pathways for institutional
development
- consultancy; self-assessment; training resources
An international professional network
- regular conferences; involvement in funded projects
Certificates of achievement – CEFR levels
- training support, numbered certificates, website archive
EAQUALS Projects
• Quality management
- ISO: standard for providers of language learning
- Council of Europe: code of practice for integration of
adult migrants
- Leonardo: Quality evaluation of on-line teacher
training, Quality Guidelines
• CEFR
- Core Inventories (English; French)
- Listening and Reading scenarios (Eng, French)
- EAQUALS Certificate of achievement
• Teacher management & development
- profiling grid for teachers (EPG Project)
- teacher development framework
EAQUALS Accreditation
•
•
•
•
international and plurilingual
across educational sectors
consultative and developmental
based on EAQUALS Charters
EAQUALS Standards
•
•
•
•
designed for language education services
12 main areas of assessment
specific indicators of quality for each area
50 key assessment criteria:
- clear and transparent
- derived from the Charters
Becoming a Member
Attend an EAQUALS Conference
and get to know EAQUALS: details on www.eaquals.org
Complete a Self-assessment
and identify areas that may need improvement
Have a One-day Advisory Visit
and receive consultancy on achieving EAQUALS standard
Arrange your First Inspection
after working through the points raised
The EAQUALS
Certificate of Achievement
A Guarantee for Quality Assessment
Brief introduction to the scheme
Main aims of the scheme
• To increase EAQUALS’ profile: the first
organisation to award CEFR certificates of
achievement across different languages Europewide
• To help members differentiate themselves from
their competitors and demonstrate their
competence
• To open the possibility, at a second stage, of
accrediting CEFR certificates awarded by schools
that are not yet accredited members of
EAQUALS
Core message
• A guarantee for quality assessment in the
accredited institutions by thoroughly
screening all assessment processes
• CEFR-implemented approach to syllabus
and curriculum design
• Feasibility of standardisation and
moderation techniques used by the
institution
EAQUALS
Certificate of Achievement
• Phase 1: voluntary – Accredited
Members only: accreditation carried out as
a “top up” to the EAQUALS inspection
• Phase 2: compulsory for Accredited
Members; voluntary for schools which are
members of Associate Members: the
accreditation process is fully integrated into
the EAQUALS inspection and carried out as
a top up to inspections of Associate
Members
Why EAQUALS Certificate?
Benefits for:
• the institution
• the course participants
• the stake holders
• EAQUALS profile
Main prerequisites to apply
for the Certification Scheme
A) CEFR-based curriculum
B) Standardisation training
C) Moderation techniques
CEFR-based curriculum
1. CEFR-related level descriptions and
syllabus documents
o the levels referenced to CEFR (CEFR criteria grids
adopted or adapted to local circumstances)
o learning aims specified by „can-do” statements based on ELP or CEFR scales and adapted to the
institution’s own needs
2. Written guidelines for teachers on levels,
teaching and assessment
3. Application of CEFR in teaching
o coherence between curriculum and syllabus aims and
classroom practice
Implementing CEFR-based
curriculum and syllabuses
• Core of the syllabuses is the practiceoriented background of the CEFR
• Can-do-statements developed for
each type of course and level / sublevel
• Can-do-statements serve as a basis
for tests development
• Can-do-statements used as a selfassessment instrument
Standardisation training
- requirements
•
•
•
Familiarisation with CEFR levels and
descriptors
Systematic standarisation training
with CEFR illustrative samples
(DVDs or scripts: the EAQUALS
Standardisation packs – for English,
French and German)
Procedures for CEFR-novice and
CEFR-experienced teachers
Examples
of standardisation training
• Means of writing standardisation (making
use of local samples)
• Means of speaking standardisation (with
official samples)
• Standardisation of the placement testing
(with standardisation videos and other
CEFR-related tasks)
Moderation
Systematic reduction of
subjectivity in the assessment.
Moderation techniques -
requirements
• Collective assessment techniques
o grading by two assessors
(all performances, a structured sample
or with less-experienced staff), or
o assessing initially as a panel
• Quality control
o liaison between teachers
o systematic monitoring by academic
managers
Examples of moderation
techniques
• Doing buzz observations during testing
periods to monitor application of
assessment standards
• Monitoring inexperienced teachers’
assessment of spoken production and
interaction / written work
Assessment procedures
should include at least two of:
a) teacher judgements through continuous
assessment in relation to CEFR/ELP
descriptors
b) performances in specific spoken/writen
tasks assessed with CEFR-related criteria
c) use of tests (progress test giving an
indication of good performance - e.g.15/20),
or an independent, validated “CEFR level
test” reporting a CEFR level
Procedure to accredit
assessment processes
Step 1: Introduction of a CEFR-based
assessment approach
»Standardisation training
»Assessment procedures
»Moderation techniques
Step 2: A formal application
Step 3: Evaluation by the EAQUALS
Assessment Panel
Introduction
Please, discuss the following questions
with the person sitting next to you:
o How often do you do standardisation
training at your institutions? Is it enough?
o Do all teachers have to take part in it? If
not, why?
o What problems do you face when you do
standardisation training?
Warmer
• Look at the descriptors for spoken
production for B1 level. Fill in the gaps
with the missing salient features for the
level.
• Compare your ideas with the person
sitting next to you.
CEFR Speaking Skills
overview
• Arrange the descriptors for the six basic
•
levels in the correct order.
Compare with your answers from the
previous stage with the actual descriptor
for B1
Oral sample
• Watch a short performance for 3-4 min.
Decide on the speakers’ level.
Bell Krakow_Anna & Magda.mpeg
• Write a short description of the speaking
abilities of the students in the video.
• Compare your answers with the CEFR
descriptor.
• Provide a rationale for your decision by
referring to the most salient features of
the descriptors.
Criteria for speaking assessment
• List the basic criteria that you use for
speaking assessment.
• Compare your ideas with a suggested set
of criteria, used at one EAQUALS
institution.
Sample mark sheet
Candidate
A
Candidate B
1 2 3 4 5
1 2 3 4 5
Pronunciation: Can produce a limited number of words / phrases in a
relatively recognizable manner and make her/himself understood in
simple everyday situations but there is very noticeable interference from
mother tongue in terms of individual sounds, word and sentence stress
1 2 3 4 5
1 2 3 4 5
Interaction: Can ask and answer questions about personal details.
Can interact in a simple way but communication is totally dependent on
repetition, rephrasing and repair
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
Criteria
Discourse management Can manage very short, isolated, mainly
pre-packaged utterances, with much pausing to search for expressions,
to articulate less familiar words, and to repair communication. Can link
words or groups of words with very basic linear connectors like “and” or
“then”
Grammar and vocabulary: Has a very basic repertoire of words and
simple phrases related to personal details and particular concrete
situations. Shows only limited control of a few simple grammatical
structures and sentence patterns in a memorized repertoire.
Global achievement: Overall impression mark
Initial standardisation
Task 1:
• Watch the standardisation video again and
assess the speakers’ performance using
the mark sheet
• Discuss your grades with a partner, then
compare ideas with another pair
Thank you!
www.eaquals.org