Transcript As slippery as an eel? Assessing speaking and writing Part One

As slippery as an eel?
Assessing speaking and writing
Part One
Ülle Türk
University of Tartu
Estonian Defence Forces
23rd CSW, Tampere, 27-29 March 2009
Testing writing?









Fill in the gaps with suitable words so that the text is true for you.
Re-write the text in the future tense.
Fill in the form with the information given in the box.
Read the letter and write an answer.
Write an essay on the topic “Why study English?”
Study the pictures, put them in the order you think best and write
the story.
Read the text and write a short summary of it.
You bought a new dictionary yesterday, but found later that several
pages were missing. Write a letter to the manager of the shop
informing him of the problem and telling him what you want him to
do about it.
Read the basic facts about Australian history and then write a short
report.
2
Questions


What is it exactly that we assess when we say we
assess students’ speaking and writing skills?
How do we arrive at a common understanding of
what is ‘good’ writing, what is a ‘good’ oral
presentation or what constitutes ‘good’ spoken
or written communication?
3
Terms

Assessment




Formal  informal
Continuous  fixed-point
Formative  summative
Testing





Achievement
Proficiency
Diagnostic
Placement
High-stakes  low-stakes
4
Assessment/ test quality





Validity
Reliability
Authenticity
Washback
Practicality
5
Validity: definitions


A good test needs to be valid. = It must test
what it is meant to test.
an integrated evaluative judgement of the degree
to which empirical evidence and theoretical
rationales support the adequacy and
appropriateness of inferences and actions based
on test scores.

S. A. Messick (“Validity” in R. L. Linn (ed.) Educational
Measurement. 1989, p. 13)
6
Validity




Does the test match the curriculum, or its
specifications?
Is the test based adequately on a relevant and
acceptable theory?
Does the test yield results similar to those from a
test known to be valid for the same audience and
purpose?
Does the test predict a learner’s future
achievements?
7
Validity





Content validity
Construct validity
Criterion-related validity
Predictive validity
Construct validity is indeed the unifying concept that
integrates criterion and content considerations into a
common framework for testing rational hypotheses
about theoretically relevant relationships.

Messick, S. A. “Test validity and the ethics of assessment.”
American Psychologist 35, 1980, p. 1015
8
Threats to test validity


construct irrelevant variance
construct under-representation
9
Factors affecting validity






Lack of specifications
Lack of training of item/ test writers
Lack of / unclear criteria for marking
Lack of piloting/ pre-testing
Lack of detailed analysis of items/ tasks
Lack of feedback to candidates and teachers
10
Communicative competence 1

Canale & Swain (1980), Bachman (1990):
language knowledge types



Linguistic knowledge
Discourse knowledge
Sociolinguistic knowledge
11
Grabe & Kaplan (1996): Model of
Writing

Components of language knowledge relevant to
writing




linguistic knowledge: written code, morphology,
vocabulary, syntax
discourse knowledge: cohesion, structure, genre
sociolinguistic knowledge: functional uses of writing,
register, situational parameters
Influential in teaching and testing of writing
(e.g., Weigle, 2002)
12
Communicative competence 2

Bachman & Palmer (1996): communicative
language ability


Language knowledge
Strategic competence
13
Douglas (2000): Specific Purpose
Language Ability

Language knowledge


grammatical knowledge
textual knowledge





functional knowledge
sociolinguistic knowledge
Strategic competence





rhetorical organization
cohesion
assessment
goal setting
planning
control of execution
Background knowledge

discourse domain
14
Common European Framework of
Reference for Languages: Learning,
teaching, assessment (2001)


General competences
Communicative language competences
15
General competences
Declarative
knowledge
knowledge of
the world
sociocultural
knowledge
intercultural
awareness
Skills and
know-how
‘Existential’
competence
Ability to learn
practical skills
(social, living,
vocational/
professional,
leisure skills)
attitudes
language and
communicative
awareness
intercultural skills
personality factors study skills
motivation
values
beliefs
cognitive styles
general phonetic
awareness and skills
heuristic skills
16
Communicative language
competences
Linguistic
competences
Sociolinguistic
competences
Pragmatic
competences
lexical competence
linguistic markers of social discourse competence
relations
grammatical
competence
politeness conventions
semantic
competence
expressions of folk wisdom
phonological
competence
register differences
orthographic
competence
dialect and accent
functional competence
orthoepic
competence
17
Communicative language
activities and strategies





productive activities and strategies
receptive activities and strategies
interactive activities and strategies
mediating activities and strategies
non-verbal communication



practical actions
paralinguistics
paratextual features
18
Oral production


public address (information, instructions, etc.)
addressing audiences (speeches at public meetings,
university lectures, sermons, entertainment, sports
commentaries, sales presentations, etc.)





reading a written text aloud
speaking from notes, or from a written text, or from visual aids
acting out a rehearsed role
speaking spontaneously
singing
19
Spoken interaction









transactions;
casual conversation;
informal discussion;
formal discussion,
debate;
interview,
negotiation;
co-planning;
practical goal-oriented co-operation.
20
Oral mediation



simultaneous interpretation (conferences,
meetings, formal speeches, etc.)
consecutive interpretation (speeches of welcome,
guided tours, etc.)
informal interpretation:





of foreign visitors in own country
of native speakers when abroad
in social and transactional situations for friends
family, clients, foreign guests, etc.
of signs, menus, notices, etc.
21
CEFR levels
The Common European Framework of Reference (Council
of Europe 2001) defines communicative proficiency

At six levels, arranged in three bands
A1
A2
B1
B2
C1
C2

in relation to six skills: listening, reading, spoken
interaction, spoken production, written interaction,
written production

in the form of “can do” statements
22
Getting to know the levels

The self-assessment grid is not enough

More specific scales:



CEFR Ch 4: descriptors of communicative activities

CEFR Ch 5: descriptors of linguistic competence
The ELP (European Language Portfolio)
Manual: Relating Language Examinations to
the Common European Framework of Reference
for Languages: Learning, Teaching, Assessment
(CEFR)
23
I can deal with most situations likely to
arise whilst travelling in an area where
the language is spoken. I can enter
unprepared into conversation on topics
that are familiar, of personal interest or
pertinent to everyday life (e.g. family,
hobbies, work, travel and current
events).
Self-assessment grid (CEFR and standard adult passport)
24
CercleS ELP: goal-setting and selfassessment checklists
Level B1
My
next
goal
*
**
***
I can readily handle conversations on most topics that are familiar or of personal
interest, with generally appropriate use of register
I can sustain an extended conversation or discussion but may sometimes need a
little help in communicating my thoughts
I can take part in routine formal discussion on familiar subjects in my academic or
professional field if it is conducted in clearly articulated speech in standard dialect
I can exchange, check and confirm factual information on familiar routine and
non-routine matters within my field with some confidence
I can express and respond to feelings and attitudes (e.g., surprise, happiness,
sadness, interest, uncertainty, indifference)
I can agree and disagree politely, exchange personal opinions, negotite decisions
and ideas
I can express my thoughts about abstract or cultural topics such as music or films,
and give brief comments on the views of others
I can explain why something is a problem, discuss what to do next, compare and
contrast alternatives
I can obtain detailed information, messages, instructions and explanations, and
can ask for and follow detailed directions
I can handle most practical tasks in everyday situations (e.g., making telephone
enquiries, asking for a refund, negotiating purchase)
I can provide concrete information required in an interview/consultation (e.g.,
describe symptoms to a doctor), but with limited precision
I can take some initiatives in an interview/consultation (e.g., bring up a new
subject) but am very dependent on the interviewer to provide support
25
Questions to ask

What competences should my students have in






Spoken interaction
Spoken production
Written interaction
Written production
What tasks should they be able to perform to
demonstrate their mastery of the competences?
How well should they be able to perform them?
26
Reliability


A test needs to be reliable. = It must produce
consistent results at different times.
NB! A test that is not reliable cannot, by
definition, be valid.
27
Reliability





If I take the test again tomorrow, will I get the
same result?
If I take a different version of the test, will I get
the same result?
If the test had had different items, would I have
got the same result?
Do all markers agree on the mark I got?
If the same marker marks my test paper again
tomorrow, will I get the same result?
28
Factors affecting reliability






Poor administration conditions – noise, lighting,
cheating
Lack of information beforehand
Lack of specifications
Lack of marker training
Lack of standardisation
Lack of monitoring
29
References




Bachman, Lyle F. (1990) Fundamental Considerations
in Language Testing, Oxford University Press, Oxford.
Bachman, Lyle F. and Palmer, Adrian (1996) Language
Testing in Practice, Oxford University Press, Oxford.
Cushing Weigle, Sara (2002) Assessing Writing,
Cambridge University Press, Cambridge.
Douglas, Dan (2000) Assessing Languages for Specific
Purposes, Cambridge University Press, Cambridge.
30