Transcript Developing Descriptors Brian North www.eurocentres.com; [email protected]; www.eaquals.org [email protected] Stages in Developing Descriptors 1. Conceptualisation > Clarifying the construct.

Developing Descriptors
Brian North
www.eurocentres.com;
[email protected];
www.eaquals.org
[email protected]
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Conceptualisation
CEFR
> CEFR descriptors: observable, functional outcomes
“competence” descriptors also mainly observable proficiency
> Interaction (BICS) / Production (CALP)
> Illustrative videos of 16-18 yr olds: difficulty with BICS “C2”
LoS more complex than modern languages
> Language aspects / non-language aspects
> Discourse emphasis: genres; cognitive skills
> Developmental – linked to cognitive growth
Far less known about LoS than modern languages
> 20 years experience with descriptors 1975-1995
> 20 years developing descriptive scheme 1975-1995
Characteristics of LoS or (C) ALP
We/you know it involves more:
> specific, formal, abstract
> explicit, detailed, conventionalised (= expectations)
> cohesive and structured (e.g. sequencing)
> coherent (goal-oriented)
> planning, self-monitoring, internal feedback, editing
> rhetorical skills and structures, strategies
BUT
> How much is really known about academic discourse?
> Reception of exposition by the teacher
> Interaction in class
> Production by the teacher
> To what extent are skills transversal – a common core?
Need for Collaboration & Research
Vollmer
>
>
>
>
Pooling expertise and materials
Corpus of curricula and examination papers
Classroom observation and research
Interviews with teachers
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Analyse textbooks
Need for Collaboration & Research
CEFR – Preparatory Work
>
>
>
>
Clarify concept: 1975 (Threshold) – 1992 (Proposal)
Experience with descriptors (BN: 1983-93)
Classroom discourse analysis (BN: 1984-9)
Involvement of stakeholders (Working Party 1992-6)
CEFR – Project Design
>
>
>
>
Analyse and align existing systems
Interactive definition of categories with Authoring Group
Swiss National Research Project
Involvement of teachers in qualitative validation - Workshops
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Key Questions I
> Relationship to:
> content standards
> European Qualifications Framework
> CEFR
> Categories to be described
>
>
>
>
Transversal categories as in Table 5 of ERDLE proposal (p52)
Subcategories of Recep, Inter, Prod, Interp, Evaluation, Mediation?
Cognitive skills & strategies from Situation analysis (Beacco et al)
What else?
> Style
> concrete-salient features (CEFR-style) / abstract
> Length – including assumptions
“Can make a complaint”: B1
> broad-holistic / atomistic-analytic / both (Fleming)
Key Questions II
> Thresholds to be described
>
>
>
>
expected language proficiency levels
types of discourse
stages of cognitive development
strategies
> How to deal with “difficult parts” (non-language) e.g. Bildung
> consideration of others
> critical thinking, sound judgement and courage to express it?
> flexibility in thinking and argumentation
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Construction
> Creating a classified bank of descriptors:
> Collate / deconstruct all source systems
> Eliminate doubles, redundancy
> Identify gaps
> Editing and drafting
>
>
>
>
>
Confirm style
Harmonise use of verbs (not done in CEFR English!)
Harmonise formulations
Create variations (for missing levels)
Author missing categories
> Organisation
> Classify with serial numbers
> Translation to key languages / check translations with plurilinguals
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Set Summarise developing proficiency
Qualitative Validation
> Analysis of teachers discussing proficiency:
> Video of two learners
> Who is better? Why? Justify your choice
> “Repertory grid” analysis of categories teachers use to compare
quality
> Sorting descriptors into categories
>
>
>
>
Pile of (maximum 60) descriptors
Set of (maximum 4) envelopes labelled with the relevant categories
Discard envelope
Tick ones that are clear, relevant and useful
> Sorting descriptors into levels
> Pile of (maximum 15) descriptors for same category
> Set of (CEFR 6) envelopes labelled with levels
> Discard envelope / Tick ones that are clear, relevant and useful
Quantitative Validation - Purpose
> To construct a scale from the descriptors for the “core
construct”
> To bolt onto / link to this scale sets of descriptors for
categories that prove to be less core areas
> To find out/confirm what level specific descriptors are
> To discover which descriptors do not work
> To confirm communality of the interpretation of the
descriptors across:
> Languages
> Regions / countries / systems
> Educational sectors
Quantitative Validation = Steps
1. Identify good/best descriptors from the pool after the
qualitative validation
2. Confirm the supposed “level” of these descriptors
3. Create a set of overlapping checklists of c50 descriptors
(like ELP checklists); each checklist targeted at a “level”
4. Define a rating scale: Yes/No; 0-4 for the descriptors
5. Identify classes at approximately the right level for each
checklist
6. Arrange teacher assessment and/or self-assesment with
the checklists
7. Collect minimum 150 examples of each checklist
8. IRT Rasch Model “Rating Scale Analysis” to build scale
9. Eliminate descriptors with 80%+ or 20%- (Rasch problem)
Anchor Design: CEFR (North 2000)
Data Collection:
Quest. C
Quest. B
Quest. A
Recommended Design (after De
Jong)
Data Collection:
Vertical Scale of Descriptors
Vertical Scale of Descriptors
Extending the Core Scale I
Quantitative Validation - Prerequisites
> Construct is well-defined – common understanding of what
is being described/rated/scaled
> Descriptors are well-formulated, clear and relevant
> Teachers/learners are capable of making judgements about
the areas concerned
> There is a solid anchor design in the data collection
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Setting Thresholds between Levels
> Marking out equal intervals on the scale
> Identifying „jumps“ in content described, gaps between
clusters of descriptors
> Comparing to original scale author intention
> Comparing to Waystage, Threshold, Eurocentres,
Cambridge exam levels
> Fine-tuning for equal intervals
> Checking for consistency, coherence
CEFR 3.6 Salient Characteristics A2
The majority of descriptors stating social functions:
> greet people, ask how they are and react to news
> handle very short social exchanges
> discuss what to do, where to go and make arrangements
Descriptors on getting out and about:
> make simple transactions in shops, banks etc.
> get simple information about travel and services
CEFR 3.6 Salient Characteristics B1
Maintain interaction and get across what you want to:
> give or seek personal views and opinions
> express the main point comprehensibly
> keep going comprehensibly, even though pausing
evident, especially in longer stretches
Cope flexibly with problems in everyday life:
> deal with most situations likely to arise when travelling
> enter unprepared into conversations on familiar topics
CEFR 3.6 Salient Characteristics B2
Effective argument:
> account for and sustain opinions in discussion by
providing relevant explanations and arguments
> explain a viewpoint on a topical issue giving the
advantages and disadvantages of various options
Holding your own in social discourse:
> interact with a degree of fluency and spontaneity that
makes regular interaction with native speakers possible
> adjust to changes of direction, style and emphasis
A new degree of language awareness:
> make a note of "favourite mistakes" and monitor speech
for them
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency
Appendix
United States
“No Child Left Behind”
2001-7
US “No Child Left Behind”
2001-7
> States have a legal duty to provide the support to
ensure that every child is proficient in the academic
language they need to be successful at school.
> Must test this.
> Must be at least a grade above and a grade below
proficient.
(not just the usual US master / non-master)
US “No Child Left Behind”
>
>
>
>
2001-7
No overall framework or common reference points
Testing-led: dozens of consortia
No time for research
No systematic definition of the construct ALP
> Confusion with “English Language Arts” (= creative writing for native
speakers)
or
> Elaborated from language used in subject content standards
> No definition of “proficient:” 15 significantly different
interpretations
> Some states 3, some 4, some 5 grades; all different names,
numbers, concepts
= CHAOS
Need for Collaboration & Research
Vollmer
>
>
>
>
Pooling expertise and materials
Corpus of curricula and examination papers
Classroom observation and research
Interviews with teachers
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Analyse textbooks
US Experience
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Analyse textbooks
Analysing & Aligning Standards
> Assumptions in Subject Standards:
> Elementary School: observe, analyse, compare,
describe, record
> Middle School: identify, recognise, compose, explain
> High School: recognise, describe, explain
(Bailey and Butler 2003)
> “Extracting the language features embedded in the
content standards presented significant challenges ….
have been successful developing
standards-standards linkages that involve both language
and content standards BUT “procedures to establish such
linkages … remain to this day in their infancy
> Bailey, Butler and Sato (2005)
(Chaloub-Deville 2008)
US Experience
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Analyse textbooks
Classroom Research
From 2001:
> Analysis of functions in science classrooms
> Teachers
> Students
> Repair strategies
(Bailey and Butler 2003)
> BUT
> All tests produced before any research results were
available – even in consortia aware of the problem
(Chaloub-Deville 2008
US Experience
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Analyse textbooks
Teacher Expectations
> Students must learn acceptable ways of presenting
information to the teacher – not usually explicitly taught
> Very little study
> “Teachers are rarely explicitly aware of their language
expectations”
> Dropped the idea of teacher interviews because
“anecdotal” unreliable information
(Bailey and Butler 2003)
US Experience
US “No Child Left Behind” (Bailey & Butler)
> Analyse and align existing content and language standards
> Need for observation and research on Teacher Talk
> Need for empirical analysis of performance of mother tongue
and second language students
> Interviews with teachers (expectations: Recep; Production)
> Textbooks
US “No Child Left Behind”
>
>
>
>
2001-7
No overall framework or common reference points
Testing-led: dozens of consortia
No time for research
No systematic definition of the construct ALP
> Confusion with “English Language Arts” (= creative writing for native
speakers)
or
> Elaborated from language used in subject content standards
> No definition of “proficient:” 15 significantly different
interpretations
> Some states 3, some 4, some 5 grades; all different names,
numbers, concepts
= CHAOS
Stages in Developing Descriptors
1. Conceptualisation
> Clarifying the construct. What are we describing?
> Collecting relevant example, systems
> Deciding categories in the descriptive scheme
> Clarifying key questions
2. Construction
> Creating the descriptor pool
> Editing, drafting – filling gaps in the description
3. Validation
> Qualitative: through iterative workshops with teachers
> Quantitative: through IRT scaling of use in assessment
4. Interpretation
> Set thresholds between levels
> Summarise developing proficiency