Transcript Slide 1

Using assessment
For understanding and improving
Language education
John M. Norris
University of Hawai´i at Mānoa
“Ich lerne sehen”
~R. M. Rilke
Dealing with change
In language education
Rethinking the value of language education
“English language learners (ELLs) are the fastest
growing student population in America. Today, one out
of every nine students is learning English as a second
language…By 2025, English language learners will make
up one out of every four students in our classrooms.”
Margaret Spellings (2005) – U.S. Secretary of Education
Rethinking the value of language education
“Americans need to be open to the world; we
need to be able to see the world through the eyes
of others if we are going to understand how to
resolve the complex problems we face.”
Daniel Akaka, U.S. Senator from Hawaii
Rethinking the value of language education
“The stakes are very high. The challenges college
graduates face over the next 50, 60, and 70 years will
intensify. They involve the understanding of different
cultures, the balance of global power, the depletion of
environmental resources, and the ability to continue to
grow in a rapidly changing world. It is because the
stakes are so high that I believe a focus on student
learning is so critical and why FL programs have such an
important role to play.” (Chase, 2006, p. 585).
Geoffrey Chase – Dean of Undergraduate Studies
San Diego State University
Rethinking the value of language education
Standards
Standards for foreign language learning in the
21st century
High-quality teachers
NCATE – TESOL/ACTFL Teacher
Development Program Standards
MLA White Paper
Foreign Languages and Higher Education: New
Structures for a Changed World
AAAL, ACTFL, MLA, Northeast Conference
Modeling Representation of Foreign Language
Education at the Federal Level in the United States
Rethinking the value of language education
How do we
bridge the
content vs.
language
divide?
Are we seeking
communicative competence
or quality of mind?
Language
Education
Should we adopt
an instrumental,
aesthetic,
ethical, or moral
justification for
FL learning?
What’s the
relationship between
How do we ensure
L1, L2, and academic
advanced language
achievement?
learning?
Who trains teachers
how?
Determining the value of language education
“What we assess is what we value”
--Lauren Resnick
“How we choose to assess will
determine what gets valued”
--Norris
Why worry about assessment?
Top 3 sources of pressure for
assessment in college FL programs:
1.University administration
2.The dean
3.Accreditation process
“As part of its reaccreditation, the
university has required all
undergraduate programs to
create and implement
outcomes-oriented
assessment plans.”
(survey respondent)
“Time-consuming. Takes away from the business of teaching.
Many aspects of learning can’t be measured.”
(survey respondent)
Why worry about assessment?
NO
CHILD
COLLEGE
STUDENT
LEFT
BEHIND
College
Leaving
Exam
“Does the Spellings Commission think
about language education at all?”
Michael Holquist (ADFL 2007 Summer
Seminar West)
Why worry about assessment?
A
S
S
E
S
S
M
E
N
T
Frequent teacher reactions to assessment:
“utilitarian technocracy”
“thoughtless mastery”
“Orwellian, punitive process”
“bureaucratic nonsense”
“fails to improve teaching”
“duplicates existing efforts”
“reductionist”
“discourages teaching skills that are difficult to measure”
Why worry about assessment?
FLs
need to
develop
capacity for change
faculties of problem-solving
BUT…
Perception
of
assessment
managerial model of efficiency
not appropriate to academe
Domna Stanton (2007, ADFL Summer
Seminar West)
Why worry about assessment?
Despite such problems, college FL survey respondents
desired increased use of assessments and related
processes for:
1. Understanding & improving program outcomes
2. Understanding & improving program functions
3. Improving FL education on the whole
4. Understanding & Improving the worth of the program
5. Raising awareness about FL programs
Why worry about assessment?
And some expressed a professional ethic to engage in
outcomes assessment:
“We have a social and moral responsibility towards our
students and towards society at large to state as clearly as
we can what it is that we do for them and why what we do
is valuable.”
(Anonymous survey respondent)
Why worry about assessment?
NCATE Teacher Preparation
Program Standards
ACTFL
TESOL
•Language, linguistics
•Language
•Cultures, literatures
•Culture
•Acquisition, instruction
•Planning, managing
•Integration of standards
•Assessment
•Assessment
•Professionalism
•Professionalism
Assessment as change agent
Within this milieu, what is the role to be
played by assessment?
Key
Challenge:
Define
Delimit
Detract
Dismiss
Value
of
Language
Education
Key
Opportunity:
Enable
Enhance
Engender
Empower
Traditions, trends, and the status quo
In language education assessment
100%
90%
80%
Assessment articles
70%
60%
%Assess
%FL
50%
%USCFL
40%
%Proficiency
30%
20%
10%
0%
ADFL
FLA
LT
MLJ
UP
Percentage and type of FL assessment/evaluation articles in five
journals, 1984-2002
35%
US college FL assessme nt
30%
25%
20%
15%
10%
5%
0%
ADFL 84-93 ADFL 94-02 FLA 84-93 FLA 94-02
LT 84-93
LT 94-02
MLJ 84-93 MLJ 94-02
UP 84-93
UP 94-02
High, low, and average yearly percentage of articles on
U.S. college FL assessment/evaluation, 1984-1993 and 1994-2002
Received traditions of language assessment
Swender (2002), on FL teacher professional development in assessment:
“After all, if teachers do not know how to measure what students can do
with language, how will they be able to determine whether their students
are measuring up to the expectations of the 21st century”.
Technocratic measurement problem
Received traditions of language assessment
WebCAPE Foreign Language
Placement Exam
Commercial assessment problem:
One size fits most purposes &
settings
Received traditions of language assessment
TSE – SPEAK – SLEP - TOEIC
Commercial assessment problem:
One size fits most purposes &
settings
Received traditions of language assessment
When we do address
assessment, in…
We focus on…
• FL teaching texts
• How to measure
the four skills
• State of the art
collections
• How to rate oral
proficiency
• Professional
organizations
• How to test cultural
knowledge
• NCATE/ACTFL
teaching standards
• How to assign
grades
• Accountability
requirements
• How to give
feedback
Without much
concern for…
• WHY are we
assessing?
• What assessment
methods fulfill what
purposes?
• How do we
contextualize it?
• What good is it
doing?
Advanced vocabulary learning: individual vocabulary notebooks
Learners identify
meaningful
words to study
Based on
individual needs
or interests
Record 30/week
in vocabulary
notebook
Revisit, review,
and acquire 30
over the week
From: Moir & Nation (2002)
Advanced vocabulary learning: individual vocabulary notebooks
Learners identify
meaningful
words to study
Based on
individual needs
or interests
Learners identify
Record 30/week
unsuitable words
in vocabulary
to study
notebook
Revisit, review,
Cram the
and acquire 30
night before
over the week
the weekly test
Pass the recall
test and promptly
forget the words
Received traditions of language assessment
The ‘assessment mindset’ in teacher practice:
“Although this student still has problems with grammar, the
ideas are there. He is working through the choices the
community has about their need for a better water supply
system. Hmm, this is difficult. I just wish his grammar
errors weren’t so bad then I could give him an ‘A’.”
Mohan example, TBLT 2007 Conference
Received traditions of language assessment
What is the appropriate proficiency level to adopt
as a student learning outcome for the 2-year
language requirement?
INTERMEDIATE - LOW ? ? ?
Current trends in assessment
Student Learning
Outcomes Assessment
Outcomes embody the essential purpose of an educational program:
developments in knowledge, skills, dispositions of learners
Requires rethinking of educational programs as something more than
the delivery of experiences or the exposure of learners to information
Calls for articulation of curriculum and instruction in support of
targeted outcomes, demands integrated thinking
 Provides a clear statement of educational program value; answers
the question “How do you know?” with evidence of educational
effectiveness
Current trends in assessment
Student Learning
Outcomes Assessment
Process
•State outcomes
•Identify indicators
•Measure them
•Analyze the results
•Then what?
•(Let the chair/dean do it)
Current trends in assessment
Large public institution
Accreditation pressures
to assess learning
We have to assess our “liberal
studies” core, ASAP!
Huge expenditure, $$$,
time, effort
How about an electronic
portfolio?
Thousands of student
portfolios created
BUT…
Current trends in assessment
Electronic Portfolio
…faculty didn’t understand it
…students thought:
1. *!%#$!@*
2. waste of time
…administrators wanted to
do something with it but
weren’t sure exactly what
NEVER GOT USED
Current trends in assessment
Barrington (2003), on assessment in the liberal arts:
“To design and administer (intellectually honest) assessment plans that will
measure such capabilities with a dozen or more standardized ‘learning
objectives’ is next to impossible” leading to “pestilent repercussions” for
the truly valued learning objectives that constitute the liberal arts, in that it
“discourages teaching such skills because they are difficult to measure”.
Perception problem
Traditions, trends, and the status quo
Traditions and trends
Status quo
1. Focus on doing…
1. Not useful—not used!
2. Based on commercial
testing, proficiency mvt.
2. Potential negative washback,
reductionism, waste
3. Reactive v. proactive
praxis
3. Not relevant to curriculum &
instruction, program values
4. Driven by external
impetuses
4. Not ours—mandated or
purchased off the shelf
5. Technocratic
measurement emphasis
5. Not perceived as worth the
effort by faculty
6. Little scholarly
investment
6. Minimal professional
development
Re-envisioning assessment
in language education
Resolving terminological confusion
Measurement is the consistent
elicitation of quantifiable indicators of
well-defined constructs via tests or
related observation procedures; it
emphasizes efficiency, objectivity, and
technical aspects of construct validity.
Norris (2006) MLJ Perspectives
Resolving terminological confusion
Assessment is the systematic gathering
of information about student learning
in support of teaching and learning…It
may be direct or indirect, objective or
subjective, formal or informal,
standardized or idiosyncratic…It
provides locally useful information on
learners and learning to those
individuals responsible for doing
something about it.
Norris (2006) MLJ Perspectives
Resolving terminological confusion
Evaluation is the gathering of
information about any of the variety of
elements that constitute educational
programs, for a variety of purposes
that include primarily understanding,
demonstrating, improving, and judging
program value; evaluation brings
evidence to bear on the problems of
programs, but the nature of that
evidence is not restricted to one
particular methodology.
Norris (2006) MLJ Perspectives
The nature of useful evaluations
Evaluation gets used when…
(a) intended users of evaluation participate;
(b) evaluation is pursued as a process, not an end-game;
(c) sufficient time and resources are allocated;
(d) evaluation produces interesting, credible, relevant findings;
(e) findings are reported in a timely fashion;
(f) Interpretations and recommendations are contextualized.
Evaluative assessment
Light (2001) on outcomes assessment:
“…a process of evaluating and improving current
programs, encouraging innovations, and then
evaluating each innovation’s effectiveness. The
key step is systematic gathering of information
for sustained improvement. And always with an
eye toward helping faculty or students work
more effectively.”
Richard Light (2001, p. 224)
An evaluative approach to assessment
Accountability
Raising
Awareness
Motivating
Learners
PURPOSES
Program
Development
Improving
teaching
Justifying
$ requests
Illumination
Revising
curriculum
Improving
learning
Diagnosis
Certification
Articulation
An evaluative approach to assessment
Surveys
Tests
Interviews
Focus groups
Peer
Assessment
Performance
assessment
Journals
METHODS
Portfolios
Language
Profiles
Meetings
Quizzes
Observations
Standardized
assessment
Self
assessment
An evaluative approach to assessment
Approaching assessment:
•The most important consideration is supposed to be the
use for which it is intended (cf. Bachman & Palmer,
1996; Brown, 2005)
•BUT, in test development, what is the first question that
gets asked? What are you going to measure?
“Decide on purpose of assessment:
•What abilities are you assessing?
--What is your construct or model of these abilities?”
Coombe, Folse, Hubley (2007, p. 4))
An evaluative approach to assessment
WHO?
Test
Users
WHAT?
Test
Information
INTENDED
ASSESSMENT
USE
IMPACT?
Test
Consequences
WHY?
Test
Purposes
An evaluative approach to assessment
Primary
Intended Users
Negotiate &
specify:
Assessment
Advisors
• priority uses
• methods
Stakeholders &
Audiences
• analyses
•reporting
• timelines
ENABLING USE
EMPOWERING USERS
An evaluative approach to assessment
Products:
(a)public documents on the exact roles to be played by assessments in
the FL program and the different forms that those assessments take
(b) program policies on assessment practice at the individual, classroom,
and program levels
(c) assessment methods that lead to local actions
(d) evaluative/evidentiary justification for assessments and their uses
(e) Identification of other factors in need of evaluation
Why bother?
 Align assessment with
curriculum, instruction, program
values: foster ownership
 Raise awareness & buy-in
among students, faculty, others
about roles for assessment and
 Increase the likelihood that
evaluation
data will be used
 Decrease frequency and
number of (useless)
assessments and evaluations
 Support improvements in
student learning
Why bother?
Received view of assessment:
What do we test? How do we
test? What tests are available?
Evaluative Vision:
What questions do we have about
our learners, courses, and programs?
How do we gather data appropriate
to answering those questions? In
what ways can we utilize those data
to resolve the challenges we face?
What does it look like?
Using assessment
to understand and improve language programs
Example: Needs assessment for curricular revision
Context
Problem
Role of assessment
• Japanese at
University of Hawaii
• Perception of curricular • Identify learners’
‘antiquation’
perceived L2 needs
• 2-yr. requirement
(mostly non-majors)
• Relevance to learners’
wants and needs?
• Identify teachers’
perceptions of same
• @1000 students per
year
• Shifting standards for
college FL learning
• Identify gaps between
the two
• @50 instructors
• New expectation to
pursue performance
outcomes
• Prioritize outcomes
expectations for 2-yr.
program
• How to evolve? What
values to prioritize?
• Provide basis for
revision of materials
• Diversity of
students, diverse
uses of Japanese
Example: Needs assessment for curricular revision
Instruments and Procedures
Questionnaires
Teachers (N = 46)
L2 Use Domains
Academic
Students (N = 688)
Social
Occupational
Likert scale ratings
of priorities
Tourism
+
Open-ended
questions
Example: Needs assessment for curricular revision
Identified Learner Needs
Teachers + Learners
Prioritize
Learners Prioritize,
Teachers do not
In Hawaii…
In Japan…
In Japan…
•Academic routines
•Academic routines
•Socializing
•Socializing
•Medical
interactions
•Hosting guests
•Surviving as a
tourist
•Working retail
•Interacting
w/tourists
•Using computers
•Participating in
complex
discussions
•Residing longterm (housing,
etc.)
•Working abroad
•Formal
ceremonies
Example: Needs assessment for curricular revision
Other Curricular Implications
• Instruction moves too rapidly
• Overemphasis on Kanji
memorization
• Not enough attention to culture
• Not enough focus on situationspecific speaking tasks,
performance practice
Example: Needs assessment for curricular revision
Using Assessment for Curricular Change
Related Changes
• Basis for review of texts, materials
& pedagogy development
• Dean commits to Departmentlevel Assessment Specialist
• Revised 2-yr expectations emphasize
speaking, culture, performance
• Evaluation utilized as campuswide example of ‘closing the
loop’ back to curriculum
• Attention to Japanese use in Hawaii
& long-term residence abroad (both
authentic to learners)
• Development of performance-based
outcomes assessments
• Revised balance/pace of instruction
• Increased enrollments,
improved course evals
• Principal Investigator
tenured…
See Iwai et al (1999)
Example: Realizing the curriculum via assessment
Context
• German at
Georgetown Univ.
• New curriculum:
Developing Multiple
Literacies
• Fully integrated
Language & Content
instruction
• Task- & genre-based
Problem
• New curriculum, new
instruction, old
assessment
• Accuracy of curricular
learning trajectories?
• Students meeting
expected outcomes?
• Feasible expectations
for intensive and non?
• Advanced L2 literacy • Relevance of external
target
assessments?
Role of assessment
• Operationalize
curricular expectations
• Provide evidence
regarding learner
development &
outcomes
• Enable between-level,
and intensive/non
comparisons
• Fit seamlessly into
teaching and learning
processes!!!
Example: Realizing the curriculum via assessment
Prototypical performance
writing tasks
T
E
A
C
H
E
R
S
Level performance profiles
Deliberation
Development
Task assignment sheets
Analysis
Revision
Sample student performances
L
E
V
E
L
G
R
O
U
P
S
Example: Realizing the curriculum via assessment
Consistent
assignment
framework
Task
Curricular
level
expectations
Content
Language
Assignment 1
Assignment 2
Semester…
Assignment 3
Assignment 4
Explicit
performance
criteria
Prototypical
Performance
Writing
Task
Example: Realizing the curriculum via assessment
“Assessment in this kind of a context is, I would
almost say probably an indispensable aspect in
order to clarify any number of things. Because it
is in the discourse about assessment and how we
would do that that our knowledge became
articulated or the holes in that knowledge
became clearer to ourselves. Or the cover-ups
that we had engaged in were no longer possible if
we wanted to be honest with ourselves about it.”
Example: Realizing the curriculum via assessment
Using Assessment for Curricular Change
• Forced the curriculum to become
real
• Close specification of L2 progress
within/across curricular levels
• Disambiguation of learning outcomes
in terms of task, content, language
• Curricular ‘map’ for use by teachers
and learners (what happens when?)
• Forged agreement between levels on
what can and cannot be expected
See Byrnes (2002)
Example: Assessing and improving effectiveness
Context
Problem
• LAUSD
• Does it help?
• Kindergarten/First,
ELLs with varying
proficiencies (41%)
• Does it get used?
• Supplemental
software introduced
to enhance reading
instruction
• Are conditions
appropriate for
benefiting from it?
• Do students and
teachers understand
• Wide-spread reading
how it is supposed to be
‘failure’
used?
• Should it be
maintained?
Role of assessment
• Evidence of reading
development
• Enable comparisons
between ages,
proficiencies,
treatment/control
• Capture the context of
implementation
• Provide thorough
interpretation of
processes + outcomes
Example: Assessing and improving effectiveness
Instruments and Procedures
200 classrooms
(100 treatment, 100 control)
Enable estimation of added
effect of new software
Standardized, mandated
reading assessments
Indicate English reading
abilities among children
Extensive classroom
observations
Teacher interviews
Provide for meaningful
interpretation of findings in
terms of classroom realities
Example: Assessing and improving effectiveness
Findings
•No differences in reading ability
between treatment and control
classes or proficiency groupings
•BUT, Low implementation in
classes and by individuals
•Lots of reasons: Lack of time,
competition with standard
curriculum, computer
malfunction, lack of
understanding by learners and
teachers, lack of ‘on-task’
engagement
Uses
•Provide for professional
development in software
•Articulate/align with standard
curriculum
•Individualize for students with
greatest need
•Shift school schedule to allow for
implementation to occur
•Retain the software but continue
assessing effectiveness with
students
See Llosa & Slayton (forthcoming)
Facing change:
Learning to see assessment as a useful process
•There is more to useful assessment than just good
measurement; measurement informs construct
interpretations…assessment enables informed change
•Contextualization in classes, curricula, institutions is essential
for assessment to be interpretable and meaningful
•Intentionality for specific purposes, by specific users, with
specific consequences is essential for assessment to be used
•These processes should be built in from the beginning of
assessment design; specification of intended use is one
heuristic for doing so
•Educational assessment is only useful insofar as it does good;
hold assessments accountable to that principle
A few resources…
University of Hawai‘i at Mānoa,
Second Language Studies
http://www.nflrc.hawaii.edu/evaluation
Thank you!