Transcript Slide 1

NOTE: This PowerPoint presentation has been
modified by removal of all high-resolution graphics, to
reduce the storage and downloading requirements
(sorry, no pretty photos!).
Evaluating Task-Based Language Programs
Colloquium – TBLT 2009
The colloquium
Why bother with TBLT program evaluation?
Three presentations + clarification questions (2:00-3:30):
Re-framing the evaluation of task-based language education
Evaluating a TBLT Spanish immersion program
Evaluation of TBLT in Flanders
Open audience-panel discussion (3:30-3:50)
John M. Norris
University of Hawai‘i at Mānoa
TBLT 2009
Please cite as:
Norris, J. M. (2009, September). Reframing the
evaluation of task-based language education. Paper
presented at the refereed colloquium “Evaluating
task-based language programs”, at the 3rd
International Conference on Task-Based Language
Teaching, Lancaster, UK (September 14, 2009).
Re-framing the evaluation of task-based language education
TBLL v. TBLT: Disconnects between inquiry and practice
What is task-based language
learning (TBLL)?
1. Societal need for change
in language education…
Value
Outcomes
Methods
2. Emerging notions of L2
acquisition…
Processes
Impediments
Indicators
What is task-based language
learning (TBLL)?
Practices
1. Societal need for change
in language education…
Value
Observations
Outcomes
Methods
2. Emerging notions of L2
acquisition…
Proposals
Discussions
Hypotheses
Processes
Impediments
Indicators
Findings
Opportunity for a
researched
language
pedagogy
What is task-based language
learning (TBLL)?
Pedagogic principles, such as…
Promote learning by doing, experiential learning
Use task as the unit of analysis for instruction & assessment
Provide rich L2 input
Elaborate (rather than simplify) L2 input
Respect learner-internal syllabuses
Enable inductive/chunk learning
Promote collaborative-cooperative interaction
Provide focus on form, negative feedback
(e.g., Doughty & Long, 2003; Ellis, 2003)
What is task-based language
teaching (TBLT)?
Rationales and Principles
Philosophy of Education
Cognitive Psychology
Sociocultural theory
SLA
Applied to
Task-Based
Language
Teaching
Learners
Needs
Curriculum
Instruction
Materials
Curriculum theory
Planning and policy
L2 Education Programs
Assessment
Inform???
Teacher development
What is the role of task-based
inquiry?
GENERALIZABLE
Generate
theory
Understand
what works,
where, when,
& why
SITUATED
Test
hypotheses
Task-Based
Language
Learning
Task-Based
Language
Teaching
Improve teaching
practice
DISCRETE
Discover robust, if
small, truths
Inform
curriculum,
course design
HOLISTIC
What is the role of task-based
inquiry?
GENERALIZABLE
Generate
theory
Understand
what works,
where, when,
& why
SITUATED
Test
hypotheses
Task-Based
Language
Learning
Task-Based
Language
Teaching
Improve teaching
practice
DISCRETE
Discover robust, if
small, truths
Inform
curriculum,
course design
HOLISTIC
What is the role of task-based
inquiry?
GENERALIZABLE
Generate
theory
Understand
what works,
where, when,
& why
SITUATED
Test
hypotheses
Task-Based
Language
Learning
Task-Based
Language
Teaching
Improve teaching
practice
DISCRETE
Discover robust, if
small, truths
Inform
curriculum,
course design
HOLISTIC
Challenges for task-based inquiry
1. The scope of task-based research does not match
the scope of our claims about (for or against) TBLT.
2. The focus of theoretical task-based research does not
relate to the situated realities of task-based teaching.
Needs
Goals, outcomes
Curriculum
Scope, sequence
Materials
Instruction
Teachers
TBLT
Education
Programs
Resources
Practices
History, training
Learners
Individual differences
Assessment
Intended uses, users
Framing TBLT inquiry through program evaluation
Inquiry through evaluation
Research emphasizes theoretical,
conclusion-oriented inquiry
Evaluation operationalizes decisionoriented inquiry
Cronbach & Suppes (1969)
Inquiry through evaluation
Evaluation is the gathering of information
about any of the variety of elements that
constitute educational programs, for a
variety of purposes that include primarily
understanding, demonstrating, improving,
and judging program value; evaluation
brings evidence to bear on the problems of
programs, but the nature of that evidence
is not restricted to one particular
methodology.
Norris (2006) MLJ Perspectives
Inquiry frame
and focus
Inquiry impetus
Inquiry question
prioritization
Inquiry through evaluation
Cronbach et al. (1980)
“The evaluator will be
wise not to declare
allegiance to either a
quantitative-scientificsummative methodology
or a qualitativenaturalistic-descriptive
methodology.” (p. 7)
Paradigms
Epistemology 1
Methodology 1
Epistemology 2
Methodology 2
Inquiry through evaluation
Cronbach et al. (1980)
“The evaluator will be
wise not to declare
allegiance to either a
quantitative-scientificsummative methodology
or a qualitativenaturalistic-descriptive
methodology.” (p. 7)
Paradigms
Epistemology 1
Methodology 1
Epistemology 2
Methodology 2
Inquiry through evaluation
Pragmatism
Cronbach et al. (1980)
“The evaluator will be
wise not to declare
allegiance to either a
quantitative-scientificsummative methodology
or a qualitativenaturalistic-descriptive
methodology.” (p. 7)
Who?
What?
Why?
Method 1
Method 4
When?
Method 2
Method 3
Method 5
Inquiry through evaluation
1. Participation – stakeholders, representatives, primary intended users
2. Prioritization – challenges, questions in immediate need of answers
3. Instrumentation – what data will answer the questions?
4. Collection – how can we get data in available time/resources?
5. Interpretation – what do findings mean in context?
6. Utilization – what decisions & actions are taken?
Participation
by
educatorsuses
is essential
A focus
on language
specific
intended
for
Language
educators
are ultimately
responsible
throughout
evaluation
ifiscontextual
relevance
findings
essential
from
the
forevaluation
what happens
in language
education.
is sought.
outset, if evaluation
is to make any difference.
Context: Program features
Learner needs, institutional resources, program goals
and outcomes, curriculum, materials, instruction,
assessment, teachers, teacher development,
learners, etc.
Context:
Intended uses
Evaluating
TBLT
Programs
Understand
Improve
in situ
Educate
Demonstrate worth
Hold accountable Questions + Methods
Empower
Values clarification
(Test theory)
Implementation
Process-product
Outcomes
Context:
Intended users
Teachers
Administrators
Curriculum writers
Learners
Parents/public
Funders
(Researchers)
Patton (1997)
Utilization-focused
evaluation
Learning from evaluation examples: the brief history of TBLT
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project –
Prabhu’s “Bangalore Project” (See Prabhu, 1987)
Context
English L2
Education in
Bangalore,
India;
Seeking
Improvement
via Innovation
Theory
L2 learning
by processing
meaning;
Unconscious
grammar
construction
by learners
Program
Project/taskbased work;
4 experimental
schools;
Implemented
1979-1984
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project –
Phase 1 (See Beretta & Davies, 1985)
Initial inquiry, final year of the project:
“To assess, through appropriate tests, whether there is any
demonstrable difference in terms of attainment in English between
classes of children who have been taught on the CT project and their
peers who have received normal instruction in the respective schools.”
Beretta & Davies (1985)
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project –
Phase 1 (See Beretta & Davies, 1985)
Purpose
Test theory
Demonstrate method
effectiveness
Findings
Claims???
Structures test:
Control > CTP
Task-based learners
achieved as much or
more than traditional
on all but the least
functional outcomes
What do we really know???
Methods
Contextual grammar:
Control = CTP
Quasi-experimentation
Dictation:
Control = CTP
Class/method
comparison
List/Read comp:
CTP > Control
Outcome achievement
assessments
Task-based test:
CTP > Control
Task-based
instruction is
successful
Warranted claims???
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project –
Phase 2 (See Beretta, 1986, 1990, 1992)
Purpose
Findings
Understand program
implementation
Lack of comparability
(intact classes, no
baseline data)
Illuminate relation
with apparent outcomes
Methods
Retrospective
interview protocols
Teacher level of
concern questionnaires
Document analysis
More qualified
teachers in CTP classes
Implementation of
CTP highly variable
(over time, between
classes, with structures)
More confident
teachers = better results
Learning from TBLT evaluation
Evaluating the Communicational Teaching Project –
Lessons Learned (See Beretta, 1992)
Theory testing, methods comparisons, “what works” claims are
rarely feasible in real educational programs
Apparent differences in learning achievement, behaviors,
etc. can only be explained by observation of multiple factors as
they are experienced in real program contexts
Even poorly executed evaluations (e.g., post-hoc) can shed
light on how programs function and help explain why learning
does or does not occur
Teachers (beliefs, training, commitment, time) play a key
role in implementing programs: what they actually do must
be understood
Learning from TBLT evaluation
Evaluating a university French curriculum – Student
perspectives (See Towell & Tomlinson, 1999)
Context
French FL
Education, UK
university,
Salford;
Restructuring
advanced
FL teaching
Theory
Input, text,
task;
Learning
through formfunction
mapping in
tasks
Program
Task-based
syllabus;
Multiple levels
at university;
Implemented
1988-96
Learning from TBLT evaluation
Evaluating a university French curriculum – Student
perspectives (See Towell & Tomlinson, 1999)
Multiple iterations of development, implementation, evaluation, revision:
“Curriculum design, evaluation, application and enhancement is a slow
process, and subject to a number of extraneous influences which make
it impossible to measure with totally scientific precision…use of
diaries and questionnaires on the first occasion enabled a number of
lessons to be learned and these helped considerably in creating a
second application where the testimony of the student population
through a detailed questionnaire shows the success of the operation.”
Towell & Tomlinson (1999)
Learning from TBLT evaluation
Program: Initial TBLT long
group projects
Program: Revised TBLT
staged, short projects
Methods:
Methods:
S
T
A
G
E
•Learner diaries
1
Findings:
•Learner surveys
•Assessments/exams
S
T
A
G
E
•Learner surveys
2
Findings:
•Focus groups
•Assessments/exams
•Projects too long (6 wks)
•Increased satisfaction
•Training in group work
•Higher learning of skills
•Staged task objectives
•Improved oral translation
•Gains in text/task learning
•Written translation?
•Developing accuracy?
•Developing accuracy?
Learning from TBLT evaluation
Evaluating a university French curriculum – Student
perspectives (See Towell & Tomlinson, 1999)
Student/learner perspective on teaching with tasks sheds
important light on the realities of implementation (how + how well)
Building evaluation activities into curricular delivery from
the outset (e.g., student diaries), enables longitudinal insights
about change, development, response to instruction
Triangulated learner feedback (diaries, self-assessments,
questionnaires, exams) can lead to effective improvements in
curriculum and task design, and in turn to higher evaluations
Learners can change how they learn—acculturating to
TBLT—especially when tasks, instructions, assessments are
intentionally designed and staged to do so
Learning from TBLT evaluation
Evaluating TBLT for EAP – Developmental evaluation in
Thailand (See McDonough & Chaikitmongkol, 2007)
Context
English FL
education,
Thai
university;
Improving EAP
instruction
Theory
Integrated-skills,
communication;
Life-long
learning;
Learner needs
+ interest
orientation
Program
Task-based
syllabus;
English
department;
Implemented
12 months
Learning from TBLT evaluation
Evaluating TBLT for EAP – Developmental evaluation in
Thailand (See McDonough & Chaikitmongkol, 2007)
Inquiry for developing and improving TBLT experiences:
“…relatively few empirical studies have documented how teachers and
learners react to entirely task-based courses, as opposed to the use of
individual tasks…The purposes of this case study were (a) to identify
teacher and learner reactions to the course and (b) to describe how
their concerns, if any, were addressed.”
McDonough & Chaikitmongkol (2007)
Learning from TBLT evaluation
Evaluating TBLT for EAP – Developmental evaluation in
Thailand (See McDonough & Chaikitmongkol, 2007)
Methods:
Findings:
Uses:
•Learner task
evaluations
(repeated)
•Increased learner
independence, language skills,
learning strategies
•Intro unit on language
learning
•Learning notebooks
•Decreased grammar obsession
•Class observations
•Non-specific real-world
relevance
•Teacher’s guide to
instruction + workshop
•Need time to adjust (T&L)
•Enhanced task
guidelines, built-in
feedback opportunities
•Teacher/student
interviews
•More support, guidance from
teachers
•Reduced number of
tasks
•Observer field
notes
•Too much to cover, disparate
materials
•Consolidated
materials
•Student course
evaluations
Learning from TBLT evaluation
Evaluating TBLT for EAP – Developmental evaluation in
Thailand (See McDonough & Chaikitmongkol, 2007)
Cycles of evaluation planned into TBLT innovation, and carried
throughout, can lead to increased likelihood of effectiveness
Systematic evaluation (a) from multiple stakeholder
perspectives and (b) focused on multiple program elements
(materials, preparedness, outcomes) enables balanced change
Teachers and learners both require support in
implementing TBLT, especially during early phases of
introducing task-based instruction
TBLT based on learner needs can work well in EAP
contexts, especially when evaluation is used to support ongoing effectiveness of delivery from the outset
Learning from TBLT evaluation
Evaluating TBLT teacher training – Cyclical evaluation in
Belgium (See Van den Branden, 2006)
Context
Dutch SL
Education in
Flanders;
Nationwide; K-16;
Ensuring
Educational
Access, Equity
Theory
Large-scale
Task-Based LT
Innovation;
Improving
Functional
DSL Abilities
Program
School-based
TeacherTraining
Programs;
Enabling
Change,
1994-2003
Learning from TBLT evaluation
Van den Branden (2006): “…the teacher tries to act as a true interactional partner, negotiating
meaning and content with the students, eliciting and encouraging their output, focusing on form
when appropriate and offering them a rich, relevant and communicative input” (p. 217).
Evaluation PROBLEMS
Teacher cognition
•What do they theorize about TBLT?
•How do they learn about TBLT?
Teacher action
•Are they willing to change with syllabus?
•How do they adopt/adapt TBLT in practice?
Teaching context
•What are the social constraints on T-Dev?
•How can T-Dev be optimized?
Evaluation USES
Understand teachers
Illuminate context
Improve T-dev program
Encourage teacher agency
Ensure teaching success
Enable TBLT learning
Demonstrate outcomes
Learning from TBLT evaluation
Program: Theoretical
inservice training
S Methods:
T •Teacher survey
A
G •Training observation
E
Findings:
1
•Transmission model
•Short term (3 hrs.)
•“Try that with my
students”…Postcoursal depression!
Program: TB training +
syllabus support
S
T
A
G
E
2
Methods:
•Teacher logs,
interviews, classroom
observations
Findings:
+awareness of TBLT
+student enthusiasm
?teacher adoption
-teacher control
-task complexity
groupwork
Program: Training +
coaching + agency
S
T
A
G
E
3
Methods:
•Coaching obs,
classroom obs, coach/
teacher interviews
Findings:
+conscious decisions
+TBLT adaptation
+self-evaluation
?teacher control
transfer
groupwork
Learning from TBLT evaluation
Program: Sustained implementation of TBLT
T-Dev with coaching, support
S
T
A
G
E
Methods:
4
+incorporation of TBLT correlated with
higher Dutch L2 proficiency outcomes
•Pre-post student learning outcomes, teacher
surveys, classroom observations
Findings:
+3-year gains in DSL higher in TBLT intensive
adopting schools
?mixed incorporation of TBLT across schools,
teachers
Learning from TBLT evaluation
Evaluating TBLT teacher training – Cyclical evaluation in
Belgium (See Van den Branden, 2006)
Long-term evaluation of TBLT sheds light on how ideas are
implemented, how participants change, and what support is needed
Multi-directional evaluation (political, social, school,
individual) increases our capacity to explain why task-based
ideas work or do not
Persistent follow-through on evaluation findings (use)
underlies effective innovation
Teachers can learn to engage with TBLT, but change takes
time, requires individualized support, and must be valued
Learning from TBLT evaluation
What have we learned? Reframing evaluation in TBLT
From summative to intentional
From assessment-driven to multi-methodological
From external to participatory
From method-testing to program-illuminating
From one-shot to longitudinal, cyclical
From theoretical conclusions to
educational decisions
Research, evaluation, and the future of task-based education
TBLL research
 Sociocultural, cognitive, and other theories provide useful
starting points for thinking about language teaching and
learning, and offer principles for building educational programs
 Task-based language learning research helps in that it raises
our awareness about particular factors that we should pay
attention to in the instructed L2 learning process
 Task-based language learning research cannot tell us much
about how or why language education programs work; findings
from TBLL research should not be interpreted as direct
implications for TBLT education
TBLT evaluation
Answers questions & informs decisions of local interest
Sheds light on how TBLT ideas work in practice
Intentional
Provides truths situated in rich contexts of programs
Evaluative
Relates outcomes to TBLT delivery and other factors
Inquiry
Focuses on scope that is meaningful to teachers, learners
Tests and informs innovation on the ground, in situ
Empowers participants to learn, and learn to change
Forces an honest accounting of TBLT
TBLT evaluation
Resources: It takes time and money to do evaluation
well and to sustain it within L2 educational programs.
Challenges
For TBLT
Evaluation
Training: Effective evaluation calls upon skills that may
not be easily available among personnel at hand.
Dissemination: There are few venues for publishing
evaluation reports, thereby limiting learning.
Actual uses: There are many possible uses/needs for
evaluation that we are not sufficiently attuned to, yet.
Cheers! (Mahalo!)
References
Beretta, A. (1986). Program-fair language teaching evaluation. TESOL Quarterly, 20, 431-445.
Beretta, A. (1990). Implementation of the Bangalore Project. Applied Linguistics, 11(4), 321-340.
Beretta, A. (1992). What can be learned from the Bangalore evaluation? In J. C. Alderson and A. Beretta (eds.),
Evaluating second language education (pp. 250-273). Cambridge: Cambridge University Press.
Beretta, A., & Davies, A. (1985). Evaluation of the Bangalore Project. ELT Journal, 39(2), 121-127.
Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation. San Francisco: Jossey-Bass.
Cronbach, L., & Suppes, (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York:
Macmillan, 1969.
Doughty, C., & Long, M. H. (2003). Optimal psycholinguistic environments for distance foreign language learning.
Language Learning & Technology, 7, 50-80.
Ellis, R. (2003). Task-based language learning and teaching. New York: Oxford University Press.
McDonough, K., & Chaikitmongkol, W. (2007). Teachers’ and learners’ reactions to a task-based EFL course in Thailand.
TESOL Quarterly, 41(1), 107-132.
Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern
Language Journal, 90(4), 590-597.
Norris, J. M. (2009). Task-based teaching and testing. In M. Long and C. Doughty (Eds.), Handbook of language teaching
(pp. 578-594). Cambridge: Blackwell.
Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.
Prabhu, N. S. (1987). Second language pedagogy. Oxford University Press, Oxford.
Towell, R., & Tomlinson, P. (1999). Language curriculum development research at university level. Language Teaching
Research, 3(1), 1-32.
Van den Branden, K. (2006). Training teachers: Task-based as well? In K. Van den Branden (ed.), Task-based language
teaching in practice (pp. 217-273). Cambridge: Cambridge University Press.
http://www2.hawaii.edu/~jnorris