Transcript TEC 2013 Hyderabad The role of programme evaluation in curriculum development OR ‘How are we doing?’ Richard Kiely University of St Mark & St John, UK My.

TEC 2013 Hyderabad
The role of programme evaluation in
curriculum development
OR
‘How are we doing?’
Richard Kiely
University of St Mark & St John, UK
My life in programme evaluation
1.
Medium of instruction evaluation (Pupil)
2.
Native-speaker teacher evaluation (Teacher)
3.
Evaluation for development consultancy (Consultant)
4.
International CLIL project (Evaluator)
5.
Teacher education programme impact study
(Researcher)
Some principles
1.
Evaluation cannot just test the theory
2.
Evaluation has to understand learning processes as well
as outcomes
3.
Evaluation has to engage with all stakeholders
4.
Evaluation has to facilitate action
5.
Evaluation should make sense of the programme for
everyone.
This presentation
• Overview of language programme evaluation and its
potential in curriculum and professional development.
• Evaluation purposes, designs and methods
• Identities and roles of programme stakeholders
• Using the processes and outcomes of evaluation for
improvement of programmes, and
• Management of change issues.
Overview
• Programme evaluation is ‘a set of strategies to
document and understand the programme. It
involves research activity (conventional studies or
action research by which teachers learn about and
transform aspects of their practice) and assessment
data (conventional measures of outcomes). In
addition to these, evaluation has to engage with the
social, cultural and historical identity of the
programme, as a product of the institution, as a
phase in the biographies of participants, and as a
context of personal investments of individual
stakeholders’.
Kiely 2009: 114
Programme Evaluation
TYPE 1
Experimental/
comparative
Probability
Policy-oriented
Programme theory
(explicit)
Product-focus
External
TYPE 2
Case study
Possibility
Practice-oriented
Programme theory
(implicit)
Process-focus
Internal
Programme Evaluation
TYPE 1
Experimental/
comparative
Probability
Policy-oriented
Programme theory
(explicit)
Product-focus
External
TYPE 2
Case study
Discuss:
Any of
these
Possibility
features
new to Practice-oriented
you?
Programme theory
(implicit)
Process-focus
Internal
Evaluation purposes, designs and
methods
• Purposes: accountability,
development and quality assurance
• Designs: the links between data,
theory and action
• Methods: ways of getting data.
Evaluation designs – Type 1
Template 1
Template 2
Measurement of
outcomes (language
tests; teacher
performance; teacher
qualifications;
Surveys of attitudes;
preferences; aspirations;
wants; needs
Evaluation designs – Type 2
Template 1
Template 2
Template 3
Quality
Assurance
Programme
support
Teacher
research
Evaluation designs – Type 2
Template 1
Template 2
Template 3
Quality
Assurance
Programme
support
Teacher
research
Buzz group discussion:
What kind of activities do
you think would contribute
to these designs?
Evaluation designs – Type 2
Template 1
Quality
Assurance
Template 2
Programme
support
Template 3
Teacher research
End of course
questionnaires
Mentoring
Action research
Surveys
Focus Groups
Audits
Peer observation
Reading circles
Assessment
workshops
Meetings
Reflective Practice
Exploratory practice
Study for research
degrees
Assessment data
Evaluation designs
Template 1
Quality
Assurance
Template 2
Programme
support
Template 3
Teacher research
Design issues
How do we find the right amount of activity?
How do we get a focus on learning?
How do we add value to learning opportunities?
Identities and roles of programme
stakeholders
Teachers
Identities and roles of programme
stakeholders
Students
Teachers
Identities and roles of programme
stakeholders
Students
Teachers
Teacher educators
Identities and roles of programme
stakeholders
Leaders and
managers
Students
Teachers
Teacher educators
Identities and roles of programme
stakeholders
Leaders and
managers
Students
Teachers
Remote stakeholders
Teacher educators
Identities and roles
Immediate stakeholders
Students
Customers
Learners
Participants
Practitioners
Teachers
Transformers
Transmitters
Advisors
Assessors
Remote
stakeholders
Managers
Sponsors
Parents
Employers
Student role - learning
• Using the processes and outcomes of
evaluation for improvement of
programmes
WORKSHOP
Student role - learning
• Using the processes and outcomes of
evaluation for improvement of
programmes
WORKSHOP
Buzz group
discussion:
How can teachers
workshop
evaluation
process with
students?
Student role - learning
• Processes
• Awareness raising
• Focus groups
• Structured discussions
• Reflections as part of assessment
Student role - learning
• Engaging learners in communicative encounters,
especially if their aim is to explore emotional content
and experiences, can become too bound up in itself
unless this activity also reaches an evaluation stage.
Trying to understand what has happened while
undertaking a particular task, why it was suggested
by the teacher, and contributing actively to the
evaluation of learning arrangements, sequences,
resources and input materials by means of reflection
and meta-communicative discourse – all these are
considered indispensable learner activities in ELT.
Legutke and Thomas (1991:65)
Student role - learning
• Engaging learners in communicative encounters,
especially if their aim is to explore emotional content
Autonomy can become too bound up in itself
and experiences,
unless this activity also reaches an evaluation stage.
Trying to understand what has happened while
undertaking a particular task, why it was suggested
by the teacher, and contributing actively to the
evaluation of learning arrangements, sequences,
resources and input materials by means of reflection
and meta-communicative discourse – all these are
considered indispensable learner activities in ELT.
Legutke and Thomas (1991:65)
Student role - learning
• Engaging learners in communicative encounters,
especially if their aim is to explore emotional content
Autonomy can become too bound up in itself
and experiences,
unless this activity also reaches an evaluation stage.
Trying to understand what has happened
Agencywhile
undertaking a particular task, why it was suggested
by the teacher, and contributing actively to the
evaluation of learning arrangements, sequences,
resources and input materials by means of reflection
and meta-communicative discourse – all these are
considered indispensable learner activities in ELT.
Legutke and Thomas (1991:65)
Student role - learning
• Engaging learners in communicative encounters,
especially if their aim is to explore emotional content
Autonomy can become too bound up in itself
and experiences,
unless this activity also reaches an evaluation stage.
Trying to understand what has happened
Agencywhile
undertaking a particular task, why it was suggested
by the teacher, and contributing actively to the
evaluation of learning arrangements, sequences,
resources and input materials by means of reflection
and meta-communicative
discourse – all these are
Motivation
considered indispensable learner activities in ELT.
Legutke and Thomas (1991:65)
Using the processes and outcomes of
evaluation for improvement of
programmes
Teachers in a community of practice
Using the processes and outcomes of
evaluation for improvement of
programmes
• Collaborative development (peer
observation; mentoring; coaching)
• Workshops on materials; ICT; assessment
formats; test data
Using the processes and outcomes of
evaluation for improvement of
programmes
• Collaborative development (peer
observation; mentoring; coaching)
• Workshops on materials; ICT; assessment
formats; test data
Buzz group discussion:
Do these happen in
your context? If not,
why not?
Teacher role – professional learning
• Sponsored professionalism: based on
qualification and recognition
• Independent professionalism: ‘a commitment
to careful and critical examination of the
assumptions and practices embedded in
sponsored professionalism’
(Leung 2009:53).
Teachers and Change
• Int: So after the evaluation […..] Will you do it
differently next time?
• Millie: I don’t think I’ll do it differently. […] we
have developed some good strategies and
some quite good materials, that will continue
to develop. So it is not a change in direction,
but perhaps going further in the same
direction.
Kiely 1998: 194
Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Resistance
Reflection
Innovation
CYCLE 1
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class



CYCLE 2



Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Resistance
Reflection
Innovation
CYCLE 1
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class

Anna interprets
this as selfish
thinking on the
part of students


CYCLE 2



Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Resistance

Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
Innovation

CYCLE 2



Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Resistance

Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
Innovation

CYCLE 2



Anna develops
a pedagogy
which focusses
on
comprehension
of ideas rather
than individual
words
Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Resistance

Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
Innovation

CYCLE 2
Anna gets
feedback
from students
suggesting
more
attention to
explaining
words in
class



Anna develops
a pedagogy
which focusses
on
comprehension
of ideas rather
than individual
words
Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Resistance

Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
Innovation

CYCLE 2
Anna gets
feedback
from students
suggesting
more
attention to
explaining
words in
class

Anna resists
suggestions that
her focus on
‘broad swathes
of meaning’
should change


Anna develops
a pedagogy
which focusses
on
comprehension
of ideas rather
than individual
words
Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Anna gets
feedback
from students
suggesting
more
attention to
explaining
words in
class
Resistance


Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
CYCLE 2
Anna resists
Anna:
suggestions that
‘This group say
her focus on
they want it, so
 I try to do it for
‘broad swathes
of meaning’
them’, and
should change
includes a short
activity in
Week 6
Innovation


Anna develops
a pedagogy
which focusses
on
comprehension
of ideas rather
than individual
words
Teachersand
andChange
Change (Kiely 2001:257)
Teachers
(Kiely 2001:257)
Feedback
Anna gets
feedback
from students
suggesting
less attention
to explaining
words in
class
Anna gets
feedback
from students
suggesting
more
attention to
explaining
words in
class
Resistance


Reflection
CYCLE 1
Anna interprets
Anna
this as selfish
rationalises that
thinking on the
this might not
 be the best way
part of students
of using
classroom time
CYCLE 2
Anna resists
Anna:
suggestions that
‘This group say
her focus on
they want it, so
 I try to do it for
‘broad swathes
of meaning’
them’, and
should change
includes a short
activity in
Week 6
Innovation


Anna develops
a pedagogy
which focusses
on
comprehension
of ideas rather
than individual
words
Anna spends
more time on
vocabulary in
Weeks 9 & 10,
and provides
vocabulary
tasks to texts in
Weeks 11 & 12
Teachers and Change
• Change is gradual, perhaps not always visible
to the naked eye;
• Change occurs at the teacher’s pace;
• Change is negotiated;
• Change happens.
Key guidelines from principles
1.
Evaluation cannot just test the theory
2.
Evaluation has to understand learning
processes as well as outcomes
3.
Evaluation has to engage with all stakeholders
4.
Evaluation has to facilitate action
5.
Evaluation should make sense of the
programme for everyone.
Use evaluation to ….
1.
…. explain what is going on
2.
… answer the ‘How are we doing?’ question
3.
… allow all voices to be heard
4.
… act for improvement
5.
… raise awareness and explain.
References
Kiely, R. (2012) Designing evaluation into change management processes. Overview chapter in
Tribble, C. (Ed) Managing Change in Language Education. London: The British Council pp 75-91
Kiely, R. & P. Rea-Dickins (2009) Evaluation and learning in language programmes. In Knapp, K.
and B. Seidlhofer with H. Widdowson (eds) Handbooks of Applied Linguistics: Volume 6:
Handbook of foreign language communication and learning. Amsterdam: Mouton de Gruyter.
Pp. 663-694
Kiely, R. (2011) Understanding CLIL as an innovation. Studies in Second Language Learning and
Teaching. Vol 1, No 1, pp 153-71.
http://ssllt.amu.edu.pl/images/stories/volume.1/SSLLT_11_153-171_Kiely.pdf
Kiely, R. (2009) Small answers to the big question: Learning from language programme
evaluation. Language Teaching Research Vol 13, No 1: 99-116
Kiely, R. (2006) Evaluation, innovation and ownership in language programs. Modern Language
Journal, Vol 90, No 3 pp: 597-602
Kiely, R. & P. Rea-Dickins (2005) Program Evaluation in Language Education. Basingstoke:
Palgrave Macmillan (Series editors Chris Candlin and David Hall) [Second edition – 2013 – in
preparation].
Legutke, M. & H. Thomas (1991) Process and Experience in the Language Classroom. Harlow:
Longman
Thank you
[email protected]