Transcript Document

Embedding evaluation
PREEL
15 May 2007
Caroline Daly
• What is really useful evaluation in courses
designed to embed technologies to support
learning?
• What methods are appropriate to support
really useful evaluation?
• What is the role of course tutors,
administrators, learning technologists in
evaluation?
• What are the practical priorities?
• So what is embedded evaluation and do we
want it?
The research project
Context
Design
Mixed-mode degree (MTeach)
Longitudinal – 7 months
Participants were a tutor group of
12 teachers beginning the course
Quantitative data – pre-course
questionnaire
Text-based online discussions
- compulsory
- asynchronous
Qualitative data – learner narratives
- narrative interviews
- focus group discussions
- online focus groups
- online commentaries
Only 2 participants were
experienced e-learners
Analytical approach based on
systematic reading and open-coding of
range of narratives
Main aims: to identify e-learners’ experiences in the mixed-mode context
to trial an embedded approach to evaluation
to analyse the implications for e-learning course development
• What is evaluation for?
• Who is evaluation for?
“Professional evaluation is defined as the
systematic determination of the quality or
value of something” (Scriven, 1991).
It usually serves two main purposes linked to
concrete outcomes:
– To find areas for improvement
– And/or to generate assessment of the overall
quality or value of something (usually involving
reporting/decision-making)
Thoughts?????
Purposes of evaluation – a ‘classic’ view
(Chelimsky, 1997)
accountability
development
knowledge
e-VALU-ation
“…involves not only collecting descriptive
information …but also using something called
“values” to (a) determine what information
should be collected and (b) draw explicitly
evaluative inferences from the data…
…research can tell us “what’s so” but only
evaluation can tell us “so what”
Davidson, E. (2005)
As evaluators, we need “certain
enough” answers to evaluation
questions…a “close enough
approximation to the truth”
(Davidson, E. 2005)
A way of
capturing the
learner voice
as a prime
evaluation tool
A way of better
understanding the
realities of ‘being
an e-learner’ on
our courses
Why narrative evaluation methods?
To provide
ongoing
information about
the course while it
is happening
To impact on
the learners’
abilities to adapt
to new learning
contexts
Narrative ‘postmodern’
interviews (Gubrium,
2003) where participants
construct an account of
what is happening to
them (can be telephone)
Online and face to
face focus group
discussions, timed
at key moments in
the module and
addressing issues
the students raise
What narrative evaluation
methods did we use?
Online ‘thinking aloud’ in
response to images of e-learning
(‘mediating objects’ Eraut, 2000),
when they had ‘settled in’ to the
module
Issues for online evaluation commentary
1. A sense of community?
Now that you have spent your first month as an online student on
the MTeach – what has it been like so far? Some researchers say
that it is very difficult to establish a ‘sense of community’ at a
distance. Others say that it is certainly possible to form a learning
community online, and there are several theories about what this
might look like. What are your first thoughts?
2. Writing online/talking in a seminar
Only two of you had prior experience of learning online before
joining the MTeach, so this has been a new way of communicating
to learn for everybody else in the group. These discussions form a
significant alternative to talking about issues at the face to face
days or in traditional seminars. What has it been like to ‘discuss’
by using writing to communicate with each other like this?
Group narratives
Whereas I felt more
comfortable because I wanted
to get the non-verbal feedback.
Because when you’re talking to
somebody you’re getting that
“..intra-group discussion
about
feedback all the time, aren’t
a to
particular story you?
means
I’m used towhat
talking
About whether they’re
people when
I haven’t
a
fuels
the learning
cycle”
agreeing. You’re nodding at me
clue who they
are, so…I
(Greenhalgh,
2006) now because you understand
find it pretty easy to talk
what I’m saying, whereas when
to people online.
I was
The process
is essentially
ofthat online you
you’reone
doing
more nervous
coming
don’t know whether they’re
active
engagement in the
today, having not met
understanding
evaluation of experiences,
and is you at all.
anyone than I was
Sally
driven
by
the
learners’
priorities
chatting online and
requires the sharing of
discussingand
issues.
sustained reflective narration.
Katy
We thought…
They said…
So we…
The course info was
clear that it is a mixedmode course, using
online discussion as the
core learning mode
They thought it was
…ftf…live chat…
…video-conferencing
…one to one tutoring
via email
Reviewed the course
literature and webpage
to make it even more
explicit. Use the entry
viva to clarify basics
Teachers would learn
the principles of socioconstructivist ways of
working without too
much difficulty
It is hard to trust others
at a distance. There is a
lot at stake & they had
expected to be ‘told’ the
‘answers’ by experts
Spend longer on ftf
days discussing the
course pedagogy.
Devote more time to
social bonding
The sample postings
suggested helpful ways
to contribute online
and overcome anxiety
about how to join in
The templates have a
prescriptive effect and
make them feel they
have to write in a
particular style
Reviewed the sample
postings & used ones
from ex-students to
encourage more natural
& varied writing styles
We thought…
There was an initial
stage of anxiety and
adjustment that went as
soon as they became
established as students
They said…
The process takes much
longer & they disguise
anxieties by ‘secretly’
re-writing draft postings
- time-consuming!
So we…
Devote more time ftf
discussing their online
work. They select
postings to discuss - the
thinking is not finished
Photos of group members They really liked being
in an online gallery
able to ‘see’ their online
would help to establish
peers
identities and a sense of
belonging to a group
Predicted not everyone
would post a photo. Took
photos on the induction
evening to guarantee
100% photo presence
They would be willing
to tell us about their
experiences as elearners
Built in an ongoing
evaluation forum with
key prompts from the
tutor
They were extremely
willing to tell us more
candidly than we
anticipated
Implications
There are benefits for learners of participating in narrative
evaluation activities, by which they address explicitly their (new)
experiences of learning with technologies.
There are benefits for practitioners in gaining knowledge of
unanticipated learner experiences, though these may not always
lead to simple modifications of course design or learner tasks.
There is an argument for course design to invest learner time in
developing the abilities of participants to evaluate how they are
learning. There is growing research evidence that this has a
positive impact on what they learn.
There is a need for meaningful evaluation activities which also
support learning. Engaging with evaluation becomes a different
type of experience for both tutors and students.
Planning for really useful evaluation
• What do you see as the purposes of evaluation on
your course?
• Who will be involved in evaluation activities?
• How will evaluation be conducted:
–
–
–
–
–
–
Who is responsible for the evaluation activities?
What is their frequency?
What type?
How accessible are the evaluation activities?
When will evaluation activities begin and end?
What will be done with the evaluation data?
• What will be the relationship between evaluation,
curriculum and learning experiences?
Bibliography
• Chelimsky, E. (1997) ‘Thoughts for a new evaluation
society’ Evaluation 3(1) 97-119.
• Davidson, E. (2005) Evaluation Methodology Basics
London: Sage.
• Eraut, M. (2000) ‘Non-formal learning, implicit learning
and tacit knowledge in professional work’ in (ed.) Coffield,
F. The necessity of informal learning Bristol: Policy Press
ESRC Learning Society Programme 12-31
• Greenhalgh, T. (2006) What seems to be the trouble?
Stories in illness and healthcare (Oxford: Radcliff
Publishing).
• Gubrium, J. (2003) Postmodern Interviewing London:
Sage.
• Scriven, M. (1991) Evaluation Thesaurus 4th Edition,
London: Sage.