Transcript Slide 1

St Andrews Learning & Teaching:
Innovation, Review, Enhancement
Encouraging excellence and innovation in
learning and teaching by providing support and
guidance for students and staff.
SALTIRE
St Andrews Learning & Teaching:
Innovation, Review, Enhancement
Dr Colin Mason,
Director of Learning and Teaching Development
01334 462588
[email protected]
www.st-andrews.ac.uk/saltire
All exams should be abolished
I agree, why doesn’t everybody?
So, what do exams assess?
What do consumers (exam-takers) think?
If not exams, then what else?
A personal story
The date- Monday 25th May 2004. I wake, again - the
fourth time since heading for bed at midnight previously.
It is only 2-00am. Still 7 hours to go before .. Damn .. the
start of my finals – 5 three hour exams in a week, one
per day.
The bed is drenched. I escape the swamp and look in the
mirror as I reach the bathroom, not a pretty sight! Its
been this way throughout the last three weeks. Am I
surprised? No! It’s been this way throughout my whole
education – exams, they terrify me. It’s irrational, I know,
I am an A grade student – my results prove it, but this
happens to me every time. Why?
A personal story
It’s 8.-55am I am ushered in to a large unfriendly room with the
rest of my fellow students. The threats in notices pinned
everywhere, “leave all bags here”, no forbidden materials,
and so on, trigger a guilt-ridden response - my stomach sinks
to the floor of my abdomen and I gasp for air. I wobble,
unsteadily to a vacant, uncomfortably hard, plywood-seated
chair and a desk with the inevitable right corner leg shorter
than the rest. It tilts first one way as rest my hands on the
face-down exam question paper. I note the text in bold – yet
another warning! “Do not turn over until instructed to do so
by the invigilator”. I cringe again, more stomach cramps! Did
someone see me touch the paper, do they think I am trying to
get an early start? As I withdraw my hands rapidly, the desk
tilts back again, sending an echoing rattle around the
cavernous hall, then pivots back and forth – mirroring my own
wobbling gait earlier. Why do I always get ‘the’ desk with a
problem?
A personal story
It’s 9-00am – I turn over the paper, but only after I observe
that all my colleagues have now done so. In the past,
this is the moment I have found brings some relief, a
chance to simply write – but not this time. I scan the 6
questions, I have to choose 3. All the questions are
somehow familiar to me, but I am paralysed. I cross off
one as I am not sure that I quite grasp the meaning of
the question on a topic I know well. I repeat this draining
procedure, crossing out questions 2, 3, 4, 5 and now 6. I
turn the paper over, in a futile attempt to find further
questions – there are none, but I knew that, didn’t I? My
vision begins to blur and then I feel it – warmth
spreading rapidly from my groin, and then a trickle – I
can’t stay here, I must get out, get out of this oppressive,
now very cold room. I leave, I don’t return again – for
this exam or any of my other 4 scheduled for the rest of
the week. I have failed – or have exams failed me?
Why might colleagues disagree with me?
• Academic staff represent a highly
selected sub-set of the population
– Successful exam takers
– Traditional views about exams representing a
‘gold standard’
– Self interest
• Marking loads
• Engagement in alternative approaches
• Incentives (and disincentives)
Do your qualifications look like this?
Degrees:
Assessment Strategy:
BSc
Exams, Lab Reports, Group Project
MSc
Dissertation, Coursework, Exams
PGCE
Coursework, Tutor observation, Portfolio
PhD
Supervisor feedback; peer-reviewed
articles; conference oral and poster
presentations; oral thesis defence
What does an unseen
examination really assess?
• how competent you are?
• how well you work
independently under
pressure?
• how you study and work
under tight time
schedules?
• how well you’re be able to
predict what questions are
going to be asked in an
examination?
• how good you are at
writing examination
•
•
•
•
•
how much you know?
how much you don’t know?
how fast you can write?
how good your memory is?
how much work you did the
night before?
• how well you keep your
cool?
• how well you can read the
questions?
• how well you read your own
(or even somebody else’s!)
answers
What do exam takers think?
• Pilot study: Postgraduate students St
Andrews
– E-mailed survey (1 Quantitative question and
4 qualitative questions plus comments)
– Re-emailed (slightly) modified survey
– Low response (but interesting!)
If not exams, what else?
• It’s not about exams (or not!)
• It is about curriculum design:
RELIABILITY
VALIDITY
Reproducibility:
Within marker; Between
markers; Yr to Yr
Appropriate: Meets
learning outcomes
(Alignment, Biggs,
1999)
Objectivity
Accurate: Stable and sensitive
marking instruments
Complexity: confounds
easy attempts to define
criteria eg critical
thinking
Complex outcomes of learning
•
‘Complex’ outcomes including higher order
academic abilities and ‘soft skills’ are rarely
and inconsistently defined
–
–
Advanced ‘skilful practice’ is acquired slowly
(years)
Precision & reliability of such assessment is often
only attained at the expense of validity)
Shift to
FORMATIVE assessment
So, what methods could replace
exams?
• ASSHE inventory, 1996
• LTSN, 2001
• QAA (Scotland) Enhancement themes
(2003-04)
ASSHE Project
Dai Hounsell et al, 1996
•
•
•
•
•
Assessment
Strategies
Scottish
Higher
Education
Methods of Assessment?
Individual
Written
Verbal
• EXAMINATIONS
Group
• EXAMINATION/COURSEWORK
• COURSEWORK
Who assesses whom?
Tutor
Tutor
assessment
Evaluation
Peer
assessment
Self
assessment
Student
Student
assessment
Who and What?
Individual
Written
Verbal
• EXAMINATIONS
Group
• EXAMINATION/COURSEWORK
• COURSEWORK
SHEFC/QAA
Enhancement themes 20032003-04
– Assessment
– Responding to student needs
2004-05
– Flexible Delivery
– Employability
Assessment
http://www.qaa.ac.uk/scottishenhancement/events/default.htm
Review of degree classification:
– Assessment seminar: Considering the Honours
degree classification system - 11 May 2004
8 Workshops on Good practice (Jan
– June 2004)
– Key note (international) presentations
– Case Studies (Scottish)
– Discussion groups
Book
Assessment Workshops
1 - Streamlining assessment - How to make assessment more
effective and more efficient - 13 January 2004
2 - Using assessment to motivate learning - 5 February 2004
3 - Constructive alignment of learning outcomes to
assessment methods - 27 February 2004
4 - Developing a variety of assessment methods, including self
and peer assessment - 19 March 2004
5 - Assessing online - 16 April 2004
6 - Issues of validity, reliability and fairness - 7 May 2004
7 - Improving feedback to students (link between formative and
summative assessment) - 4 June 2004
8 - Assessment of Personal Transferable Skills - 29 June 2004
Some thoughts about Change
Charles Handy, “Beyond Certainty”, 1996
‘we must not let our past, however glorious, get in the
way of our future’.
‘ … initiatives can only be judged after the event.
Organisations prefer to control and judge things before
they happen. It’s safer, but slower, more expensive
and assumes those high up in the organisation know
better.’
‘When no mistakes are tolerated, no initiatives will be
risked. Forgiveness, provided one learns, is a
necessary part of initiative-taking. It can be hard to
practice’
And a final thought about
implementation
• Review promotion and reward systems
• Value ‘teaching’
• Especially, and explicitly modify criteria to
shift emphasis from delivery (eg lectures)
to effecting high quality learning, by
rewarding assessment activities:
– quantity and quality of feedback and
constructive criticism (Feedforward, P
Knight, 2004)