Transcript Contextualising Assessment: the lecturers’perspective

Assessment design, pedagogy
and practice: What do ‘new’
lecturers think?
Lin Norton, Bill Norton, Lee Shannon and
Frances Phillips
24 October 2009
ISSOTL 2009
1
Acknowledgements
Research has been funded by the Write Now CETL
The researchers include:
Ola Aiyegbayo (formerly of Liverpool Hope University)
James Elander (Derby University)
Katherine Harrington (London Metropolitan University)
Frances Phillips (Liverpool Hope University)
Peter Reddy (Aston University)
Lee Shannon (Liverpool Hope University)
24 October 2009
ISSOTL 2009
2
Background: Is there a problem with
assessment?
• Authentic assessment can be defined as assessment that is
pedagogically appropriate- it frames students’ views of HE, it has
a major influence on their learning and it directs their attention to
what is important (Boud & Falchikov, 2007)
But…
• In the USA, many institutions tend to adopt assessment
approaches more traditional than those recommended in the
literature (Peterson & Einarson, 2001)
• In the UK context, Rust (2007) asserts that many assessment
practices are poor and not underpinned by pedagogical literature.
• There seems to be a gap between SOTL and what it has to say
about assessment and actual assessment practice
24 October 2009
ISSOTL 2009
3
So what is currently happening in HE?
• Widely held view that assessment should be for rather than of
learning (Black & Wiliam, 1998; Assessment for Learning: AfLCETL
http://www.northumbria.ac.uk/cetl_afl/)
• But… is this what lecturers(faculty) think- particularly new
lecturers, and, if so, are they able to put their beliefs into practice?
• And… what part do taught programmes for lecturers have to play?
24 October 2009
ISSOTL 2009
4
What part does a university teaching
programme play?
•
An initial interview study was carried out with 10 new lecturers
taking a university teaching course (Norton, Aiyegbayo, Harrington
Elander & Reddy, 2010)
•
Our findings suggested that the course was generally perceived as
useful and influenced the way that the new lecturers thought about
assessment:
– ‘The assessment module [in the course] revealed to me that
assessment is not just about assessing how much people know
but should be used as a tool in learning. This was an eye
opener for me.’ (G)
24 October 2009
ISSOTL 2009
5
Interview study cont..
• When asked if they could change assessment practice easily, 6 said no
and 4 said yes but with reservations:
– Not really because it is set in stone in the module proposal. You
have to jump through many hoops if you are going to change the
assessment techniques.’ (A)
– ‘No! I get the impression that they are set in stone…. I think that
hurdles of going to various panels to have your module changed
puts people off…I get the impression from talking to colleagues that
the process is long-winded and bureaucratic.’ (C)
• We concluded that what new lecturers learn about assessment does
not always readily translate into practice. This may be caused by a
complex interaction of institutional, departmental and individual factors
(Becher & Trowler, 2001; Fanghanel, 2007), which we have called
‘constraints’
• We wanted to explore these constraints further, using a questionnaire
approach
24 October 2009
ISSOTL 2009
6
Current research study
• First of several large scale quantitative studies carried out by the
Write Now CETL research team investigating lecturers’ views on
assessment, marking & feedback.
• This study focuses on lecturers who were taking, or had recently
taken, a university teaching course and their views about
assessment design.
Aims
• To analyse whether there were any potential constraining factors
hindering lecturers from putting what they had learned about
assessment into practice.
• To develop the Assessment Design Inventory (ADI) as an instrument
24 October 2009
ISSOTL 2009
7
Development of the ADI
• The Assessment Design Inventory (ADI) comprises 40 items
developed from over 600 items constructed from 84 interviews with
lecturers (new and experienced) from 18 disciplines, in 5 institutions.
• An earlier version of the ADI consisting of 68 items was piloted on a
sample of 29 lecturers from one UK university
• The ADI was sent out as an online survey aimed at those lecturers
currently on HE teaching courses or who had recently (in the last 2
years) completed such a course.
• Access to respondents was by invitation via HE teaching course
leaders at all UK institutions (150).
• Around half the institutions agreed to circulate the invitation to their
course participants
24 October 2009
ISSOTL 2009
8
Participants
• 586 fully completed inventories ( 325 females, 256 males, & 5 not
specified )
• 80 discipline areas and 66 institutions from England, Ireland,
Scotland & Wales
• 276 were currently on a HE Teaching course; 308 had already
attained an HE Teaching qualification.
• University teaching experience ranged from less than a year to 26
years:
24 October 2009
Years
Under 2
2 to 5
6 to 10
11+
N
232
197
99
49
ISSOTL 2009
9
Assessment Design Inventory(ADI)
• Initial exploratory analysis suggests 3 underlying factors:
– Constraints e.g. ‘Changes to my assessment design are
hindered by external factors (e.g. cost, high student numbers,
time).’
– Desirable practice e.g. ‘Involving students in the assessment
design would encourage them to engage in the assessment
task’.
– Purpose e.g. ‘The main aim of assessment is to document
student achievement to employers.’
24 October 2009
ISSOTL 2009
10
Constraints
• Results will be presented for the 12 items on the ADI that specifically
asked about:
– the potential constraints on turning assessment design
philosophy into assessment practice
• We were particularly interested in looking for any effects on
responses relating to:
– participation on HE teaching courses
– length of experience in teaching in Higher Education
24 October 2009
ISSOTL 2009
11
The effect of the University teaching
programme on new lecturers’ beliefs
about assessment
• My experience on the University teaching programme has
changed my views on assessment practice.
74% agree 16% disagree 10% uncertain [Mean=3.80, SD=0.99]
No effect for teaching qualification or length of experience
24 October 2009
ISSOTL 2009
12
How do new lecturers feel about
innovating assessment? (1)
• New assessment methods are needed to improve current
practice.
69% agree 17% disagree 14% uncertain [Mean=3.67,SD=0.95]
Lecturers who had already attained a teaching qualification agreed
more strongly than those currently taking a teaching course
• There is little incentive for lecturers to innovate in their
assessment practice.
61% agree 32% disagree 7% uncertain [Mean=3.40,SD=1.18]
Lecturers with 11+ years experience were less likely than those with
less than 2 years experience to agree that there was little incentive to
innovate
24 October 2009
ISSOTL 2009
13
How do new lecturers feel about
innovating assessment? (2)
• Innovation in assessment is not well received by students.
31% agree 52% disagree 16% uncertain [Mean=2.76,SD=1.15]
Lecturers with 11+ years experience were more likely than those
with less than 2 years experience to disagree.
• I feel my ideas are valued when developing assessments.
65% agree 16% disagree 20% uncertain [Mean=3.53,SD=0.92]
Lecturers with 11+ years experience agreed more strongly than
those with less than 2 years experience.
• In order to introduce innovative assessment I would consider
avoiding official procedures.
27% agree 57% disagree 16% uncertain [Mean=2.64,SD=1.09]
No effect for teaching qualification or length of experience
24 October 2009
ISSOTL 2009
14
What do new lecturers feel are
external constraints?
•
Changes to my assessment design are sometimes hindered by
external factors (e.g. cost, high student numbers, time).
75% agree 20% disagree 4% uncertain [Mean=3.81,SD=1.11]
•
QAA requirements seldom allow leeway in assessment design.
29% agree 41% disagree 31% uncertain [Mean=2.91,SD=0.95]
•
If external constraints (e.g. QAA, subject benchmarks) were
removed, I would be more willing to change my assessment
practice.
35% agree 41% disagree 24% uncertain [Mean=2.98,SD=1.10]
No effect for teaching qualification or length of experience for these 3
items
24 October 2009
ISSOTL 2009
15
What do new lecturers think about
students when designing
assessment? (1)
• It is possible for students to ‘go through the motions’ to satisfy
assessment requirements without learning anything.
57% agree 39% disagree 5% uncertain [Mean=3.22,SD=1.25]
No effect for teaching qualification or length of experience
• It is a challenge to design assessments for students of differing
levels of commitment.
77% agree 20% disagree 4% uncertain [Mean=3.78,SD=1.09]
No effect for teaching qualification or length of experience
• I find it a challenge to design pedagogically sound assessment
because of student focus on grades.
57% agree 35% disagree 10% uncertain [Mean=3.27,SD=1.09]
Lecturers with 11+ years experience found this less of a challenge
than those with 5 years or less experience.
24 October 2009
ISSOTL 2009
16
What do new lecturers think about
students when designing
assessment? (2)
• I find it a challenge to design assignments for students of
differing levels of ability.
61% agree 35% disagree 5% uncertain [Mean=3.32,SD=1.09]
Lecturers with 11+ years experience found this less of a challenge
than those with less than 2 years experience.
24 October 2009
ISSOTL 2009
17
Conclusions
We are still at the early stages in our analysis but our interim
findings suggest that:
1. Lecturers (both inexperienced and experienced) taking a university
teaching course do think that it has changed their views on
assessment.
2. Over three quarters of our respondents thought that assessment
needs improving
3. The fact that there were some significant differences in responses
between the experienced lecturers and those with less experience
suggest that the former may be more likely to find their way around
the various constraints when designing their assessments
24 October 2009
ISSOTL 2009
18
Further research
• Further analysis of our findings from this study is being
carried out and will inform our other work on developing
a marking and feedback inventory (currently under
construction)
• The ADI itself is still under development and has been
made freely available as a research tool for all on the
Write Now website http://www.writenow.ac.uk
24 October 2009
ISSOTL 2009
19
•
•
•
•
•
•
•
•
References
Becher, T. & Trowler, P. (2001) Academic tribes and territories: Intellectual
Enquiry and the Culture of Disciplines 2nd edition Buckingham: Society for
Research in Higher Education and the Open University Press.
Black, P. and Wiliam, D. (1998) Assessment and Classroom Learning.
Assessment in Education 5 (1) 7-74
Boud, D & Falchikov,N. (2007) Rethinking assessment in higher education.
Learning for the longer term, Routledge
CETL: Assessment for learning (AfL)
http://www.northumbria.ac.uk/cetl_afl/whatis/
Fanghanel, J. (2007) ‘Investigating university lecturers’ pedagogical
constructs in the working context’. York: Higher Education Academy.
http://www.heacademy.ac.uk/assets/York/documents/ourwork/research/fangh
anel.pdf
Norton, L, Aiyegbayo, O. , Harrington, K. , Elander, J. and Reddy, P.
(2010) 'New lecturers' beliefs about learning, teaching and assessment in
higher education: the role of the PGCLTHE programme', Innovations in
Education and Teaching International, 47: 4, 345 — 356.
Peterson, M.W. & Einarson, M.K. (2001) What are colleges doing about
student assessment? Does it make a difference? The Journal of Higher
Education, 72,6, 629-669
Rust, C (2007) "Towards a scholarship of assessment" Assessment &
Evaluation in Higher Education 32, 2, 229-237
24 October 2009
ISSOTL 2009
20