Business School

Assessment as Learning Experience

'

Prof Margaret Price ASKe Centre for Excellence

1

Outline

 The problem with assessment  Where should we go from here?

 Clarity about purpose and assessment standards  Reliability and the nature of assessment standards  Engaging students in the assessment process.

 Assuring standards  Implications

Business School ASKe – Assessment Standards Knowledge exchange

2

We are not doing well.

 QAA reviews  National Student Satisfaction Survey  “the Achilles’ heel of quality” (Knight 2002a, p. 107)  Summative assessment practices “in disarray” (Knight 2002b, p. 275  “Broken” (Race 2003, p. 5) 

“There is considerable scope for professional development in the area of assessment”

(Yorke et al, 2000, p7)

Business School ASKe – Assessment Standards Knowledge exchange

3

Assessment

Business School ASKe – Assessment Standards Knowledge exchange

4

Business School ASKe – Assessment Standards Knowledge exchange

5

Where should we go from here?

 Weston Manor Group, November 07 (40+ National and International Experts in Assessment)  Two days of discussions

Outcome

:

Assessment Standards: A Manifesto for Change

 Six tenet manifesto for change to assessment practice related to standards

Business School ASKe – Assessment Standards Knowledge exchange

6

Assessment: a key driver of student learning

“Assessment is at the heart of the student experience” (Brown, S & Knight, P., 1994) “From our students’ point of view, assessment always defines the actual curriculum” (Ramsden, P.,1992) “Assessment defines what students regard as important, how they spend their time and how they come to see themselves as students and then as graduates.........If you want to change student learning then change the methods of assessment” (Brown, G et al, 1997)

Business School ASKe – Assessment Standards Knowledge exchange

7

Assessment standards in HE

“

The types of assessment we currently use do not promote conceptual understanding and do not encourage a deep approach to learning………Our means of assessing them seems to do little to encourage them to adopt anything other than a strategic or mechanical approach to their studies.”

(Newstead 2002, p3)

“Conventional assessment procedures are unable to do justice to the most important outcomes of any educational process worth the name”

(Raven 1991, p1)

Business School ASKe – Assessment Standards Knowledge exchange

8

Tenet 1

“The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment

for

learning rather than assessment

of

learning.”

Business School ASKe – Assessment Standards Knowledge exchange

9

The problems with reliability

‘This quest for reliability tends to skew assessment towards the assessment of simple and unambiguous achievements, and considerations of cost add to the skew away from judgements of complex learning’

(Knight 2002b p278) “…

summative judgement itself is the problem

” (Burgess, 2007, p. 8)

Many research findings indicate a declining use of deep and contextual approaches to study as students’ progress through their degree programmes

(Watkins & Hattie, 1985; Kember et al, 1997; Richardson, 2000; Zhang & Watkins, 2001)

Business School ASKe – Assessment Standards Knowledge exchange

10

“…students become more interested in the mark and less interested in the subject over the course of their studies.”

(Newstead 2002, p2)

“ Even when lecturers say that they want students to be creative and thoughtful, students often recognise that what is really necessary, or at least what is sufficient, is to memorise”

(Gibbs, 1992, p. 10)

Our current systems focused on marks and grades aren’t working

(Elander & Hardman, 2002; Laming, 1990; Yorke et al, 2002; Bridges et al, 2002; Newstead and Dennis, 1994)

Business School ASKe – Assessment Standards Knowledge exchange

11

Tenet 2

“When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programme outcomes.”

Business School ASKe – Assessment Standards Knowledge exchange

12

Reliability and Standards

 Defining standards  Clarity  Transparency  How do students come to understand standards?

Business School ASKe – Assessment Standards Knowledge exchange

13

O’Donovan, Price & Rust 2008

Active student engagement 3. The Social Constructivist Model 4. The ‘Cultivated’ Community of Practice Model

The Future

2. The ‘Dominant Logic’ Explicit Model

The Past

1. The Traditional Model – Passive student engagement Business School ASKe – Assessment Standards Knowledge exchange

14

O’Donovan, Price and Rust. 2008

Active student engagement 3. The Social Constructivist Model

.

4. The ‘Cultivated’ Community of Practice Model

The Future

2. The ‘Dominant Logic’ Explicit Model

The Past

1. The Traditional Model –

Standards absorbed over relatively longer times informally and serendipitously

Passive student engagement Business School ASKe – Assessment Standards Knowledge exchange

15

O’Donovan, Price and Rust. 2008

Active student engagement 3. The Social Constructivist Model 4. The ‘Cultivated’ Community of Practice Model

The Future The Past

2. The ‘Dominant Logic’ Explicit Model

Standards explicitly articulated (with limitations) and passively presented to students

1. The Traditional Model –

Standards absorbed over relatively longer times informally and serendipitously

Passive student engagement Business School ASKe – Assessment Standards Knowledge exchange

16

Limitations of explicit articulation

 Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004)  “

we can know more than we can tell

” (Polanyi, reprinted 1998, p.136).

 Verbal level descriptors are inevitably ‘fuzzy’ (Sadler 1987)  There is a cost (in terms of time and resources) to codifying knowledge which increases the more diverse an audience’s experience and language (Snowdon, 2002).

 Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991)

Business School ASKe – Assessment Standards Knowledge exchange

17

Nature of criteria

 An indispensable condition for improvement in student learning is that “ the student comes to hold a concept of quality roughly similar to that held by the teacher” (Sadler, 1989)  Regulative and logical criteria “standards can be defined in terms of well-defined outcomes ” (Sadler, 1987, p. 70)

Business School ASKe – Assessment Standards Knowledge exchange

18

Nature of Criteria (2)

“  Prescriptive and constitutive criteria refer to matters of degree It would be difficult or impossible to guess the educational level at which they are applicable… ” (Sadler, 1987, p. 70).

 Such types of criteria are often interdependent and can only be assessed using holistic/professional judgement (Sadler, 2008)  Such criteria are socially constructed requiring the sharing of tacit knowledge over time (O’Donovan et al, 2004; Rust et al, 2005)

Business School ASKe – Assessment Standards Knowledge exchange

19

Tenet 3

“ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment.”

Business School ASKe – Assessment Standards Knowledge exchange

20

O’Donovan, Price and Rust. 2008

Active student engagement 3. The Social Constructivist Model

Actively engaging students in formal processes to communicate tacit knowledge of standards

The Future

4. The ‘Cultivated’ Community of Practice Model

The Past

2. The ‘Dominant Logic’ Explicit Model

Standards explicitly articulated (with limitations) and passively presented to students

1. The Traditional Model

Standards absorbed over relatively longer times informally and serendipitously

Passive student engagement Business School ASKe – Assessment Standards Knowledge exchange

21

Understanding assessment standards

 ‘making sense of the world’ is a social and collaborative activity (Vygotsky, 1978).

 Passive receipt of feedback has little effect on future performance (Fritz, et al., 2000)  ASKe assessment intervention (Rust et al, 2003;1,2,3 leaflet)  Dialogue and participatory relationships are key elements of engaging students with assessment feedback (Price et al 2008)

Business School ASKe – Assessment Standards Knowledge exchange

22

What works?

 The most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997)  The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007)  “

participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice

” (Elwood & Klenowski, 2002, p. 246)

Business School ASKe – Assessment Standards Knowledge exchange

23

O’Donovan, Price and Rust. 2008

Active student engagement 3. The Social Constructivist Model

Actively engaging students in formal processes to communicate tacit knowledge of standards

The Future

4. The ‘Cultivated’ Community of Practice Model

Tacit standards communicated through participation in informal knowledge exchange networks ‘seeded’ by specific activities.

The Past

2. The ‘Dominant Logic’ Explicit Model

Standards explicitly articulated (with limitations) and passively presented to students

1. The Traditional Model

Tacit standards absorbed over relatively longer times informally and serendipitously

Passive student engagement Business School ASKe – Assessment Standards Knowledge exchange

24

Tenet 4

“

Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best.”

Business School ASKe – Assessment Standards Knowledge exchange

25



Coming to know

Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004)  The achievement of high-level learning requires integrated and coherent progression based on programme outcomes  Where there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007)  Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004)

Business School ASKe – Assessment Standards Knowledge exchange

26

Tenet 5

“ Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards and monitor and supervise their own learning.”

Business School ASKe – Assessment Standards Knowledge exchange

27

Community and professional judgement

 Changes in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement.

 There has been slow progress in the professionalisation of university teachers  There has been limited attention paid to professional assessment practice  Professional judgement must be within a framework of professionalism

Business School ASKe – Assessment Standards Knowledge exchange

28

Tenet 6

“ Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities.”

Business School ASKe – Assessment Standards Knowledge exchange

29

Key points

 The current dominant logic of the explicit approach to assessment standards is insufficient and was not defended by any of the 40 experts gathered at Weston Manor  The quest for reliability is getting in the way of learning  Our main concern should be enabling students to achieve high level, complex learning.

 Active involvement by staff and students in the learning community is essential to reach common understandings of assessment standards  To achieve change it is likely that a review and evaluation of the allocation of time and resources within HE will be needed.

Business School ASKe – Assessment Standards Knowledge exchange

30

What next?



If you would like to be a signatory to this manifesto please visit http://www.business.brookes.ac.uk/learningandteac hing/aske/ Business School ASKe – Assessment Standards Knowledge exchange

31

           

References

Astin, A. (1997)

What matters in college? Four critical years revisited

. San Francisco: Jossey Bass Bridges, P., Cooper, A., Evanson, P., Haines, C., Jenkins, D., Scurry, D., Woolf, H. and Yorke, M (2002), ‘Coursework marks high examination marks low: discuss’,

Assessment and Evaluation in Higher Education

, vol. 27, no. 1, pp. 35-48.

Brown, G., Bull, J. & Pendlebury, M. (1997)

Assessing student learning in higher education

. London: Routledge Brown, S & Knight, P. (1994)

Assessing Learners in Higher Education

. London: Kogan Page Burgess, R. (2007) Beyond the Honours Degree Classification: Burgess Group Final Report. Universities UK Elander, J. and Hardman, D. (2002), ‘An application of judgement analysis to examination marking in psychology’, British Journal of Psychology 93, pp. 303-328.

Elwood, J. & Klenowski, V. (2002) ‘Creating communities of shared practice: the challenges of assessment use in learning and teaching’, Assessment and Evaluation in Higher Education, 27, 243-256 Fritz, C.O., Morris, P.E., Bjork, R.A., Gelman, R. & Wickens, T.D. (2000) ‘When further learning fails: stability and change following repeated presentation of text’, British Journal of Psychology, 91, 493-511 Gibbs, G (2007) Departmental leadership of teaching Presented at Learning Institute, University of Oxford 8 Feb Gibbs, G. (1992) Improving the quality of student learning. Bristol: TES Gibbs, P., Angelides, P. and Michaelides, P. (2004) ‘Preliminary thoughts on a praxis of higher education teaching’, Teaching in Higher Education, 9, 183-194 Havnes, A. (2007) ‘What can feedback practices tell us about variation in grading across fields?’ Presented at the ASKe Seminar Series, Oxford BrookesUniversity, 19th September

Business School ASKe – Assessment Standards Knowledge exchange

32

            Kember, D., et al. (1997) Case studies in improving teaching and learning from the action learning project. Action learning project, Hong Kong Knight, P. T. (2002a) The Achilles’ heel of quality: the assessment of student learning, Quality in Higher Education, 8(1), 107 –115.

Knight, P. T. (2002b) ‘Summative assessment in higher education: practices in disarray’, Studies in Higher Education, 27(3), 275 –286 Knight, P and Yorke, M (2004) Learning, Curriculum and Employability in Higher Education. London: Routledge Laming, D (1990) The reliability of a certain University examination compared with the precision of absolute judgements, Quarterly Journal of Experimental Psychology Section A Human Experimental Psychology, Vol. 42 No.2, 239-254 Newstead, S. (2002) ‘Examining the examiners: why are we so bad at assessing students?’, Psychology Learning and Teaching, 2 (2), Newstead, S. E. and Dennis, I. (1994) ‘Examiners examined: the reality of exam marking in psychology’, The Psychologist, 7, 216-19 Nonaka, I.. (1991) ‘The knowledge-creating company’, The Harvard Business Review, November-December, 96-104 O’Donovan, B., Price, M. & Rust, C. (2004) ‘Know what I mean? Enhancing student understanding of assessment standards and criteria’, Teaching in Higher Education, 9, 325 335 O’Donovan, B.,Price, M & Rust, C. (2008) Developing student understanding of assessment standards: a nested hierarchy of approaches. Teaching in Higher Education13(2), 205-218 Polanyi, M. (1998) The tacit dimension, Reprinted in L. Prusak (Ed) Knowledge in Organizations (Boston, Butterworth Heineman).

Price, M., K. Handley, & B. O’Donovan. (2008). Feedback – all that effort but what is the effect?, Paper presented at the EARLI/Northumbria Assessment Conference. August 27-29 in Potsdam, Germany.

Business School ASKe – Assessment Standards Knowledge exchange

33

              Race, P. (2003) Why fix assessment? in: L. Cooke & P. Smith (Eds) Seminar: reflections on learning and teaching in higher education, High Wycombe: Buckinghamshire Chilterns University College.

Ramsden, P. (1992) Learning to teach in higher education. London: Routledge Raven, J. (1991) The tragic illusion: educational testing. New York: Trillium Press Rust, C, Price, M. & O’Donovan, B. (2003) Improving students’ learning by developing their understanding of assessment criteria and processes, Assessment and Evaluation. 28, 147 164.

Rust, C., O’Donovan, B. & Price, M. (2005) ‘A social constructivist assessment process model: how the research literature shows us this could be best practice’, Assessment and Evaluation in Higher Education, 30 (3), 231-240 Sadler, D. R. (1987) ‘Specifying and Promulgating Achievement Standards’, Oxford Review of Education, 13, 191 –209 Sadler, D. R. (1989) ‘Formative assessment and the design of instructional systems’, Instructional Science, 18, 119-144 Sadler, D. R. (2008) ‘Indeterminacy in the use of preset criteria for assessment and grading’, Assessment and Evaluation in Higher Education, April Snowdon, D (2002) Complex acts of knowing: paradox and descriptive self-awareness. Journal of Knowledge Management, 6(Special Edition), pp. 100-111.

Watkins, D., and Hattie, J. (1985) ‘A longitudinal study of the approaches to learning of Australian tertiary students’, Human Learning, 4, 127-41 Vygotsky, L. S. (1978) Mind in society: the development of higher psychological processes. MA: Harvard University Press Yorke, M., Bridges, P. & Woolf, H. (2000) Mark distributions and marking practices in UK higher education; some challenging issues, Active Learning in Higher Education, 1 (1), 7-27 Yorke, M. (2002) Subject Benchmarking and the assessment of student learning, Quality Assurance in Education, 10(3), 155-171.

Zhang, L. F. & Watkins, D. (2001) ‘Cognitive development and student approaches to learning: an investigation of Perry's theory with Chinese and US university students’, Higher Education, 41, 236-261

Business School ASKe – Assessment Standards Knowledge exchange

34