Student Feedback on, and Evaluation of, their Learning Experience Michael Prosser Institute for Teaching and Learning The University of Sydney

INTRODUCTION

Some distinctions in the way we talk about and describe students learning experiences in higher education A. Consumer and client centred views of student learning experience - ratings of student satisfaction with teaching and services B. Learner centred views of student learning experience - indicators of student perceptions and experiences relate to learning approaches and outcomes In this presentation I wish to focus on the experience of students as learners – learner centred views 2

A. Design of learning experiences: focus on the way student experiences are designed by the teacher B. Student experiences of the design of student learning experiences: - focus on the way the student experiences the design 3

It is not (just) the way we design our courses and programs in higher education that relates to the quality of student learning but how our students experience and understand those designs

.

Finding out, understanding and responding to the way students experience our courses and programs – the student experience of our courses and programs Working towards aligning the way our student experience our courses and programs with the way we design those courses and programs 4

OVERVIEW OF THE STUDENT LEARNING PERSPECTIVE

Figure 1: Model of Student Learning CHARACTERISTICS OF THE STUDENT (e.g. previous experiences, current understanding) STUDENTS' PERCEPTIONS OF CONTEXT (e.g. good teaching, clear goals) COURSE AND DEPARTMENTAL LEARNING CONTEXT (e.g. course design, teaching methods, assessment) STUDENTS' APPROACHES TO LEARNING (how they learn e.g. surface/deep) STUDENTS' LEARNING OUTCOMES (what they learn quantity/quality) 5

Student Approaches to Learning A key variation represented by: Surface Approach:

Intention to reproduce

• rote memorise information needed for assessment • • • failure to distinguish principles from examples treat tasks as external impositions focus on discrete elements without integration Deep Approach:

Intention to understand

• meaningfully memorise information for later use • relate new ideas to previous knowledge • • relate concepts to everyday experiences relate evidence to conclusions 6

Students’ Experiences of The Learning Context Student experiences of: • Quality of teaching – including quality of feedback • Clearness of course goals and standards • Workload - too high to understand • Assessment - reproduction and not understanding are key aspects of student experiences found to relate to how they approaches their studies and to their learning outcomes Entwistle and Ramsden, 1983 7

EXAMPLE FROM MEDICINE Quantitative Survey Study of over 8000 students in first year subjects around Australia, including 618 first year medical students in three different first year courses Amongst the data collected were students responses to: • Ramsden's Course Experience Questionnaire • Biggs’ Study Process Questionnaire 1994-1996:

Australian Research Council

; Academic Departments and the Quality of Teaching and Learning; Paul Ramsden, Griffith University, Elaine Martin, RMIT, Michael Prosser, La Trobe University, Keith Trigwell, UTS 8

Surface Approach

Approaches to Study 32. Although I generally remember facts and details, I find it difficult to fit them together into an overall picture

Deep Approach

28. I try to relate ideas in this subject to those in other subjects, wherever possible Biggs’ Study Process Questionnaire 9

Student Experiences Of Learning Environment

Good Teaching

15. The staff made a real effort to understand difficulties students might be having with their work.

Clear Goals and Standards

1. It was always easy to know the standard of work expected 6. I usually had a clear idea of where I was going and what was expected of me in this subject.

Appropriate Workload

25. The sheer volume of work in this subject meant that it couldn't all be thoroughly comprehended (-).

Appropriate Assessment

8. To do well in this subject, all you really need is a good memory (-).

10 Paul Ramsden’s Course Experience Questionnaire

Analyses showed that within each course: 1. A deep approach is associated with experiences that the teaching is good and the goals and standards are clear 2. A surface approach is associated with experiences that the workload is too high and assessment tests reproduction That is, variation in students’ experiences of the learning environment

within subjects

is associated with the approaches to study

within subjects

– within subject variation in experience is not measurement 11 error.

Factor Analysis of Experiences of Learning Context and Approach to Study ________________________________________________ Scale Subject A Subject B Subject C Factors Factors Factors ________________________________________________

Experiences

Good teaching ________ 1 2 .84

________ 1 2 .77

________ 1 2 .79

.70

.69

Clear Goals & Stan .68

App. Workload App. Assessment

Approaches

Deep Approach .75

-.73

-.70

.84

-.68

-.60

.80

-.69

-.60

Surface Approach .62

.64

.72

EXAMPLE FROM MATHEMATICS Qualitative Survey First year university mathematics students, large research intensive university, analyses of short open ended written statements by students identified:

Key differences in the experiences of Mathematics: Fragmented Experience

A. Maths as numbers, rules and formulae B. Maths as numbers etc with applications to problems

Cohesive Experience

C. Maths as a way of thinking D. Maths as a way of thinking for complex problem solving E. Maths provides insights for understanding the world 13

Experience of Maths A: Numbers Quote

Maths is the study of numbers and the application of various methods of changing numbers

C: Logical system

Maths is the study of logic. Numbers and symbols used to study life in a systematic perspective and requires the mind to think in a logical and precise way

E: Complex / World

Techniques for thinking about observable, physical phenomena in a quantitative way and also for thinking more abstractly with little or no relation to the directly observable universe

14

Approach to Study of Maths Quote A: Learning by rote memorisation, with an intention to reproduce (surface) E: Learning with an intention of gaining a coherent understanding of the theory and looking for situations where the theory will apply (deep)

I liked calculus because I could remember formulas which is how I used to study. I would rote learn all the formulas and summarise all my theoretical notes Read the relevant theory and try to get on the same “wavelength” as the person who actually discovered it. Before I attempt any problems I try to think where you can use the concept: i.e. what the concept was invented for. Then I attempt problems (on my own)

15

Relationship between Experience of Mathematics and Approach to Studying Mathematics ______________________________________________ Experience Approach Total _______________ Surface (A & B) Deep (C, D & E) ______________________________________________ Fragmented (A & B) 179 (

91%)

17 (

9%)

196 Cohesive (C, D, E) 4 (

10%)

36

(90%)

40 Total 183 53 236 ______________________________________________ Chi-square=126, p<.001; Phi=.57; ES=2.1

Crawford, K., Gordon, S., Nicholas, J. and Prosser, M. (1998) Qualitatively different experiences of learning mathematics at university.

Learning and Instruction

, 8, 455-468.

16

Defining Items from the Experiences of Mathematics Questionnaire Scale Item Fragmented Experience Cohesive Experience Item 9. Mathematics is figuring out problems involving numbers 17. Mathematics is a logical system which helps explain the things around us 17

Cluster Analysis of Experiences and Understandings _____________________________________________________________ Variables Cluster 1 (n=147) Cluster 2 (n=127) _____________________________________________________________

Prior Experiences and Understandings

Fragmented experience of Maths Cohesive experience of Maths 0.31

-0.39

-0.46

0.45

Prior Surface Orientation Deep Orientation 0.34

-0.56

-0.44

0.59

Academic Ranking (TER)

Experiences and Approaches

Good Teaching Clear Goals and Standards Appropriate Workload Appropriate Assessment Surface Approach Deep Approach

Post Experiences and Understanding

Fragmented experience of Maths Cohesive experience of Maths Achievement (Final Mark in Maths) -0.29

-0.37

-0.23

-0.30

-0.12

0.44

-0.54

0.34

-0.27

-0.34

0.35

0.37

0.24

0.32

0.12

-0.45

0.57

-0.41

0.27

0.40

18 ___________________________________________________

Cluster Analysis of Variables (medium to large differences) _____________________________________________________________ Variables Cluster 1 (n=147) Cluster 2 (n=127) _____________________________________________________________

Prior Experiences and Understandings

Fragmented experience of Maths Cohesive experience of Maths High Low Low High Prior Surface Orientation Deep Orientation High Low Low High

Experiences and Approaches

Good Teaching Clear Goals and Standards Appropriate Workload Appropriate Assessment Surface Approach Deep Approach

Post Experiences and Understanding

Fragmented experience of Maths Cohesive experience of Maths Achievement (Final Mark in Maths) Low Low High Low High Low Low High Small difference High Small difference Low High Low High High ___________________________________________________

EXAMPLE FROM OXFORD UNIVERSITY (Honours: n=428, Approach: n=2320) _________________________________________________ Experience Outcome Approach Honours _________________ Clear Goals and (1,2a,2b) (+ve) Surface Deep ________________________________________________ Good teaching Large Low High Standards Appropriate Assessment Appropriate Workload Large Medium Medium Low Low Low High High High _________________________________________________ http://www.learning.ox.ac.uk/iaul/pdf/OLCPFinal.pdf

20

In summary: Relationships between students experiences of their course, their approaches to study and their learning outcomes View of student learning experiences: •

learner centred (learning)

cf consumer / client centred (satisfaction) •

student experience of learning designs

cf design of learning experiences 21

EXAMPLE FROM THE UNIVERSITY OF SYDNEY

Graduates:

Graduate Course Experience Questionnaire -

Program

population survey of all coursework graduates Postgraduate Research Experience Questionnaire -

Program

population survey of all research graduates 22

Currently enrolled students:

Student Course Experience Questionnaire –

whole year

stratified random sample of all enrolled coursework students Student Research Experience Questionnaire –

whole year

population survey of all enrolled research students Unit of Study Evaluation -

course

compulsory triennial survey of all units of study 23

Supplemented and supported by: Focus group discussions to further explore issues identified in surveys Academic Board Reviews of Faculties interview groups of postgraduate and undergraduate students to better understand their experiences Performance based funding model for teaching to reward Faculties enhancing the student experience Faculty Teaching and Learning Plans with a focus on Student Experience 24

IMPLICATIONS FOR EVALUATING STUDENT LEARNING EXPERIENCES

1. Try to understand how and why student experience their courses and programs they way they do – quantitative survey, open-ended responses and follow up interview and focus groups 2. In responding , try to respond to why, not how, they experience their courses and programs the way they do – eg workload, assessment 3. Implement institutional policies, procedures and strategies that recognise and reward teachers, schools and departments and faculties efforts to respond to and enhance students learning experiences 25