Transcript The impact of formative and summative coursework

The impact of case studies
formatively and summatively
assessed on students’ examination
performance.
Geeta Hitch
(Senior Lecturer, Dept of Pharmacy)
Janet Webber
(Senior Lecturer, Dept of Physiotherapy/Principal Lecturer LTI)
Background
• PMM module /2nd year MPharm degree
• Previously taught as lectures only
• Concern- ‘decontextualised’ as well as there
was a divide between experience of learning
and that as a practitioner
Aim
• To investigate the the impact of case studies
formatively and summatively assessed on
students’ examination performance
Seven Principles of Good Practice in ‘Higher’
Education (Chickering & Gamson, 1987).
• Encourages contact between students and
lecturers
• Develops reciprocity and cooperation among
students
• Encourages active learning
• Gives prompt feedback
• Emphasises time on task
• Communicates high expectations
• Respects diverse talents and ways of learning
Method
• Use of ‘hybrid’ PBL in PMM module
– Bridge the divide between theory and practice; reallife situation problems.
• Incorporation of constructive alignment (Biggs,
2000) in the use of problem solving exercise
– exam question format is case study style
– Inclusion summative assessment of presentation &
poster
– PMM module 50% exam (25% on case study style
question); 50% CW (of which 5% was allocated to this
assignment)
Method: encompassed Seven Principles of Good Practice in ‘Higher’ Education
(Chickering & Gamson, 1987).
•
•
•
•
•
•
Students placed in groups of 6;
communicated with peers online via
Studynet discussion site
‘Hybrid’ PBL- case study with structured
questions
Tutor acted as facilitator
Students given a timeline
Constant feedback was provided online
Had an oral presentation /poster (10 mins)
– Followed by Q&A
– Assessed by panel of 3 lecturers
– CW mark of 5% in summatively
assessed cohort
•
•
•
– Encourages contact between
students and lecturers
– Develops reciprocity and cooperation
among students
– Encourages active learning
– Respects diverse talents and ways of
learning
Emphasises time on task
Gives prompt feedback
– Encourages contact between
students and lecturers
– Develops reciprocity and cooperation
among students
– Encourages active learning
– Communicates high expectations
Respects diverse talents and ways of
learning
Exam question:
25% of the marks- had a choice of doing one of the 2 case study style based
questions
Method
• Cohorts in which study was carried out:
Academic year
Lectures only
(no hybrid PBLs
so used as
Control Group)
2007-08 (n=95)
×
2008-09
(n=123)
Lectures plus
Hybrid PBLs
(formatively
assessed)
×
2009-10
(n=132)
Feedback
Questionnaire
Lectures plus
Hybrid PBLs
(summatively
assessed)
×
×
×
Data analysis
• Overall examination performance between all 3 cohorts
was compared- Statistical Package for Social Science (SPSS
for MS windows version 16).
– The Students t test was used to compare the probability level
set at 5% i.e. P < 0.05. If the calculated p- value was below the
threshold chosen for statistical significance of 0.05, then the
null hypothesis was rejected in favour of the alternative
hypothesis.
– Therefore any P values less than 0.05 were regarded as
significant.
• Null hypothesis: assessed case studies should make NO
impact on students’ final exam performance
• Alternative hypothesis: assessed case studies should make
an impact on students’ final exam performance
Data analysis
• Questionnaires
– A qualitative analysis of the student feedback
questionnaire was also carried out from the
academic cohorts of 2008-09 (‘formative cohort’)
and 2009-10 (‘summative cohort’)
– Group dynamics and the impact of inclusion of
summative and formative assessment was
examined as to whether it was a driving force in
participation of group work.
Results:
Percentage of student cohorts in 2007-08 (Control cohort), 2008-09 (formative cohort)
and 2009-10 (summative cohort) showing marks range scored (%) in the final PMM
Examination question.
% of students 2007-08 cohort
(Lectures only; No case studies)
30
25
% of students 2008 -09 cohort
lectures and (Formative
assessment of case studies)
15
10
% of students 2009-10 cohort
lectures and (Summative
assessment of case studies)
5
Range of marks (%)
90 to 100
80 to under 90
70 to under 80
60 to under 70
50 to under 60
40to under 50
30 to under 40
20 to under 30
10 to under 20
0
0 to under 10
% students
20
Mean marks:
2007-08: 38.48 %
2008-09: 38.41 %
2009-10: 46.04 %
Results
Percentage of student cohorts in 2007-08, 2008-09 and 2009-10 scoring more
than 40% in final PMM Examination question
80
% of students 2007-08 cohort
(Lectures only; No case studies)
70
60
% students
50
% of students 2008 -09 cohort
lectures and (Formative assessment
of case studies)
40
30
20
% of students 2009-10 cohort
lectures and (Summative assessment
of case studies)
total % of students scoring from
40-100
90 to 100
80 to under 90
70 to under 80
60 to under 70
50 to under 60
0
40 to under 50
10
More than 65% of students have scored greater than 40% in their final PMM
exam question in 2009-10 cohort (summative vs lectures only) p≤0.0001;
Results
• No significant difference in exam performance for
students when formatively assessed case studies
were used (2008-09 cohort) and when lectures
only were used (2007-08 cohort) to deliver the
curriculum (student T test; p≤0.97).
• Significantly better overall performance in the
relevant exam question when the case studies
were summatively assessed in comparison to
formative assessment (student T test; p≤0.0005);
2007-08 (control cohort)
Comparison of final exam performance in the same cohort between 2
different pharmacy modules (PMM- pharmaceutical Microbiology &
Manufacture and MPP3- Medicines & Professional Practice Level 3)
45
2007-08 MPP3 exam
40
35
2007-08 PMM exam
% students
30
25
Mean mark:
PMM: 38.48 %
MPP3: 55.38 %
20
15
10
5
0
0 to
11 to
20 to
30 to
40 to
50 to
60 to
70 to
80 to 90 to 100
under 10 under 20 under 30 under 40 under 50 under 60 under 70 under 80 under 90
Marks range %
2008-09 (formative cohort)
Comparison of final exam performance in the same cohort between 2
different pharmacy modules (PMM- pharmaceutical Microbiology &
Manufacture and MPP2- Medicines & Professional Practice Level 2)
2008-09 MPP2 exam
40
35
2008-09 PMM exam
% students
30
25
Mean mark:
PMM: 38.41 %
MPP2: 51.5 %
20
15
10
5
0
0 to
11 to
20 to
30 to
40 to
50 to
60 to
70 to
80 to
90 to
under 10 under 20 under 30 under 40 under 50 under 60 under 70 under 80 under 90 100
Marks range %
2009-10 (summative cohort)
Comparison of final exam performance in the same cohort between 2
different pharmacy modules (PMM- pharmaceutical Microbiology &
Manufacture and MPP1- Medicines & Professional Practice Level 1)
45
2009-10 MPP1 exam
40
2009-10 PMM exam
35
% students
30
Mean Mark:
PMM: 46.05 %
MPP1: 51.2 %
25
20
15
10
5
0
0 to
11 to
20 to
30 to
40 to
50 to
60 to
70 to
80 to
90 to
under 10 under 20 under 30 under 40 under 50 under 60 under 70 under 80 under 90 100
Marks range %
Cohorts general performance in other
examinations
• All 3 cohorts appear to be performing at the
same level in 2 sets of exams results
compared.
• This bears a significance in terms of inclusion
of case studies because:
– The overall performance appears to be similar in
2007-08 and 2008-09 cohorts.
– The overall exam performance in case study style
based question is performed significantly better in
the 2009-10 summative assessed cohort.
Questionnaires
• Response rate:
– 65% from 2008-09 (n=123) (Formative cohort)
– 73% from 2009-10 (n=132) (Summative cohort)
• Enjoyed taking part in group work:
– 75% of formatively assessed cohort - yes
– 90% of summative assessed cohort- yes
• 58% of students participated in equal input into group work
when case studies were formatively assessed in comparison to
83% when summatively assessed.
• When students were asked if they participated more when
summative assessment took place,
– 76% replied yes and 66% of these stated that they were driven by a genuine
desire to study by participating in the case study.
– The other 34% were just performing a task as it was summative and exam
marks mattered more to them.
Questionnaires
– 56% of the formatively assessed cohort felt that
they learnt better in group work than working
alone
– 64% of the summative assessed cohort felt that
they learnt more in group work than by working
alone.
Questionnaires
• Formative assessments were not viewed as important to
the students in achieving their learning outcomes for a
variety of reasons. In the case of this cohort (2008-09),
several reasons were cited by students in the feedback
questionnaire:
– “Too many summative assignments to hand in and therefore
formative assignments are my last priority”
– “Many students tend to rely on others to do the group work and
then when they are told to get on with it, the work they hand in
is not reliable”
– “Some students were clearly not willing to prepare or
participate in the presentations as there was no marks awarded
for this work and so the rest of the group ended up doing their
part of the work”
Discussion
• Gibbs, (1999) - “Assessment is perceived by
students as the curriculum and as such the
power of assessment needs to be used
strategically to help students learn”
• Biggs (2002) identifies this- states that
“students will only learn what they think will
be assessed on as opposed to what is required
of them to learn in the curriculum”.
Discussion
• General observations: inclusion of case studies
does improve overall performance- students
engage more effectively when learning is
contextualised and is of relevance to practice
• Students also perform better when marks are
allocated to assignments adding to overall
module mark
• If marks are not allocated towards assignments
and towards module marks in general, students
can be reluctant to participate in CW elements of
the curriculum.
Finally we conclude:
“That by inclusion of the 7 principles of good
practice in undergraduate education, we have
shown in this small study that how we
practice the teaching is as much as important
as to what the students learn.”
Seven Principles of Good Practice in ‘Higher’
Education (Chickering & Gamson, 1987).
• Encourages contact between students and
lecturers
• Develops reciprocity and cooperation among
students
• Encourages active learning
• Gives prompt feedback
• Emphasises time on task
• Communicates high expectations
• Respects diverse talents and ways of learning
References
1. Chickering, A. W. & Gamson, Z. F. (1987). ‘Seven
principles for good practice in undergraduate
education.’ American Association for Higher Education
Bulletin, 39(7), 3–7.
2. Gibbs, G (1999) Using assessment strategically to
change the way students learn, in: S. Brown & A.
Glasner (Eds) Assessment Matters in Higher Education
(Buckingham, SRHE & Open University Press).
3. Biggs, J. (2002) Aligning the curriculum to promote
good learning, paper presented at the Constructive
Alignment in Action: Imaginative Curriculum
Symposium, LTSN Generic Centre, November 2002.
Available online at: www.ltsn.ac.uk
Thank you
Any questions?