Transcript Self and Peer Assessment Automatic & Online

Self, Peer and Tutor Assessment of Text
Online. Design, Delivery and Analysis
Dr Richard Parsons
Centre for Learning and Teaching
University of Dundee
[email protected]
Self, Peer and Tutor Assessment of Text
Online. Design, Delivery and Analysis
• Overall position of CAA at
University of Dundee
• Why self and peer
assessment?
• Development story
• Key features
– Academic author sets
exercise online
– Students use web, email
and text messaging
– Robust reliable, secure
– Checks for copying
• Demonstration – create
an exercise, live
• Using the system – key
points
– Good question design
– Involve the students
– Teach the students
• Student activities and
responses
• Marking moderation
and plagiarism
• The future
University of Dundee
Learning Technology Systems
• Blackboard VLE (5.5 Unix, upgrading to Enterprise 6.0 Unix)
• QuestionMark Perception 3.4, dual server system
– 1. Low stakes/formative server coupled to Blackboard (Oracle
Unix database)
– 2. High stakes/summative server runs stand alone (Oracle Unix
database)
• Provide full support for CAA including a “CAA Online” staff
education course, initial authoring service
• Have Policy & Procedure documents for CAA (based on BS 7988),
(also for VLE and eLearning strategies)
• Provide support for a range of additional services including online
evaluation forms, plagiarism detection service, online staff
eLearning tuition, publicise innovative solutions for supporting and
enhancing learning (e.g Self and Peer Assessment Program)
University of Dundee
MLE
P3
User exists 3
weeks before
appointment
Completed
daily
e-Directory
Create user
name and
email process
LDAP
authentication
Remove user
process
Student
Management
System
Student exists
Staff details also
stored in SMS
Module enrolments of
students
Module assessment
mark storage
LDAP client?
LDAP
Novell Fileservers
Student Desktop
VLE Integrated
Systems
GroupWise
Web Email
QMP formative
server CAA-01
Module
enrolment
Bb client
Mark return
Blackboard
VLE – ELS 6.0
Hardware 2 * 480s
Version 6.0.8
Users created from
SITS SMS system
Module enrolment
Staff assignments
from web system
LDAP client
Module assessment
mark storage
Consultancy visit 12
May – 16 May
Training teaching
staff
Staff Assignment
Record
Secure web
access
QMP Exam
server CAA-02
Special exam
enrolment
Mark return
Library
Catalogue
access
Resource
management
system
LDAP client?
Richard Parsons, CLT LEU
Why Self and Peer Assessment?
• Assessment is one of the keys to learning
• Being able to assess your own work is a vital skill
• Developing peer assessment skills is important for
future careers
• By carrying out assessments, students learn about the
assessment process
• Shifts tutor work from post to pre-assignment
• Web-based and innovative, so can interest the students
Development Story
• First a system to replace some lectures when I was
overseas
• First self-assessment only, but peer assessment seen as
important, results emailed and texted to students
• Developed into a system that allowed online authoring
• Added the ability to tutor mark as well as self & peer
assessment
• Made more robust based on usage
– Remove the back button
– Disable right click
– Provide more guidance to staff and student users
• Added a check for copying and plagiarism
• Tidied up code to W3C compliance
Online Self & Peer Assessment of Text
(Outline of the process)
Instructor designs
question, sets word limits
and details criteria and
scoring ranges
All files created,
URLs provided to
instructor
Question provided,
answer researched and
composed
1. Student submits their text answer (may
copy and paste)
2. Self assessment completed by student
using criteria offered
Peer 1 assessment
emailed
anonymously to
student author
Peer 1 assessment
emailed
anonymously to
student author
3. Student completes an assessment of
one of their peers
4. Student completes an assessment of the
second of their peers
5. Summary of marks presented.
Student thanked for completing
exercise, brief feedback completed
Tutor can mark a
selection of answers
and moderate and
collate results
Final marks are
checked and
transferred to register
Technical Details
HTML pages
for input
Author
completes
webform
Sets question &
marking criteria
Perl program
creates
And sets file
protections
All operates on a
Unix web server,
with hyperlinks as
required to
Blackboard
Perl files to
process
input
User
interaction
HTML pages
for output
Perl file to
process
output
Author
access to
results
Demonstration of the Program
Create a NEW exercise, live and interactive?
www.dundee.ac.uk/learning/ilt/
Using the System – Key Points
• Good question design
• Involve the students
• Teach the students
Student Instructions – Online Self and Peer Assessment Exercise
Week containing
Tuesday 11 March
Receive the first assignment,
question and instructions
Research the first topic, craft
your first text answer
Mark your first answer using the criteria
provided and mark two peer’s answers
Receive the second
assignment and instructions
Tutor
marking of
exercise 1
Marks sent to
student author
Week containing
Tuesday 15 April
Research the second topic,
craft your second text answer
Tutor
marking of
exercise 2
Mark your second answer using the criteria
provided and mark two peer’s answers
Complete a short evaluation form where your
opinion of the exercise will be sought.
Summary marks returned
via VLE – Self, peer and
tutor marks compared.
Statistics provided overall
to judge the success of the
exercise
Dr Richard Parsons
[email protected]
Student Performance
Environmental science exercise - Write a short piece of text
that describes the major scientific reasons why global
warming is almost certainly occurring at present. The
total length of the text must be between 200 to 300
words.
• Students performed well, range of marks awarded and
obtained. Reasonable correlation between self, peer and
tutor marking
• Students recognised that they were learning about
assessment. Positive feedback received
• One of those exercises where you see the students
learning
Criterion vs Norm-based Marking
100
Self / Peer / Tutor mean
80
60
40
20
0
0
20
40
60
80
Expert (Norm-based reference)
100
Student comments demonstrate
increased understanding
“A good computer program that more classes should use”
“A great learning tool for the students. Enjoyable too, better than
the labs”
“Useful for understanding what markers look for in a piece of work”
“It was interesting to see how the marking process takes place, and
how you can have an essay that in its own right is well written but
does not contain the points needed by the marker”
“Assessing others gives me comparison to my own”
“Good to see how we can improve our work by our mistakes”
“Useful for the ability to receive results via text message”
“It was good to be able to compare your own work with others and
offered an insight into the basic processes staff must use”
“Makes you realise that you need more depth in your answer and to
read the question”
“It is quite funky”
Demonstration of plagiarism/copying/
collusion detection
A check for copying is now embedded within the program
http://www.personal.dundee.ac.uk/~rparsons/leu/zz02087process.htm
Operates in the following manner:
Recursive check for 6 word sets, ignoring most punctuation
Tutor can insert text from web pages, textbooks and lecture
notes as required
No service is used to store the student’s work, so no
complicated legal issues
Will not pick up copying from year to year, or institution to
institution
Online Self and Peer Assessment Exercise
Formative to Summative Assessments: Considerations
Formative
Summative
Low stakes
High stakes
Checks
No tutor marking,
copying detection
Experience
Initial exercise
limited training
Moderation
Little moderation
of marking
Random tutor
marking, copying
detection
Training in
assessment
Anomalous
marks adjusted
Comparative tutor
marking, plagiarism
detection
Students experienced at
self and peer assessment
Students gain marks for
marking correctly
Further development
Should be able to soon offer
• to include images
• check submission step
• present criteria to students
before exercise
• students receive marks for
marking
And perhaps eventually include
• feedback on feedback
• computer marking of free text
Self and Peer Assessment
What I leave you with
• Online system is available
• Relatively easy to use with supporting documentation
• It has some features that can engage students (text
messaging) and staff (plagiarism detection)
• Academic issues remain important
–
–
–
–
Question design
Students trained in assessment techniques
Student understanding of what constitutes plagiarism
Policy for dealing with plagiarism (formative work)
• Further developments are possible, suggestions
welcomed