Transcript Assurance of assessment quality in Higher Education

UQ-VN PDSS
ASSURANCE OF ASSESSMENT QUALITY
IN HIGHER EDUCATION
UQ Vietnam Professional Development Seminar
Series 4: April 2012
Ho Chi Minh City: Hanoi: Hue: Danang
Dr Clair Hughes [email protected]
Senior Lecturer in Higher Education
The Teaching and Educational Development Institute (TEDI)
Session overview
This session will address:
1.
2.
3.
4.
5.
6.
The influence of the international standards agenda on
quality assurance practices
The importance of assessment in providing credible
assurance of the quality of student learning
Approaches to the assurance of the quality of assessment
tasks
Characteristics of assessment quality
Approaches to the assurance of assessment judgements
Implications for assessment quality assurance responsibilities
Definitions
•
Program - a period of study of several years duration leading to the award of a
degree
•
Course - a period of study usually of a semester’s duration – a component of
a program
•
Assessment - the process of forming a judgment about the quality and extent
of student achievement or performance (Sadler 2005, p. 177).
•
Evaluation - the process of forming a judgement about the quality of (in this
context) an educational experience or one or more of its components
•
Criterion – a distinguishing property or characteristic of anything, by which its
quality can be judged or estimated. (Sadler 2005, p. 178)
•
Standard - a definite level of excellence or attainment.(Sadler 2005, p. 189)
The assessment task
• Your colleague Ms Hoa is new to your department and
has asked for your advice on an assessment task she has
developed for a new course in English language. The
main learning objectives are for students to:
– improve their English vocabulary and pronunciation
– develop awareness of the type of information discussed in a job
interview
– present their skills and knowledge well to a future employer.
CRICOS Provider No 00025B
Activity 1:
• Ms Hoa has not taught this type of course before and so
she asks your advice on this task:
• Her ideas include asking students to:
A. think of 3 likely questions they will be asked in an interview and
prepare written answers to these questions
B. role-play a job interview with a partner using questions provided
by the teacher
C. write an essay on how to behave in a job interview
D. prepare a CV and a letter of application
E. watch a video of a job interview and write about what went well
and what could have been improved
• What feedback would you give her on these suggestions?
Why?
CRICOS Provider No 00025B
Focus of assessment quality assurance (1)
Task quality
Purpose
Does the task allow students
to demonstrate intended
learning outcomes – it is
aligned?
Type of text
Does the task require the
students to produce the type
of work common to your
discipline?
PURPOSE
TEXT TYPE
Roles and relationships
Does the task give students
a realistic role or audience –
is it authentic?
Roles and
Relationships
Subject
Matter
Assessment as
TEXT
production
Subject matter or topic
Does the task address
significant subject matter?
Mode &
Medium
Mode
Does the task require students
to use forms of
communication used in your
discipline – spoken, written,
visual?
Medium
Does the task require students
to use the media used in your
discipline – face-to-face,
webpages, posters, journals
etc?
Hughes 2009
Focus of assessment quality assurance
Task quality- Example from Engineering (Project report)
Purpose
Task requires students to
address learning outcomes
related to teamwork,
engineering construction and
project management
PURPOSE
TEXT TYPE
Roles and relationships
Report requires students to
take the role of a
professional engineer
working with a client
Mode
The report contains written
and visual material and is
also presented orally
Roles and
Relationships
Subject
Matter
Assessment as
TEXT
production
Subject matter or topic
Project requires students to
use wide range of
engineering skills and
reflect on teamwork skills
Type of text
The report is to be of the
same quality as those
produced by professional
engineers
Mode &
Medium
Medium
An electronically produced
report is common in the
engineering industry
Hughes 2009
Activity 3: Task analysis
Think of an assessment task with which you are familiar and
analyse it as for the example on the previous slide
Purpose
Type of text
PURPOSE
TEXT TYPE
Roles and relationships
Mode
Roles and
Relationships
Subject
Matter
Assessment as
TEXT
production
Subject matter or topic
Mode &
Medium
Medium
Hughes 2009
Focus of assessment quality assurance (2)
• Also consider task in relation to:
•
•
•
•
•
•
•
•
CRICOS Provider No 00025B
compliance with assessment policy and rules
clarity of assessment description
other tasks in this course (how they fit together)
assessment tasks in other courses studied at the same time
(spread, timing, workload)
assessment tasks across the program (how they allow for
progression in difficulty and skill development)
accreditation requirements of professional bodies
provision of feedback
its contribution to overall grade
Activity 2
•
After seeking advice from a few people Ms Hoa decides on a 2-part
assessment task:
1. Students develop a CV and letter of application that addresses a
particular job they have found in a newspaper
2. Students role play a job interview with one student in the role of the
interviewer and one in the role of the applicant
•
•
She is anxious not to make any mistakes and asks you for further advice:
Select two of the following as being important things for her to do:
A. Read the university assessment policy and guidelines
B. Talk to the program coordinator
C. Not to do anything as the task is already quite good
D. Look at the assessment for other courses in this department
E. Look at student survey ratings for other courses in this department
F. Read some articles on assessment in educational journals
G. Other
CRICOS Provider No 00025B
Approval
• The university has an approval process in place for new
courses. Ms Hoa gains approval for the task from:
– the program coordinator
– The university Course and Program Approval Committees
CRICOS Provider No 00025B
Activity 3
• Ms Hoa wants to collect some evidence of her
assessment quality so you advise her to (select two of the
following):
A.
B.
C.
D.
E.
F.
Make notes while students are doing the role-play
Nothing as it is more important to spend time on her research
Look at the assessment items on the student survey ratings
Ask some students for their opinion in a group interview
Ask the program coordinator to sit in on some of the role plays
Ask the program coordinator to mark some of the CVs and letters
of application
G. Analyse what students did well or poorly on their written CVs and
letters of application
H. Other?
CRICOS Provider No 00025B
Activity 4
•
As there were a lot of students in the class, Ms Hoa had the help of 2 tutors in
marking the CVs, letters of application and role plays. Some students
complained that one of the tutors gave much lower marks than Ms Hoa and
the other tutor. What advice would you offer her so that this does not happen
next time the course is offered? Select two of the following.
A.
B.
C.
D.
E.
F.
G.
H.
Keep some of the CVs and letters of application (with student permission) from
the first course and use them to train the two tutors before they mark student work
Develop a detailed marking guide and give it to the tutors
Collect a random sample of marked work to check the marks of both tutors
Video all student role-plays and look at a random sample of them to check the
marks
Sit in on the first couple of role-plays and discuss tutor marks before they go
ahead with the rest of the marking
Mark a couple of CVs and letters together before doing the rest of the marking
Meet with the tutors to discuss examples of high and low marks and any that were
difficult to mark
Other
CRICOS Provider No 00025B
Activity 5
 Global activity in Quality Assurance reflects a shift of emphasis from
teaching to learning
• Tuning in Europe, South America
• Tuning USA – Lumina Foundation
• Learning and Teaching Academic Standards project of the
Australian Learning and Teaching Council (now OLT)
• As a result, the Ministry of Education is asking universities to collect
evidence that they are encouraging teachers to engage in the
assurance of assessment quality. Your University conducts a survey
and Ms Hoa is one of a group of randomly selected teachers who is
asked to write a short report on what she has done to assure
assessment quality in her course. List all the activities she could
include in her report.
CRICOS Provider No 00025B
SUMMARY
CRICOS Provider No 00025B
The international standards agenda
 Assuring the quality of student learning outcomes through:
• articulation of minimum or threshold standards
• standardised testing (AHELO, CLA)
• external examining and moderation - comparison of
standards within and between institutions
• development or revision of student satisfaction surveys
 Funded projects to identify and support effective practice
and explore emerging issues
http://www.itl.usyd.edu.au/projects/aaglo/
Assessment – a key curriculum component
Learning
Objectives
What do we
want students
to learn?
Assessment
How will students
demonstrate their
learning?
Credible
evidence of
student learning
outcomes
Learning activities
What will students
do to develop their
learning?
Teaching
activities
How will teachers
facilitate student
learning?
Assessment quality is the responsibility of.....
Senior management
• VC
• DVC (Academic)
• President of academic
Board
• Chair of Teaching and
Learning Committee
• Chair of Teaching and
Learning Sub-Committee
• Chair of Program Approval
Committee
• Chairs of relevant Working
Parties (e.g. policy on
Assessment, Academic
Integrity, Curriculum and
Teaching Quality
Assurance, Student
Charter)
CRICOS Provider No 00025B
Faculty and school
leaders
• Faculty Dean
• Heads of School
• Assistant Deans
(Academic)
• Chief Examiners
• Chair of Faculty/School
Teaching and Learning
Committee
• Chair of faculty or school
program approval
committees
Program and course
staff
• Program coordinators
• Course coordinators
• Teaching staff (including
casuals)
• Tutors
• Administration staff
Other
• Members of working
parties
• Academic Development
Unit staff
Different roles – different QA
responsibilities
External,
government
agencies
Monitor institutional
processes and
outcomes
Professional
bodies and
employers
Institutional
leaders
Support and assure
industry standards
Establish policies a
supporting
infrastructure
Faculty
leaders
Support &monitor
institutional
policies
Program
&course
coordinators
Comply with
assessment and
QA policies
Teachers
&admini.
staff
Students
CRICOS Provider No 00025B
Deliver program
and course
assessment
Undertake
assessment with
integrity
Quality assurance of assessment
Policy and
infrastructure
Create appropriate roles with
descriptions that refer to
responsibility for assessment
quality
Develop and cross-reference
policy, guidelines and rules for:
• Assessment
•Quality assurance
•Academic Integrity
•Student Charter
•Appointment and promotion
•Academic roles
Create infrastructure required for
effective implementation , e.g.
•Data collection strategies
•Support technology
•Professional development
•Staffing
CRICOS Provider No 00025B
Approval
Take a whole of program
approach to assessment
planning (e.g. map learning
objectives and/or assessment
across program)
Put approval processes in place
for new or revised assessment
Make the approval focus
appropriate for level e.g.
•Course – detail of course quality
•Institutional – policy compliance
Make approval consequential
(i.e. a response is required when
proposed assessment needs
improvement)
Review/evaluation
Put in place regular and
systematic review or evaluation
Collect data on assessment
quality (e.g. student satisfaction,
student outcomes, grade
distributions, benchmarking with
comparable programs) and make
it available
Make evaluation formal and
consequential (i.e. systematic
processes are guided by
checklists or templates, it is clear
who is responsible, findings must
be addressed and
recommendations and responses
are reported)
Streamline institutional QA and
accreditation processes – no
gaps, no duplications
Assurance of the quality of judgements
Calibration – activities that develop
shared understanding of standards
before judgements are made
(Consensus) moderation – activities
to compare, confirm, defend or adjust
judgements and standards
• Develop activities to induct new
academics – e.g. provide workshops
to make and discuss judgements on
samples of student work from
previous years (video used for
performance or presentation)
• Hold regular meetings of tutors to
discuss expectations and agree on
advice provided to students
• Have all assessors mark some
samples of student work and then
agree on standards before marking
the rest
• Coordinators can check random
samples of all assessors judgements
and negotiate necessary adjustments
• Coordinators can attend oral
presentations or performance and
compare their judgements with those
made by tutors
• Student work can be double-marked
and then discussed
• Chief examiners or school
committees can monitor grade
distributions across courses and ask
academics for an explanation where
necessary
• Colleagues from other institutions can
review and comment on standards
CRICOS Provider No 00025B
External assurance of quality of
student learning outcomes
• Development of discipline-specific graduate learning outcomes
(QAA in the UK: Tuning in Europe: ALTC in Australia)
• Quality audits (AUQA and now TEQSA in Australia: QAA in the
UK)
• Standardised testing
– Generic (AHELO: CLA)
– Discipline specific (AHELO Engineering and Economics:
Under consideration by national discipline communities –
chemistry, mathematics, medicine)
• External examining or review (Well established external
examining system in UK: QVS and other funded projects trialled
in Australia)
• Accreditation panels of professional associations
• National student surveys
CRICOS Provider No 00025B
Emerging issues
• Variable or ‘hidden’ standards
• Modular or otherwise fragmented approaches to curriculum and
assessment
• Increasing class sizes, student diversity and academic
workloads
• Inconsistent or limiting policies
• Debatable validity of ‘generic’ approaches to assessment –
strong arguments for discipline-specific approaches and
instruments for standardised testing
• Other criticisms of generic, standard instruments – e.g. Does
CLA measure aptitude or learning gains?
• Doubtful value of external examining processes based on
partial information
• Emphasis on QA rather than QI
CRICOS Provider No 00025B
Effective practice
• Characteristics of effective quality assurance (QA) and improvement
(QI) of assessment quality processes
– Clear standards – process and learning outcomes
– Systematic and formal processes coordinated across all administrative
levels and policies, integrated with professional development and
appropriately resources
– Whole of program approaches – avoidance of fragmentation
– Widespread participation – assessment quality the responsibility of all
– Balance of efficiency (e.g. automation) with meaningful involvement –
avoidance of perfunctory or ritualistic compliance
– Consequential – assessment strengths are recognised and rewarded and
weaknesses are addressed
– Appropriate use and interpretation of standardised processes and
instruments
– Collaboration and peer review – both internal and external – as
encouragement of interaction and dialogue that support QA and QI
CRICOS Provider No 00025B
Activity 6: Ongoing quality improvement
• Thinking about your role in the university:
–
–
–
–
What have you learned?
What workshop ideas are most relevant to your role?
How has the workshop affirmed what you are already doing?
What are the implications for your future practice?
CRICOS Provider No 00025B
References
•
•
•
•
•
•
•
•
•
•
AAGLO Project. (2011). AAGLO Summary 2: Assurance of graduate learning outcomes through external review.
Retrieved from http://www.itl.usyd.edu.au/projects/aaglo/pdf/AAGLO_Summary%202_Final.pdf.
Barr, R. B., & Tagg, J. (2004). From Teaching to Learning - A New Paradigm for Undergraduate Education. Retrieved
from http://ilte.ius.edu/pdf/BarrTagg.pdf
Hughes, C. (2009). Assessment as text production: drawing on systemic functional linguistics to frame the design and
analysis of assessment tasks. Assessment and Evaluation in Higher Education. 34(5), 553-563.
Krause, K., Scott, G., & Colleagues. (2011). Learning and Teaching Standards Project: Inter-university peer review
and moderation of coursework and assessment. Retrieved from http://www.uws.edu.au/LATS
Lumina Foundation. (n.d.). Tuning USA. Retrieved from http://www.luminafoundation.org/?s=tuning
OECD. (2011). Assessment of Higher Education Learning Outcomes. AHELO Project Update - May 2011. Retrieved
from http://www.oecd.org/dataoecd/8/26/48088270.pdf
The Quality Assurance Agency for Higher Education. (2012). UK Quality Code for Higher Education Part B: Assuring
and enhancing academic quality. Retrieved from http://www.qaa.ac.uk/AssuringStandardsAndQuality/qualitycode/Pages/Quality-Code-Part-B.aspx
Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and
Evaluation in Higher Education, 30(2), 175–194
Tertiary Education Quality and Standards Agency. (2011). Developing a Framework for Teaching and Learning
Standards in Australian Higher Education and the Role of TEQSA: Discussion Paper. Retrieved from
http://www.deewr.gov.au/HigherEducation/Policy/teqsa/Documents/Teaching_Learning_Discussion_Paper.pdf.
Tuning Educational Structures in Europe. (n.d.). Tuning Educational Structures in Europe: What is Tuning? Retrieved
from http://www.unideusto.org/tuningeu/