Transcript Presentation title here and here

The Go8 Quality Verification System
Information and advice for reviewers
Introduction
This induction presentation includes information about: the current focus on
academic standards in Australia;
• the QVS as a distinctive approach of the Go8;
• the structure and objectives of the QVS;
• your role as a QVS reviewer and the activities outside the scope of your
role; and
• your role in evaluating the QVS.
The context for the QVS
Intensification of interest in setting and monitoring
academic standards
• Regulatory settings introduced from 2011 include a Higher Education
Standards Framework
• These standards are used to audit higher education providers under the
Tertiary Education Quality and Standards Agency (TEQSA) regulatory
framework.
• The Higher Education Standards Panel is currently revising the Standards
Framework.
The context for the QVS
• The Australian Qualifications Framework (AQF) has been revised to
describe each course level qualification in terms of the knowledge and
skills that a graduate will possess.
• Relevant international developments include:
- the Tuning Project in Europe to align degree structures (Tuning, n.d.);
and
- the OECD’s Assessing Higher Education Learning Outcomes
(AHELO) feasibility study in which Australia is participating (OECD,
2011).
The academic achievement of final year
students
A key reference point for standards
There are clear international trends towards accepting:
• students’ academic attainment as the primary reference point for
monitoring academic standards; and
• attainment in final year of study as a suitable point for benchmarking.
‘If academic standards are primarily defined by academic
attainment, the quality and robustness of process for assessing,
grading and reporting individual and group attainment are
paramount’.
(Richard James, 2010, presentation to the AUQA Auditors Meeting)
Aims of the Quality Verification System
(QVS)
Within the focus on academic standards:
• to demonstrate the appropriateness of the standards of learning
outcomes and grades awarded in Go8 universities;
• to maintain and improve the academic standards of Go8 universities;
• to enable comparisons of learning outcomes in similar subjects across
Go8 universities; and
• to promote discussion on good practice in teaching and learning in the
Go8 universities.
Distinctive features of the QVS
The QVS:
•
focuses upon assessment in a sample of final year subjects in
undergraduate programs (restricted to assessment tasks amenable to
review – e.g. verbal presentations are excluded);
•
concentrates on benchmarking for comparing grades awarded in similar
subjects across Go8 universities;
•
is conducted by senior academics in the relevant discipline who will have
an understanding of academic standards in leading universities around
the world; and
•
is sufficiently flexible to complement other quality assurance
mechanisms within Go8 universities to minimise duplication of efforts.
Core elements of the QVS
• The QVS is a process of external, discipline-led, academic peer review of
final year undergraduate student outcomes. The QVS will review a
minimum of 25 percent of final year assessment requirements (normally
at least two core subjects) in undergraduate programs across Go8
universities.
• The Go8 QVS Reviewers will:
o review the appropriateness and comparative quality of the specified
learning outcomes, assessment tasks, assessment criteria and
assessment processes set in samples of final year subjects in selected
fields of education; and
o report on the appropriateness of the grades awarded to stratified
random samples of student work in these subjects, drawing on their
academic judgment and the set of documents provided.
The QVS approach
Quality verification rather than grade moderation
The QVS involves quality verification by external reviewers. It is not
moderation of grades by external examiners. Your review will retrospectively
verify the appropriateness of grades awarded to a sample of student
assessment after their results have been published. Your review will not
influence the grades awarded to students enrolled in the subjects you
review.
In your role as an external QVS reviewer, you are not expected to:
• design assessment tasks;
• re-mark student work; or
• standardise student grades.
The QVS since 2011
Pilot (2011 and 2012)
The pilot reviewed:
• Physics
• History
• Psychology
• Accounting
• Chemistry
• Economics
• Philosophy
From 2013
From 2013 onwards, the QVS reviews a small number of new fields each
year, plus any fields still outstanding from previous years.
2013: Maths, English and Chinese (Mandarin)
2014: Political Science, Sociology and Geology
Process steps to ensure robustness and
credibility
• All QVS reviewers will declare any potential conflict of interest (e.g.
being involved in collaborative teaching, research or consultancy work
with colleagues teaching in the subjects being reviewed) at the outset
of the review and in the written report which will be published on the
website of the university being reviewed.
• Reciprocal reviewing arrangements of individual departments/faculties
between universities are avoided as far as possible.
Sensitive nature of materials for review
As a QVS reviewer,
• you sign a confidentiality agreement with the university being reviewed
before commencing the review;
• you must not disclose any information provided to you for the purpose
of the review without written permission of the QVS coordinator in the
university being reviewed;
• you must keep in mind that your report will be discussed in
departmental/faculty committees that are likely to have student
representatives;
• you accept that the university being reviewed will own copyright of any
material produced in relation to the review; and
• you will not retain any rights (including copyright and moral rights) in
connection with the materials produced for the review.
Your role as a QVS reviewer
You will be provided with the following materials for the subjects you will
review:
• the subject outline, subject guide and learning objectives;
• information on how the learning outcomes of this subject relate to the
degree program level outcomes;
• copies of the actual assessment tasks which students have completed
(such as exam scripts, specific essay assignments, workplacement requirements, performance-based assessment
arrangements, etc);
• the grading criteria used for each assessment task; and
• grade nomenclature and categories used within the university (e.g.
below pass, pass, credit, distinction and high distinction).
Verifying grades awarded
For each assessment task, you will also review a randomly selected and deidentified sample of 5% of student assessment at each of the following grade
levels or their equivalent – below pass, pass, credit, distinction and high
distinction.
• For subjects with small enrolments, a minimum of 5 items of student
work will be reviewed.
• For subjects with large enrolments, a maximum of 25 items of student
work will be reviewed.
• Where appropriate, stratified samples will include items of student work
from all campuses including off-shore campuses.
Carrying out a review
Within 2 weeks of receiving the materials, you will:
• review all of the materials you have received;
• liaise with the person who sent you the relevant material to obtain
further information for your review, if necessary;
• send your report (Attachment A of QVS Guidelines) to the QVS
coordinator in the university being reviewed (listed in the next slide);
and
• send a copy of your report to the Go8 Secretariat.
QVS coordinators
Monash University
The University of Western Australia
The University of Melbourne
The University of Queensland
The Australian National University
The University of Sydney
UNSW Australia
The University of Adelaide
Patrick Orchard
Project Advisor, Office of the Pro ViceChancellor (Learning and Teaching)
Winthrop Professor Grady Venville
Dean, Coursework Studies
Dr Marija Maher, Director, Academic
Strategy
Pauline Pavier
Executive Officer, Office of the Deputy
Vice-Chancellor (Academic)
Suzie Alcorn
Manager
Academic Standards and Quality Office
Professor Marie Carroll
Pro Vice-Chancellor (Academic Affairs)
Sonia Powell
A/g General Manager, Office of the Pro
Vice-Chancellor (Students)
Jessica Raeburn
Learning and Quality Support Officer
Key questions for review
• Are the learning outcomes appropriate for a final year undergraduate
subject?
• Are the learning outcomes comparable to those of final year subjects in
similar universities?
• Are assessment processes and the determination of grades rigorous,
equitable for diverse students and fairly conducted based on the
material you have been provided?
As a senior academic, you have the academic experience and judgment
to review the material provided to you. For the purpose of this review, the
next three slides suggest issues that you might consider for inclusion in
your report.
Review of learning objectives
• To what extent is the information provided about learning objectives
clear and sufficient?
• To what extent are the learning objectives “precise, challenging and
complete” (Laurillard 2002: 183)?
• How do the learning objectives specified for the subject compare with
those of final year subjects in similar universities?
Review of assessment tasks
• To what extent are the assessment tasks suitable for the specified
learning objectives?
• Is there sufficient variety and complexity in assessment tasks?
• To what extent are the assessment tasks timed appropriately?
• Is the language used in assessment tasks unambiguous, appropriate
and inclusive of all students?
• Are the marking criteria sufficiently clear?
• How do the assessment tasks and marking criteria compare with those
of final year subjects in similar universities?
Suggestions for summary comments
• In your experience, how do the specified learning outcomes and student
achievements compare with those of final year subjects in similar
universities?
• Are there key issues which should be brought to the attention of
supervising committees in the faculty/department, division or wider
university?
• Are there examples of good practice that might be noted and
disseminated more widely as appropriate?
(Adapted from the QAA code of practice on external examination)
Summary judgement on each unit
 The learning outcomes, assessment tasks and assessment processes
set for the subject I have reviewed were appropriate.
Any recommendations made are for the purposes of enhancement to the
subject and its assessment.
 The learning outcomes, assessment tasks and assessment processes
set for the subject I have reviewed were appropriate.
HOWEVER, there are some risks to the future quality assurance of the
subject and its assessment, as outlined in my recommendations.
 There are immediate concerns or risks relating to the learning outcomes,
assessment tasks and/or assessment processes set for the subject I
have reviewed.
These require immediate action on behalf of the University to prevent
reoccurrence in the next review.
(Adapted from Report Coversheet for External Examiners, the
University of Cambridge)
Report cover sheet
The Report Coversheet records the extent to which you agree with the
grades awarded to the sample of student work you have reviewed.
Number of
assessments
Reviewed in total
Agree with grade
awarded
Believe grade
awarded to be
unduly high
Believe grade
awarded to be
unduly low
Below
pass
Pass
Credit
Distinction
High
Distinction
Discussion and publication of review
summaries
• Your review report will be received by the QVS coordinator of the
university being reviewed and discussed with the relevant
Faculty/Department.
• The QVS coordinator will provide you with the University’s overall
response to your report.
• Review recommendations and schedule of discipline reviews will be
published on the websites of the participating universities.
Your evaluation of the QVS
You will be asked to provide feedback on your experience of being a QVS
reviewer:
•
Did you have access to sufficient information for the purpose of this
review?
•
What other information would have helped you undertake the review
more effectively and efficiently?
•
What is your overall assessment of the effectiveness and usefulness of
the process?
o What are the best aspects of this process?
o What could be improved?
Feedback from reviewers (and reviewees) will be to continuously
improve the QVS
•
References
James, R. (2010). The academic perspective on academic standards: The challenges
in making the implicit explicit. Presentation made to AUQA Auditors Meeting at Gold
Cost, June 30. Retrieved 24 April, 2011, from The University of Melbourne website at
http://www.cshe.unimelb.edu.au/people/james.html
Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the
effective use of learning technologies (2nd ed.). London: Routledge.
University of Cambridge (2009). Report coversheet for external examiners. Retrieved
April 24, 2011, from the University of Cambridge website at
http://www.admin.cam.ac.uk/offices/education/examiners/eecoversheet.pdf
Tuning (n.d.). Tuning Educational Structures in Europe. Retrieved April 24, 2011, from
http://tuning.unideusto.org/tuningeu/
OECD (2011). Testing student and university performance globally: OECD’s AHELO.
Retrieved April 24, 2011, from the OECD website at www.oecd.org/edu/ahelo
QAA (2004). Code of practice for the assurance of academic quality and standards in
higher education: External Examining. Retrieved 10 April, 2011, from the Quality
Assurance Agency for Higher Education website at www.qaa.ac.uk
Contact us
If you have any queries, or need more information, please contact
The Group of Eight secretariat at +61 (02) 6239 5488.