Transcript Effective application of quality assurance & enhancement

Effective application of quality
assurance & enhancement
procedures to e-learning courses
Workshop at
University of Reading
7 July 2008
Welcome
Project team
Institute of Education
Dr Harvey Mellar
Dr Magdalena Jara
University of Reading
Maria-Christiana
Papaefthimiou
University of Derby
Dr Dave O'Hare
UoL External System
Brian Sayer
University of Teesside
Eileen Webb
Quality Assurance
Agency
Dr David Cairns
Workshop background

IOE Pathfinder Pilot – PREEL
Included revision of QA procedures for e-learning courses

IOE research study on QA/QE in e-learning
Examined effectiveness of internal quality assurance procedures for online
courses

Reading Pathfinder DIRECT
Driving Institutional Reform: Exploring Change with Technology

Quality Assurance Agency
In 2006 the QAA began a new cycle of audits with a more enhancementfocused approach
Purposes of the workshop

Identify any issues in quality assurance/enhancement
procedures that arise from the use of technology in
teaching and learning

Identify how these issues are reflected in your own
current institutional procedures

Identify how the internal QA/QE procedures of other
institutions deal with these issues

Identify ways to make your institutional QA/QE
procedures more effective in assuring and enhancing the
quality of TEL.
Day overview
10.00
Introduction
10.15
Part 1: Making terms explicit
10.55
Part 2: Internal QA/QE procedures for TEL
11.35
Coffee
11.45
Part 3: Issues affecting effective implementation of
internal QA/QE procedures for TEL
12.45
Lunch
13.30
Part 4: Planning actions to improve the effective
application of QA/QE procedures to TEL
14.40
QA/QE in e-learning SIG
15.00
Close
Part 1
Making terms explicit
Technology
Technology enhanced learning
 E-learning
 On-line teaching

QAA’s framework - IQAPs

Internal quality assurance procedures
Each [HEI] is responsible for the standards and quality of
its academic awards and programmes. Each has its own
internal procedures for attaining appropriate standards
and assuring and enhancing the quality of its provision.
In particular, institutions address their responsibilities for
standards and quality through:
 the assessment of students;
 their procedures for the design, approval, and the
monitoring and review of programmes.
QAA (2003) A brief guide to quality assurance in UK higher education. www.qaa.ac.uk
Quality enhancement
.. the process of taking deliberate steps at institutional
level to improve the quality of learning
opportunities....[…] … an aspect of institutional quality
management that is designed to secure, in the context of
the constraints within which individual institutions
operate, steady, reliable and demonstrable
improvements in the quality of learning opportunities.
QAA (2006) Handbook for institutional audit: England and Northern Ireland
The definition of 'enhancement' QAA has adopted ….
leaves room for institutions to follow their own definitions
of 'enhancement'. Some institutions may define
enhancement as 'continuous improvement', others as
'innovation' and there may be other definitions.
QAA (2007) Audit and enhancement: further guidance for institutions and QAA audit teams
QE and TEL

Enhancement


improvement or innovation
TEL


Is an innovation, hence enhancement?
Need to improve TEL?
Task 1
(15 minutes)
In each group…


What do you mean by QE?
How can you use the internal quality assurance
procedures to achieve QE?
Report back – one point per group
Swinglehurst, D. (2006) Peer Observation of Teaching in the Online Environment:
an action research approach. Final Report. Centre for Distance Education
QA/QE – the difference


Quality assurance and quality enhancement are not the
same as the first is concerned with determining that
objectives and aims have been achieved, while quality
enhancement is concerned with making improvements.
Quality enhancement is part of a wider framework in
which quality control, quality assurance, quality
enhancement and transformation are stages in the
management of quality
Common belief that quality assurance leads naturally to
quality enhancement is not correct, as most quality
assurance efforts are by and large concentrated in
accountability
Middlehurst, R. (1997), 'Enhancing Quality'. In F. Coffield and B. Williamson (eds),
Repositioning Higher Education. Buckingham: SRHE and Open University Press.
Retrospective vs prospective

Quality assurance may be either ‘retrospective’ or
‘prospective’ depending on the type of quality it is aiming
to assure.
Retrospective QA looks into the past to make a
judgement with a focus on accountability

Prospective QA is concerned with the present and
future, focusing on quality as fit for purpose, and
encouraging improvement

Defines QE as the internal mechanisms that an
institution puts in place to continually review and improve
practice
Biggs, J. (2001), 'The reflective institution: assuring and enhancing the
quality of teaching and learning'. Higher Education, 41, 221 - 238
Accountability vs improvement

… external evaluations have accountability and
compliance focuses rather than the encouragement of
continuous quality improvement of the student
experience.
In most institutions where it occurs, improvement of the
student experience is a function of internal review and
monitoring processes, usually heavily reliant, nowadays,
on student feedback, examiners reports, internal
improvement audits, periodic revalidation of programmes
of study and staff teams critically self-reflecting on their
everyday practice.
All internal processes of quality monitoring have a
greater effect on the quality of the provision than the
external monitoring processes
Harvey, L. (2005), 'A history and critique of quality evaluation in the UK'.
Quality Assurance in Education, 13 (4), 263 – 276.
Part 2
Internal QA/QE Procedures
for TEL
Internal QA procedures
1.
Validation
2.
Module evaluation
3.
Annual course review
4.
Periodic course review
5.
Student representation
6.
External examiner
7.
Peer observation
8.
Other…
Task 2
(15 minutes)
In each group…
Considering the internal QA procedures in place in your
own institution, discuss:

Does TEL require new/changed internal QA procedures?

Does TEL enable new/changed internal QA procedures?
Report back
National overview
HEIs' Audit Reports overview regarding QA of e-learning
Total = 129
Reports no e-learning,
(31) 24%
Does not mention
QA of e-learning
(8) 6%
Mentions particular QA
procedures
of e-learning
(80) 62%
Mentions QA of
e-learning in general
terms only
(10) 8%
(QAA audits 2003-2006)
Percentage of HEIs mentioning each QA procedure for e-learning
0%
External Examiners
Student feedback
Student Representative
Staff feedback
Team meetings
Annual Review
Periodic Review
Validation/approval
Peer review
Other QA procedures
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
HEIs reporting variations in the application
of the procedures to e-learning
Total
(n=129)
Procedure
External Examiners
Student feedback
14
11%
Annual Review
11
9%
Periodic Review
10
8%
Validation/Approval
35
27%
Student representatives
Staff feedback
Team meetings/staff development
Peer Review
Other QA procedures
Variations
Student
feedback

Annual
Review

Periodic
Review

Specially tailored forms
Implementation of online surveys and discussion
forums to collect feedback (to improve low response
rates)

Additional issues to be considered by review team
Additional scrutiny
Additional section to be discussed

External assessor with experience in e-learning


To include an evaluation of the development and
effectiveness of teaching, learning and assessment
strategy and method
Validation Additional consultations/approval steps:


Consultation/approval by specialised units
Proposals seen by additional committees
Sustainability:
 Additional scrutiny of economic viability and availability
of resources (risk assessment, contingency plans)
E-learning mode of delivery:
 Demonstrate appropriateness of delivery method,
explain how student discussions will be facilitated
Materials:
 Provide sample materials for review
 Internal and external peer review
Validation Panel members:
 At least one member of the panel needs to have
expertise in the area
Academic staff:
 Extra checks on staff development needs
Coffee break
Part 3
Issues affecting effective
implementation of internal
QA/QE procedures for TEL
QA/QE for TEL
The literature identifies differences between TEL
and campus-based learning:

Disaggregated processes

Distributed teams

Distant location of students

Openness to review
Research study

Four case studies




Postgraduate courses in a range of universities
Explore how dual-mode HEIs approach the application of their
internal QA procedures to their e-learning courses
For each case study

QA documentation

Interviews with stakeholders
Comparative examination of data:


Map of issues not captured by the QA procedures
Identification of those aspects of the courses which were
impacting on the implementation of the procedures
Research results
The application of the QA procedures was affected by:

Organisational context in which courses are located

Disaggregated processes

Distributed teams

Distant location of students

Openness to review
1. Organizational context - detachment
Online courses were often in a ‘detached’ position in their
institutions, which created both a sense of autonomy and
isolation



On-line courses off senior management’s agendas
Courses without central oversight
This led to failure to collect relevant information
2. Disaggregated processes
Distributed organisation of teams in terms of roles affected
the levels of coordination and communication among team
members
Unclear distribution of responsibilities
 Collaboration with external and specialised units added
further complexity

3. Distributed teams - coordination
Online courses were often taught by a mixture of full time
tutors, tutors with fee-based contracts and tutors working
from home or elsewhere. Course teams were scattered
and course leaders were often not aware of the
coordination requirements of a distributed team
On campus team with face-to-face meetings, tutors fully
integrated
 Course leaders tend to use the same coordination and
feedback mechanisms used for on-campus staff

4. Distant location of students
Distant location of students also affected the quality
assurance mechanisms, as it obstructed the
implementation of some of the procedures in their current
form (e.g. the use of student representatives)
Compensated for by a strong and trusting relationship
between students and tutors
 Opportunity was only occasionally taken up by teams

Effect on internal procedures
Annual Reviews
 Module evaluations
 Team meetings

Annual Reviews

Approached as an administrative burden, which had to
be written up just for accountability purposes
 Senior management is perceived as not prepared/not
able to understand the relevance of TEL issues

Annual review often detached from its enhancement
function
 Perceived as an event rather than a process
 Teams render two different accounts, one for external
consumption and another to be used internally
Module evaluations

Distant location of students…
 low response rates
 teams discarding the results as invalid, regardless
of their content

Distributed organisation of teams…
 unclear allocation of responsibilities
 led to responses being left untouched or only
superficially analysed
Team meetings

Distributed teams
 absence of a structure of formal meetings
 put at risk the team’s capacity to deal with the
issues identified and the monitoring of their
resolution
Evidence:



consistency in the information managed by team members
mechanisms for coordination and feedback relying on informal
encounters
Adjustments to overcome the limitations usually took the
form of increasing the formalisation of the
communication and coordination channels either online
or face-to-face
Task 3
(35 minutes)
Select one internal QA procedure
In each group:
 Quickly review the procedure’s purpose and how should it work
 Quickly review how the procedure is intended to aid quality
enhancement
 Describe any issues impacting on the use of this procedure for
QA/QE arising from the use of TEL
 Suggest solutions
Produce a one page written report:
 List issues identified affecting the procedure selected
 List suggestions for changes/improvement
Lunch break!
Part 4
Planning actions to improve the
effective application of QA/QE
procedures to e-learning courses
Techniques - research & practice

Validation criteria

Evaluation of e-learning courses

Embedded evaluation

PROPP Peer Observation

Periodic review framework
1. Additional validation criteria (IoE)

Staff experience of, or training in, use of on-line teaching
methods

Evaluation plan

Arrangements for the support of students with disabilities

Explicit course management procedures (including the
role of administrators, and a named person responsible
for quality assurance)

Arrangements for obtaining feedback from students

Explicit arrangements for tutor peer observation
2. Recommendations for evaluation

Obtain feedback from all stakeholders: students, tutors,
administrators, technical support

Organise frequent formal staff meetings, face-to-face
and/or online, with agendas defined by the staff, and
covering all key issues

Carry out evaluation as an integral part of the activities of
the course, both during and after the course

Ensure that responsibility for collection and analysis of
results is clearly assigned

Take advantage of the technology in use in the course to
collect feedback
Evaluation of e-learning courses WLE Occasional Paper 4
3. Embedded evaluation

Project studying e-learners’ experiences in a mixedmode course

Student learning is enhanced by evaluation which is
concurrent with teaching

Embedding evaluation tasks as part of the activities of the course

encouraging students to think about their own learning and how the
course design, materials and/or activities have supported them (or not)
in this pr
Daly, C. Pachler, N., Pickering, J. & Bezemer, J. (2006)
A study of e-learners’ experiences in the mixed-mode professional degree programme, the Master of Teaching.
Examples of evaluation questions
Writing online or in face to face seminar
MASTER OF TEACHING
Only two of you had prior experience of learning online before
joining the MTeach, so this has been a new way of communicating
to learn for everybody else in the group. These discussions form a
significant alternative to talking about issues at the face-to-face days
or in traditional seminars. What has it been like to ‘discuss’ by using
writing to communicate with each other like this?
RESEARCH SYNTHESIS FOR POLICY AND PRACTICE
Conducting a systematic review is a collaborative process, involving
both face-to-face meetings and online discussions/activities. The
course has been designed to mimic this process - carrying out
activities both in the workshops and online - how has it been to learn
like this? Please offer your thoughts…
4. PROPP Peer Observation

Project studying the implementation of peer observation
of teaching for tutors of an online course

Development & piloting of approach to peer observation

regular meetings course tutors are held

at each meeting, individual tutors bring an example of a problem, issue
or a project (e.g. feedback prepared for a student, transcript of a virtual
seminar, marks awarded for student’s work)

this material is used as the basis for discussion with other tutors
supporting reflection, and challenging assumptions
Swinglehurst (2008) Peer Observation of Teaching in the Online Environment: an action research approach
Example of a PROPP meeting
Topic

How do we assess our students?
Evidence:

Two student tutor marked assignments with the marks and tutor feedback
Main outputs




The instructions in the assignment task were ambiguous, and so needed to
be altered
The task needed to be broken down into smaller components
Written feedback should include specific examples rather than relying on
general comments which may not be well understood by students
Feedback should refer back to previous feedback given to students, it is
worth exploring the possibilities for enabling this more easily within the VLE
5. Periodic review framework
University of Reading
Pathfinder DIRECT - Driving Institutional Reform: Exploring Change
with Technology
Plenary discussion
Summaries of discussions from Task 3
Discussion:

How can your institutional procedures be made more
effective in assuring and enhancing the quality of TEL?

What obstacles stand in the way of improving these
institutional procedures ?
QA-QE SIG

JISCmail list: ‘QA-QE SIG’
to join go to http://jiscmail.ac.uk and sign up

More information:
www.lkl.ac.uk/research/qa-elearning
Evaluation of workshop

External evaluator (Charlotte Creed)

By email: link to online evaluation (10 min max)