Transcript University of Pretoria

Quality Management
in E-learning:
Towards an
integrated approach
Dr Jill Fresen and
Lesley Boyd
Dept of Telematic Learning
and Education Innovation,
University of Pretoria
Independent
Quality Assurance Consultant
Problem
“Quality assurance and online
delivery are hot topics in
universities, yet until recently
discussions of each have had
little to do with each other.”
Reid (2003)
Academic Exchange Quarterly, 7(1), 17-23.
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Towards integration
Where to next?
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Towards integration
Where to next?
Terminology
Contact Education
Distance Education
E-learning
face-to-face
Other terms:
Websupported
learning
paper-based
Web-based learning
Internet-based learning
Online learning
Technology-enhanced learning
E-Learning @ UP
Web-support
Multi-media (on CD-Rom)
Video production
Video conferencing
Interactive Television
Graphic services
Photographic services
Role players
Stakeholders
management
government
funders
community
Clients
lecturers
students
Practitioners
project manager
instructional designer
education consultant
programmer
graphic artist
library consultant
Who are our clients?
Lecturers
•
•
•
•
instructional design
educational support
training
technical support
Students
• end users
• training
• technical support
Case Studies
University of Southern Queensland
Distance Education Centre (DEC) first ISO9001 accreditation
University of Bangor in Wales - tools
for judging pedagogical quality
Frameworks, benchmarks and best
practice indicators: critical success
factors for e-learning
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Building blocks for the future
Where to next?
Methodology of the UP project
Apply standard QA theory to
e-learning applications, by
implementing a Quality
Management System
Investigate what factors
promote quality e-learning
Implement systematic
evaluation and customer
feedback systems
Methodology of the UP project
Apply standard QA theory to
e-learning applications, by
implementing a Quality
Management System
Investigate what factors
promote quality e-learning
Implement systematic
evaluation and customer
feedback systems
QMS Approach - 1
Pro-active evaluation and
improvement rather than
external accountability
Ownership and commitment
from all practitioners in the
E-education Unit
QMS Approach – 2
formal
online
Documented, auditable, visible
Minimise paperwork
process based
Understand the entire ID process
from beginning to end and how
awareness of this helps people to
work together effectively.
consensus driven
Based on consensus ownership not prescription
ISO9000 cognisant
Recognise useful principles behind ISO9000
QMS – Timescales
2001 TLEI management request a formal
quality approach to be implemented
Nov 2001 – May 2002 Training Workshops for
all management and staff
May 2002 – Sept 2003 Production of procedure
documentation & development of online QMS
Sept 2003 – loading of procedure documents
into online QMS
Oct 2003 Launch of ‘live’ QMS
2003 – 2005 ‘Death valley’ of implementation
2006 Self evaluation of system – recognised as
benchmark for services provided to clients
QMS – Objectives
To provide a defined framework for all role players to work together
consistently along the entire project timeline.
To enable everyone, including new staff, to understand ‘the way
things are done around here’.
To identify together areas for improvement.
To provide an integrated and simple method to access and use
supporting documentation eg checklists, forms, templates.
To ensure that the right tools are available to allow for
comprehensive checks and to minimise errors.
To try and catch any errors as soon as possible before it’s too late or
too expensive to fix them.
To evaluate completed projects and help to assess their impact on
teaching and learning at UP.
To learn lessons which may help to improve future projects.
To share more with each other about ways of doing things.
To demonstrate to any external stakeholders (e.g. auditors or UP
management) that TLEI has a formal quality management system in
place to control e-education projects.
QMS - Methodology
QA training workshops
Formalised intuitive
documentation such as the
Instructional Design Toolkit
and the Project Timeline
QMS Triangle
Ons slakkie
Quality Management System
Measurements and Targets
Feedback Loop
ACT
Strategy
Policies
Objectives
Processes
Procedures
Standards, checklists, forms
PLAN
DO
Detailed work instructions
Records
Boyd, 2001 – adapted from Waller, Allen & Burns, 1993
CONTROL
ISO 9000 model
Value adding activities
Information flow
SABS (2000)
Project Timeline
Based on traditional ADDIE model
of instructional design
Illustrates the major PROCESS
Project Timeline
Analysis
Procedures
Each step documented into a
PROCEDURE using:
•
•
•
•
task teams (jigsaw and pizza)
a procedure template
a fully worked example
a set of self evaluation questions
Procedures
Rapid prototyping and crossreferencing between task teams
All procedures then uploaded
into the live QMS with a
document control master list
Live system launched and left to
sink or swim!
Methodology of the UP project
Apply standard QA theory to
e-learning applications, by
implementing a Quality
Management System
Investigate what factors
promote quality e-learning
Implement systematic
evaluation and customer
feedback systems
Assumptions
Technology infrastructure
• reliability, accessibility, availability
• accuracy, stability of student data
Commitment, motivation of clients
Computer literacy of clients
Information literacy of clients
Positive attitude of lecturers
Consultation and support to lecturers
Sound instructional design practice
Sound teaching and learning practice
Commitment to cont. improvement
Exogenous factors
Quality of the institutional LMS
Stability of national telecoms infrastructure
Class size
Workload of lecturers and students
Recognition and incentives for lecturers
Critical Success Factors
Six studies from literature
Six categories:
•
•
•
•
•
•
Institutional Factors
Technology Factors
Lecturer Factors
Student Factors
Instructional Design Factors
Pedagogical Factors
Methodology of the UP project
Apply standard QA theory to
e-learning applications, by
implementing a Quality
Management System
Investigate what factors
promote quality e-learning
Implement systematic
evaluation and customer
feedback systems
Four levels
Reaction
Learning
Behaviour
Results
Customer
satisfaction
Return on
investment
Kirkpatrick, D. 1998
Client satisfaction
Student feedback survey –
online at end of each semester
Lecturer interviews – pilot
group of 22 active users
Student feedback survey
July 2003 baseline data
Overall Experience survey:
4 650 respondents
Response rate 27.4 %
•
•
•
•
•
Technical adequacy
Educational support
Affective domain
Online communication
Perceived learning
Student frustration - 2003
Frustration index:
73%
80
60
40
20
17%
10%
0
Low
Moderate
High
83% of students experience
moderate to high levels of frustration
Frustration index factors
insufficient computers available
insufficient printers available
extent of technical difficulties
insufficient support from student CD
inadequate student training in WebCT
an impersonal learning experience
slow response from classmates
feelings of annoyance and/or stress
Student satisfaction - 2003
Satisfaction index:
50
40
30
20
10
0
44%
43%
Moderate
High
13%
Low
87% of students experience
moderate to high levels of satisfaction
Satisfaction index factors
feeling comfortable communicating via
online tools
feeling more freedom to express oneself
than in a traditional classrooom
learning from the contributions of other
students
promoting one’s ability to work as a team
or group member
increased ability to plan one’s own work
enriching learning environment
Longitudinal measures
Measures of student satisfaction, 2003 – 2005
2003
2004
2005
I have access to my own computer (either at home,
at work or in the residence)
80.5%
78.6%
73.1%
I find it difficult to find a computer on campus when
necessary
85.1%
50.3%
36.5%
I find it difficult to find a printer on campus when
necessary
73.3%
45.3%
42.2%
My technical difficulties were solved within 24 hours
43.7%
79.3%
76.1%
I felt comfortable communicating via online tools
72.9%
80.0%
82.8%
Web-supported communication helped me to
express myself more than I would have in a
traditional classroom
79.1%
57.9%
60.9%
I found web-supported learning to be an enriching
learning experience
81.9%
77.7%
78.7%
I experienced feelings of annoyance or stress during
this web-supported learning experience
80.0%
37.9%
32.5%
I found the opportunities for ‘anywhere, anytime’
learning on the web to be convenient
73.8%
87.0%
88.3%
Lecturer feedback survey
Pilot interviews: 22 lecturers
Benefits
• useful for organisation and admin
• communication and interaction
• excellent support and service
Problems
• library / copyright issues
• human element is difficult
Lessons learnt
• training for staff and students
• e-learning adds value
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Towards integration
Where to next?
Lessons: Risks
formal Resistance to generation of documentation
taken versus perceived benefits
online Time
Work load pressures for development of
Expectation
that consultant should be
online system
process based
providing best practice procedures
Once created, it must be maintained and kept
Resistance
todate
ideasfor
ofinternal
process;functionality
associated
Inappropriate
expectations;
eg that theand
to up
to
consensus
driven
with system
commerce
and
inappropriate
HE?
will
guarantee
quality to
improvements
external
visibility
ISO9000
cognisant
Inability
to reach consensus amongst role
players or to drive this forward
Automatic dismissal and perceptions of
Expectation
consultant
be
bureaucracy
as soon asthat
ISO9000
term should
is
mentioned!providing ‘answers’
Tendency to overlook the value of the system
– that it is only a document management tool
Lessons: Benefits
Gartner hype cycle:
Source: http://news.bbc.co.uk/1/hi/technology/3577746.stm#graphic
Lessons: Benefits
1. Consensus that the QMS is on the slope of
enlightenment
2. Self evaluation and improvement of each step in
the process; consensus on ‘the way we do things
around here’
3. Rationalisation of pre-existing documents,
checklists and forms
4. The QMS is a repository for the most up to date
versions of all resources for planning, developing
& evaluating projects
5. System informs new staff on UP procedures and
enables them to feel self sufficient more quickly
Lessons learnt
1. Fundamental ID model (process)
2. Importance of Analysis and Evaluation
3. Timely training in the basics of QA theory
and practice
4. Doubts about the need for a formalised
QMS
5. Time required for reflection on practice
6. Roles and responsibilities of all role players
7. Manage lecturers’ unrealistic expectations
8. Understand that the system will go through
a change management cycle
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Towards integration
Where to next?
Conceptual framework
Applies process theory, critical success
factors, and four levels of evaluation into
an integrated framework
Conceptual framework
inputs
process
Institutional
factors
Instructional
Design
Process
Technology
factors
Lecturer
factors
ID factors
Student
factors
Pedagogical
factors
outputs
measures
Products
websupported
courses &
multimedia
products
feedback loop
Lecturer
satisfaction
Student
satisfaction
Agenda
Background of QA in e-learning
Methodology of project at UP
Lessons learnt
Towards integration
Where to next?
Where to next?
Use the QMS and the conceptual
framework to produce measures about
effectiveness, efficiency and contribution
of e-learning to teaching and learning
Use measures to prioritise improvements
Implement summative evaluation as a
joint exercise between all role players
Integrate evaluation of e-learning with
institutional programme review
Further reading
Fresen (2005)
Quality assurance practice in web-supported
learning in higher education. PhD thesis.
Available at http://upetd.up.ac.za/UPeTD.htm
Fresen & Boyd (2005)
Caught in the web of quality. International
Journal of Educational Development, 25(3),
317-331.
Fresen (in press)
A taxonomy of factors to promote quality websupported learning. International Journal on
E-learning. To appear in 2006.
Where to next?
Quality in e-learning means learning
transformed and enhanced with the
integration of technology where ‘e’
means:
empowered
enabled
engaged
enlightened
emotional
enthusiastic
Dr Ulf-Daniel Ehlers: Presentation at E-learning in Post-Secondary
Education Conference, Alberta, Canada June 2005
Sharing experiences
Jill Fresen:
[email protected]
Lesley Boyd:
[email protected]
Online QMS:
http://www.up.ac.za/telematic/quality/quality.htm