Transcript Quality & Effectiveness of E-Learning Courses – Some Experiences from Singapore

Quality & Effectiveness of E-Learning
Courses – Some Experiences from
Singapore
Lim Kin Chew
16 – 17 Dec 2010
7th International E-Learning Conference for a Knowledge-Based
Society
Srisakdi Charmonman IT Center
Assumption University
Bangkok, Thailand
Contents
•
•
•
•
•
Introduction
What is E-Learning Quality?
What is E-Learning Effectiveness?
Quality initiatives in Singapore
What more can we do to improve E-Learning
Quality?
• Conclusion
Introduction
This is my blog on
Quality Matters in
E-Learning. I
started this blog
some time in 2007
mainly to share my
experiences and
information on
learning objects
and e-learning in
general.
http://eunisim.blogspot.com
Request for Consultancy
Received e-mail request from a government body’s elearning Team:
1. E-learning features that support the pedagogical
design of e-learning content / instructional
objectives
2. Criteria for e-learning effectiveness
3. Evaluation of the various teaching & learning
domains, i.e. facilitation; transference; retention of
learning, etc
4. Key competencies which e-learning aims to equip
learners
Some deductions
• Quality and effectiveness of e-learning or etraining are becoming important issues, not
just the development of contents or the use of
LMSs
What is E-Learning Quality?
• The predominant view is that quality relates
to obtaining the best learning achievements
(50%). Together with 'something that is
excellent in performance' (19%), this primarily
pedagogical understanding was more
widespread than options related to best value
for money or marketing.
(from “Quality in e-Learning – use and dissemination of quality
approaches in European e-Learning – A study by the European
Observatory – www.cedefop.eu.int)
Business Times, Monday, 3 Sep 2001
BizIT Section
Business Times is a newspaper in Singapore that
deals mainly with business matters.
Quotation from BT journalist, Chua Hui Min:
“Effectiveness, rather than technology, is
the key, and defining this is perhaps both
the most difficult and important step in
deciding how to invest in e-learning.”
Entities where e-learning is
implemented widely
•
•
•
•
•
ICT Masterplan for schools
Polytechnics and Universities
Corporate bodies
Singapore Armed Forces
Civil Service College
The ICT Masterplan Journey - 4
The ICT Masterplan Journey - 1
• Managed by the Ministry of Education (www.moe.gov.sg)
• E-learning was given a major boost with the implementation
of the following IT Masterplans for Education:
– 1st Masterplan for IT in Education (1997 – 2002) – MP1
– 2nd Masterplan for IT in Education (2003 – 2008) – MP2
– 3rd Masterplan for IT in Education (2009 – 2014) – MP3
• 1st Masterplan for IT in Education:
– Build the strong foundation for schools to harness ICT
– Provide basic ICT infrastructure
– Equip teachers with a basic level of ICT integration &
competency
– 363 schools had fundamental building blocks in place to
use ICT in the curriculum
– 30% of ICT usage in curriculum
The ICT Masterplan Journey - 2
• The Second Information Technology in Education Study (or
SITES 2) conducted by the International Association for the
Evaluation of Educational Achievement (or IEA) in 1999
reflected this paradigm change – accepting ICT as a tool in
education.
• It reported that Singapore stood out favourably in terms of
the level of teachers’ knowledge of ICT-based instructional
practice.
• They had more avenues to develop their ICT skills than their
counterparts in most other countries.
• Singapore principals were very positive towards ICT in schools.
• They had an attitude score of 90 on a scale of 0 to100.
• Mental barriers of old mindsets had been successfully
breached.
The ICT Masterplan Journey - 3
• 2nd IT Masterplan for Education:
– built on foundation from MP1 to strive for an
effective and pervasive use of ICT in education by,
for example,
• strengthening the integration of ICT into the
curriculum,
• establishing baseline ICT standards for students, and
• seeding innovative use of ICT among schools.
The ICT Masterplan Journey - 5
The ICT Masterplan Journey - 6
The ICT Masterplan Journey - 7
• West Zone Sharing of Resources Project, WeSHARE in
short, is such an example of a first digital repository
project developed for teachers by teachers in 2006.
• The project has now been expanded to every zone as
part of the Inter-cluster Sharing of Resources project,
or iSHARE. To date, a total of 12,450 teachers have
gone into iSHARE both to upload and use the more
than 66,000 resources in the repository.
• These resources have been metatagged using the
CEMS (Content Exchange Metadata Standard) that is
based on Dublin Core Metadata Set.
The ICT Masterplan Journey - 8
Lessons learnt:
• A gap continues to exist between familiarity
with ICT and translating this into effective
teaching.
• Teachers need to still base effective outcomes
on sound pedagogical principles when they
use ICT tools to bring out a learning point.
The ICT Masterplan Journey - 9
Wrong use of technology:
• Students were asked to use the Internet to search for
information. However, they were given only 15
minutes to discuss what they have found before they
put up a PowerPoint presentation. What’s more
important? Discussion or using technology?
• This use of ICT neither transforms nor enhances the
learning experience.
Better use of technology:
• There were well constructed lessons where teachers
require their primary students to use simple
technological tools like discussion forums to create
and write stories that teachers and their classmates
could critique.
The ICT Masterplan Journey - 10
• Masterplan for ICT in Education 3 Learning Outcomes:
– Strengthen competencies for self-directed learning
– Tailor learning experiences according to the way that each
student learns best
– Encourage students to go deeper and advance their
learning
– Learn anywhere
Source:
http://www.moe.gov.sg/media/speeches/2008/08/0
5/opening-address-by-dr-ng-eng-h-1.php
Polytechnics & Universities - 1
• 5 polytechnics in Singapore:
–
–
–
–
–
Singapore Polytechnic
Ngee Ann Polytechnic
Temasek Polytechnic
Nanyang Polytechnic
Republic Polytechnic
• All polytechnics and the Institute of Technical
Education use e-learning widely.
• Every polytechnic has a dedicated department to
look after the design and development of e-learning
content and use of other ICT technologies like video
conferencing.
• No centralization of e-learning contents.
Polytechnics and Universities - 2
• Universities:
–
–
–
–
–
National University of Singapore (NUS)
Nanyang Technological University (NTU)
Singapore Management University (SMU)
SIM University (UniSIM) – Private University
Singapore University of Technology and Design (SUTD)
• Quality of e-learning is only one component in the Quality
Assurance Framework for Universities (QAFU). All universities
must go through this QAFU audit. QAFU was set up in 2003.
• Responsibility of e-Learning rests with the individual
university’s department responsible for e-learning.
• There is very little sharing or collaboration amongst all the
universities. Partly this is because of rivalry.
Source:
http://www.moe.gov.sg/media/press/2003/pr20030528.htm
Corporate Bodies -1
• E-LEAP – E-Learning Early Adopters’
Programme:
– Introduced by the IDA in 2002.
– Purpose is to encourage companies and corporate
bodies to use e-Learning for staff training
– Funding provided for non-governmental
companies to implement e-Learning
• E-Learning Competency Centre (ECC) helped
to ascertain the courseware quality
Corporate Bodies - 2
Courseware
Quality
Corporate Bodies - 3
Source: International Journal of The Computer, the
Internet and Management Vol. 12 No. 2 (May - August,
2004) pp 64 -71
Corporate Bodies - 4
• 2003 - 2004, the ECC developed some
Courseware Evaluation Guidelines for the ELEAP.
• Guidelines are on gathering evaluation
opinions on 10 criteria
Corporate Bodies - 5
–10 Quality assessment criteria
i. Interface
ii. Navigation
iii. Operation
iv. Content
v. Presentation
vi. Practice
vii. Feedback
viii.Assessment
ix. Engagement Techniques
x. Support
Singapore Armed Forces – SPOT-ON - 1
• 1997 – Introduced the Self-Paced On Time On
Need (SPOT-ON) programme
• Intention is to tailor as much training as possible
in accordance to individual needs, rate of learning
and time of convenience
• Initially the move was more of distance learning
packages using manuals and CD-ROMs
• When the Internet became popular, e-Learning
becomes an extension of SPOT-ON.
Singapore Armed Forces – SPOT-ON - 2
• 1999 - the Battalion Tactics Course (BTC) went online.
• 2000 – the National Service Command and Staff Course
(NSCSC) went online.
• The Instructional Technology Centre at SAFTI developed a
five-point management indicator to analyze the
following:
• course results,
• internal validation reports,
• focus group surveys,
• interviews and the
• instructors’ feedback.
Singapore Armed Forces – SPOT-ON - 4
The P’s
Purpose
Explanation
The effect of e-Learning on meeting the
organization's aim and course objectives
Players
The effect of e-Learning on the students
and instructors
Pedagogy
The effect of e-Learning on adult
education
Process
The effect of e-Learning on training
system processes ranging from
curriculum design to delivery
Performance The quality of the end-product
Civil Service of the Singapore Government
Some of the e-learning programmes for the civil servants are
organized and offered by the Civil Service College of Singapore
(http://www.ipamonline.com)
They also managed the quality & effectiveness of all the elearning courses.
Ministries and statutory boards can choose from over 200 titles
in the comprehensive library of e-Learning courses. Public
organisations implementing e-Learning with Open Academy can
now enjoy a 50% development grant on courses.!
CIVIL SERVICE COLLEGE - IPAM OPEN ACADEMY
Model used in SIM University
Quality & Effectiveness Model for UniSIM
Use of Kirkpatrick’s 4-level Model
(done by ASTD in 1996)
Level 1 (Reaction)
92%
Level 2 (Learning)
34%
Level 3 (Behaviour)
11%
Level 4 (Results)
2%
Most of the e-learning courses do have some sort of evaluations done by the
learners but these tend to be at Level 1 of the Kirkpatrick’s Model.
There is no available information on surveys done on the effectiveness.
What more can we do to improve eLearning quality?
• Promote the awareness of research work on quality
of e-learning e.g. the ISO/IEC JTC1 SC36 Working
Group 5 on Quality assurance and descriptive
frameworks
• Create collaborative projects involving quality and
learning measurements
• Disseminate case studies on quality and learning
measurements
• Share resources on quality and learning
measurements
• Carry out quality audit of e-learning courses
What more can we do to improve e-Learning quality?
Move from activity measures to performance measure
Source:
http://www.elearningguild.com/pd
f/2/110303mgt-h%5b1%5d.pdf
Conclusion - 1
• All organizations doing e-learning know the importance and
value of checking the quality of e-learning.
• However, there is no consensus of how to go about doing it in
a systematic way.
• Some organizations, especially the government bodies, are
very interested in measuring the performance of participants
who have completed e-learning courses.
• Schools are more interested in the pedagogical values that
the students have attained.
• Quality checking and/or measurements of e-learning courses
in the corporate bodies depend on the initiatives of their
individual Human Resource Department.
• Much work need to be done in raising the need to monitoring
the quality of e-learning programmes and also to measure it.
Conclusion - 2
From Robert Wager (1997):
To rise from zero
To big campus hero
To answer these questions you’ll strive to answer
the following questions:
Where am I going,
How shall I get there, and
How will I know I’ve arrived?
End of Presentation
Lim Kin Chew
[email protected]