Transcript Quality Evaluation methodologies for e-Learning systems (in the frame of the EC Project UNITE) Tatiana Rikure Researcher, Riga Technical University (RTU), Latvia UNITE “Unified e-Learning Environment.

Quality Evaluation methodologies for
e-Learning systems
(in the frame of the EC Project UNITE)
Tatiana Rikure
Researcher, Riga Technical University (RTU), Latvia
UNITE
“Unified e-Learning Environment for the School”
• EC 6th FP IST, STREP
• started: February 2006, duration: 30 months
• 13 partners from 11 countries (coordinator: Fraunhofer Institute for Computer
Graphics)
• The strategic goal:
is to contribute to the improvement of European wide education in schools based on
common, innovative principles in technology, pedagogy, scenarios as well as validation
of success
• The strategic objective:
is the development of a set of frameworks to support Europe-wide eLearning in schools
and to adapt these frameworks to national needs by "instantiation".
UNITE
Results
“Unified e-Learning Environment for the School”
•
UNITE e-Learning concept (new eLearning scenarios based on innovative
principles of e-Learning and a sound
pedagogical model)
•
UNITE system (a flexible e-Learning ICT
platform based on an innovative e-Learning
platform & pedagogical methodology)
•
Validation & evaluation methodology
(the quality of the UNITE system & the
success of the e-Learning scenarios)
•
Socio-economic evaluation
•
Europe-wide network of schools
•
Europe-wide repository of the re-usable eLearning content
•
etc.
• will be implemented in 14 secondary
schools in 11 countries
UNITE “Unified e-Learning Environment for the School”
System
Building the basis of UNITE, offering:
• Multilingual schooling environment.
• Personalized space and resource areas for pupils, teachers, parents.
• Personalised agenda for pupils, teachers, parents.
• Process-oriented assignments and evaluations.
• Asynchronous / synchronous communication tools.
• Multilevel access management.
• Parents’ monitoring tools.
• Logging facilities.
• Any other necessary tools to be identified by the schools.
Microcosmos
communication
& collaboration
UNITE
mobile
learning
LMS
m-learning
Integration of mobile learning facilities in UNITE:
• Incorporation of different uses of mobile phone devices in enhancing
the learning experience and test their effectiveness.
MTS
• Creation of pedagogically meaningful “chunks” of learning over mobile
phone devices (P900, multimedia mobile phone, XDAs, etc ).
• Incorporation of the use of a ‘media board’ to combine “in-house”
and “outside” schooling activities.
Incorporation of the MTS Learning Management System (LMS):
• Adaptation of MTS to match the specific school requirements.
• Central repository for learning elements and experiences.
• Development facilities for eLearning resources repositories in schools.
• Reusable learning materials, able to be combined with structured
lessons following certain pedagogical roles.
• Recording of learning histories.
UNITE “Unified e-Learning Environment for the School”
Validation framework
•
•
Test the impact and viability of the technology and the e-Learning
activities in a range of school settings
Show the appropriateness, meaningfulness and usefulness of the
UNITE system and the e-Learning scenarios
Riga Technical University (RTU) in
UNITE “Unified e-Learning Environment for the School”
RTU (Division of Applied Systems Software) main tasks:
 Designing the quantitative assessment methodologies for the validations
 Leading the validation activities in Latvia
Quality evaluation will be developed and executed on 2 levels:
•
Quality Assessment of the system:
• at the design phase,
• of the final product,
•
Measuring Efficiency (success \ impact) of the system:
• using qualitative evaluation,
• using quantitative evaluation.
Quality approaches
•
any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Design phase
Software testing
Software verification approaches:
• Static:
- Consistency techniques
- Measurement techniques
• Dynamic:
- White-box (structural) testing
- black-box (functional) testing
- Random testing
Expert evaluation
Bloom’s Taxonomy
Quality assurance
frameworks
Benchmarking
• Quality assurance guidelines
• Quality evaluation
criteria
• Quality marks
• Best Practices
Formative evaluation
• Critical commenting
• Developmental testing
• Revising
• Reviews (of literature, learners, learning
context, existing solutions, etc.)
• Walk-through
• Storyboards with potential users & experts
• Observations, video-taping, user-tracking,
interviews with users & experts
• Peer reviews
Final
product
Efficiency of
the system
Quality approaches
•
any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Final product
Design phase
Software testing
Software validation approaches:
• Formal methods
• Fault injections:
- Hardware
- Software
• Dependability analysis:
- Hazard analysis
- Risk analysis
Usability testing
Expert evaluation
Evaluation of the learning
content
Benchmarking
• Quality assurance guidelines
• Quality evaluation
criteria
• Quality marks
• Best Practices
Summative evaluation
• Feedback from users
(questionnaires, interviews, pre and
post tests, etc.)
• Revising
• Expert / Peer review
Efficiency of
the system
Quality approaches
•
any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have
the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE
systems’ validation.
Efficiency of the system
Design phase Final product
• Expert evaluation
• Feedback from learners and users
• Kirkpatrick's 4
Levels of Evaluation
• ROI
• Pedagogical
experiments
• Learners performance
testing
• Benchmarking
• Interviews (oral guided, written, telephone, etc.)
• Questionnaires
• Lists of criteria (checklists)
• Concept-mapping exercise
• Quality audits
• BLOGs
• Observations
• Case Studies
• Quality
management
activities
• WATTS (web-assessment techniques
and tools)
• Quality assurance guidelines
• Quality evaluation
criteria
• Quality marks • Knowledge
• Best Practices
• Collaborative assessment
techniques
surveys
Conclusions
•
The development of e-Learning products and the provision of e-learning
opportunities is one of the most rapidly expanding areas of education and training. ELearning is also one of the areas that attracts the most research and development
funding. If this investment is to be maximised, it is our challenge to generate robust
models for the systematic evaluation of e-learning and produce tools which are
flexible in use but consistent in results.
•
Measuring the effectiveness of e-Learning is an important issue in the development
and use of any e-Learning system or strategy, which not only shows the value of their
use, but also allows improving in the future.
•
Measuring the quality of the e-Learning systems should be done with the use of
multiple-method evaluation strategy that considers both qualitative and quantitative
criteria.
•
Evaluation of e-Learning is fundamentally the same as the evaluation of any other
learning but with particular groups of variables playing a more prominent role and the
impact of others differs significantly from their impact in traditional learning.
Acknowledgements
•
UNITE (Unified e-Learning Environment
for the School)
•
IST4Balt (Information Society
Technologies Promotion in Baltic States)
http://www.balticit.com/ist4balt/
•
eLOGMAR-M (Web-based and Mobile
Solutions for Collaborative Work
Environment with Logistics and Maritime
Applications)
http://www.elogmar-m.org
Contacts
Tatiana Rikure
Riga Technical University (RTU)
Faculty of Computer Science &
Information Technology
Division of Applied Systems Software
Meza str. ¼, Riga, Latvia
+371 7089096; fax: +371 7089572
[email protected]