Transcript 10th EES Biennial Conference

Using Open Source Survey Tools for
Qualitative Inquiries on Educational
Development at a Distance Online
University
Corinne Bossé
Derek Briton
Cindy Ives
10th EES Biennial Conference
Helsinki, Finland
October 5, 2012
Introduction
“The web is more a social creation than a technical
one. I designed it for a social effect—to help
people work together—and not as a technical toy.
The ultimate goal of the Web is to support and
improve our weblike existence in the world”.
Tim Berners-Lee (2000). Weaving the Web: The Original Design and
Ultimate Destiny of the World Wide Web, (p.123).
Text
Quick Facts
• Almost 40,000 students (7800 FTEs)
• Almost 900 courses in 90 programs
• Annual operating budget of $128 million
• Student Stats:
~ average age: 29.4 undergrad.; 38.2 grad.
~ 81% work while they study
~ 67% are female
~ 70% are first in family to earn a degree
~ from every Canadian province & more than 90
countries
• Faculty & Staff Stats:
~ over 1300 on 4 Alberta campuses
~ academics – 177 full-time; 177 part-time
~ tutors & markers – 382 part-time
History and Processes
•
•
•
•
•
•
•
Unpaced, individual study
Continuous enrollment
Author-editor development model
Industrialized production model
Open University processes adapted
ISD approach > template for courses in a box
Behaviourist, cognitivist pedagogies
Institutional Context
• Distributed workforce (4 locations, home offices)
• Self-paced mode of undergraduate course delivery,
continuous enrollment
• almost 900 courses; 90+ programs
• Institutional goals
⁻
⁻
⁻
⁻
Quality and access
Flexible and open
Focused on student success and learning
Online as appropriate
Institutional Mandate (2009)
• remove barriers to undergraduate and graduate
education
• provide high-quality, interactive learning
environments
• actively pursue technological innovations that can
enhance teaching, research and administrative
functions
Conceptual framework
• Formative evaluation framework for the
qualitative enquiries conducted on 2 evaluation
projects
• Educational development perspective:
– Identify teaching and learning needs
– Document & assess an ongoing process of
improvements to online course learning activities
• Participatory methods and utilization-focused
approach (Patton, 2008)
Patton, M.Q. (2008). Utilization-Focused Evaluation. (4th ed.). Thousand Oaks, CA:
Sage.
Uses of Open Survey Tools
• Open education is an integral part of Athabasca
University’s organizational culture as one of the
pioneering online and distance teaching universities.
• Strong institutional support for open source tools such
as Lime Survey and Moodle, which is the university’s
learning management system (LMS).
• Technical integration within the institution makes it
easier to access and use these open source survey tools
as part of the academic practice for both faculty and
professionals.
Uses of Open Survey Tools
• Integrating open source tools to conduct
qualitative inquiries on recent educational
development initiatives sponsored by AU‘s Centre
for Learning Design and Development (CLDD) can
be viewed as a strategic alignment towards
supporting innovative teaching and learning
activities.
Uses of Open Survey Tools
• Rationale for using Moodle to conduct a needs
assessment:
– building on AU faculty’s familiarity with the LMS to raise
their awareness about the Moodle questionnaire module.
– One of the outcomes is to make use of this feature to
gather additional qualitative feedback from students to
enhance course design and development
• Rationale for using Lime Survey to conduct expert
review
– Providing an opportunity for faculty and professionals to
test the tool as well as responding to the object of the
qualitative inquiry focused on improving future course
learning objects design.
Evaluation Project I: Moodle
Questionnaire (2010)
• The main purpose of the needs assessment was
to help determine which educational
development activities could contribute to an
enhanced teaching and learning experience at
Athabasca University
Evaluation Project I: Moodle
Questionnaire (2010)
Evaluation Project II: Lime Survey (2011)
• Community Adjustment Fund Project ( part of
multimillion dollars funded by provincial
government to digitize AU resources)
• Formative evaluation of 20 Athabasca
University (AU) largest registration courses to
assess the usability of the learning objects from
both a technical and a pedagogical dimension
Evaluation Project II: Lime Survey (2011)
Evaluation Project II: Lime Survey (2011)
• Purpose of Expert Review :
– to obtain feedback from a community of expert
practitioners in learning design and educational
development that will inform and improve future
designs and uses of AU course learning objects (LOs)
– to assess the usability of selected learning objects on
two design dimensions: technical and pedagogical
Evaluation Project II: Lime Survey (2011)
• Reviewers’ selection
– Learning designers (LDs) and instructional media
analysts (IMAs) with at least five to more than ten
years’ experience in their field participated in the
expert review of showcase-course learning objects
Evaluation Project II: Lime Survey (2011)
• Questionnaire Design
– Drew from several sources based on literature review of LOs
– Relevant sections of these validated evaluative instruments
were adapted and contextualized to examine both the
technical and pedagogical dimensions of the CAF showcase
course enhancements
– Iterative process: piloted with an AU expert team in learning
design and educational development and it was revised a
number of times before finalizing the version used in the
formative evaluation
– Scope: 30-45 minutes to complete an in-depth review of Los
& complete the online Lime Survey
Evaluation Project II: Lime Survey (2011)
Evaluation Project II: Lime Survey (2011)
• An overview of the expert review key findings for all
the CAF showcase course enhancements:
• Technical Dimension
– The expert review indicates high levels of agreement about the
presentation design with 97% indicating that the overall look and
feel of the LOs is effective; 96% finding the LOs easy to use; and
around 82% indicating an appropriate level of technical integration
of the CAF showcase courses within the learning management
system (i.e. Moodle).
– The perceived level of technological interactivity tends to be low for
the majority (74%) of LOs.
– This information has subsequently being used to improve the latest
version of all the learning objects that were assessed in the
formative evaluation. Biology course enhancements were perceived
to have the highest level of technological interactivity.
Evaluation Project II: Lime Survey (2011)
• Pedagogical Dimension
Top 5 Intended Uses
– The majority of LOs (57%) are intended to help learner with
foundational concepts
– More than half (52%)of LOs are intended be used as revision or
review of new knowledge, a concept or skill and to provide for
multiple learning preferences beyond reading
– Slightly less than half (48%) of LOs are intended to help learners
develop new knowledge , a concept or skill
– About a third (35%) of the LOs are intended to be used as an
orienting or tuning-in activity
– Slightly less than a third (28%) as an instructor-directed
demonstration tool
– In general, several components along the pedagogical dimension
were rated with overall high levels of agreement in terms of
intended use, design interaction, accessibility, reuse, learner
interaction, motivation/engagement and effectiveness.
Recommendations/Future Directions
In sum, it is hoped that the findings of the formative evaluation will be useful ‘to
inform an ongoing cycle of reflection and innovation’ (Patton, 2008, p.116). A
potential course of action that can be inferred from the expert review results of the
course enhancements :
– Specifying both the intent and expected outcomes while designing and
integrating LOs into courses
– Examining the showcase course enhancements from students’ perspective by
integrating targeted evaluations on revised technical and pedagogical elements
of usability into the various courses; triangulating the perceived technical and
pedagogical dimensions with students’ actual use of the learning resources and
actual outcomes.
– Revisiting the reuse pedagogical components to integrate multiple formats and
configurations into which these learning resources can be embedded in online
learning environments
– Further adapting and revising a taxonomy of technological interactivity to better
define and assess how well LOs achieve interactivity
Recommendations/Future Directions
– Incorporating to greater extent built-in automated feedback into
the design of LOs to increase learners’ control (Nielsen, 2003)
– Expanding the design feature of accessibility to be more inclusive
by taking into account different range of teaching and learning
contexts comprised of diverse learners who might experience
disability (The Le@rningFederation ,2007)
– Using expert reviews as one of the learning design strategies to
enhance integration of multimedia learning resources into online
course design and development
– Tracking the actual use and reuse of these multimedia learning
resources by integrating them into open education resources
platforms that use analytics
– Monitoring and evaluating the usability of the learning resources
that are integrated into online courses over time to establish
institutional design benchmarks.
Conclusions
• Patton (2008) argues persuasively that
participation and involvement of evaluation’
stakeholders substantially increase the likelihood
of using the results of an evaluation in order to
advance societal as well as institutional interests.
• From an educational development standpoint, a
valuable outcome of using open source web-based
evaluative tools is:
– to engage people both in the process of evaluation itself
– to using tools to improve the online learning environment in
which they operate
Conclusions
• At this exploratory stage of the comparative
review, it is anticipated that Moodle and Lime
Survey will be embedded as part of AU
systematic research-based responses to
appropriately identify and address educational
development needs and challenges.