Transcript Document

Issues for M-learning research
Tom Brown
South Africa
Introduction
The exponential growth in the adoption and
implementation of m-learning is still characterised
by exploratory pilot projects and trials.
There are important issues to note and research
questions to answer in order to ensure that the
mainstream deployment of m-learning will be
successful. Some of these are:
• Is technology driving learning or are we adapting
and using technology to optimise learning?
• Are we imposing new technological tools onto old
teaching methods or are we focusing on the
innovative and transformative use of technology?
• How much emphasis do we put on the design and
development of learning environments?
Why is m-learning
thriving?
Why is m-learning thriving?
• “Around 62% of all adults across the major
European countries now use a mobile phone,
according to the research.
• Currently, 41% of European adults use SMS,
compared to 30% that use the Internet / email.
• SMS is particularly popular in the UK where 49%
of adults use it, compared to 39% who are online.
• In Germany, 43% of adults use SMS as opposed
to 29% of adults who use the Internet/email.
In France, 30% use SMS compared to 25% who
go online.”
Statistics from Gartner (2002)
Why is m-learning thriving?
•
•
•
•
•
•
“Over 50% of all employees spend up to half of their
time outside the office.
More than 525 million web-enabled phones will be
shipped by 2003.
Worldwide mobile commerce market will reach $200
billion by 2004.
There will be more than 1 billion wireless internet
subscribers worldwide by 2005.
Multi-purpose handheld devices (PDA and telephone)
will outsell laptop/desktop computers combined by
2005.
Most major US companies will either switch to or adopt
wireless networks by 2008.”
Statistics from Empowering Technologies Incorporated
cited by Keegan (2003)
Why is m-learning thriving?
• Between 1997 and 2001, the number of mobile
phone subscribers in Africa annually had a tripledigit growth rate. (Shapshak, 2002)
• Mobile subscribers in Africa have increased by
over 1000% between 1998 and 2003 to reach
51.8m (ITU, 2004)
• In 1999 Tokyo had more telecom connections than
Africa combined. In 2003 Africa had twice as
much as Tokyo. (Gourley, 2004)
• On average, in South Africa, 98% of rural students
have basic mobile phones
• Africa is leapfrogging from an unwired, nonexistent e-learning infrastructure to a wireless
e-learning and m-learning infrastructure.
Why is m-learning thriving?
• Desmond Keegan recently (2003) published his
latest book called: ‘The Future of Learning: From
eLearning to mLearning.’ In chapter four of his
book, Keegan presents and analyses no less than
30 m-learning initiatives across the globe in 2001.
• NKI Distance Education in Norway has 400 elearning courses. During 2003 and 2004 it
announced that it had made available mobile
learning versions of all its 400 courses. This
represents a massive introduction of mobile
learning.
Why is m-learning thriving?
• Exponential growth in wireless networks, services
and devices
• Learners are continually demanding more mobile
services and experiences
• Greater personalisation, flexibility and mobility
• Improved access anywhere, anytime
• Fills small gaps of time with useful learning events
(“stolen moments for learning” David Metcalf)
• M-learning enhances collaborative, co-operative
and active learning
• Mobile communication devices provide
opportunities for the optimising of interaction and
communication between lecturers and learners,
among learners and between members of COPs.
Why is m-learning thriving?
“For the first time in ICT history, we have the
right time, the right place and the right idea to
have a huge impact on education: handheld
computing.”
Soloway (2003)
Why is m-learning thriving?
“The mixing of distance learning with mobile
telephony to produce mLearning will provide
the future of learning.”
Keegan (2003)
Why is m-learning thriving?
“By 2006, data network access from
personally owned mobile devices will be the
leading problem facing higher education IT
managers.”
Gartner (2004)
Why is m-learning thriving?
“The mobile revolution is finally here.
Wherever one looks, the evidence of mobile
penetration and adoption is irrefutable: cell
phones, PDAs, MP3 players, portable game
devices, handhelds, tablets and laptops
abound. No demographic is immune from
this phenomenon. From toddlers to seniors,
people are increasingly connected and are
digitally communicating with each other in
ways that would have been impossible to
imagine only a few years ago.“
Wagner (2005)
Why is m-learning thriving?
“Whether we like it or not, whether we are
ready for it or not, mobile learning represents
the next step in a long tradition of technology
mediated learning. It will feature new
strategies, practices, tools, applications, and
resources to realize the promise of
ubiquitous, pervasive, personal, and
connected learning. It responds to the ondemand learning interests of connected
citizens in an information-centric world.”
Wagner (2005)
What is the relation
between m-learning
and e-learning?
m-learning vs e-learning
• E-learning is the macro concept that includes
online and mobile learning environments.
• M-learning is a subset of e-learning.
• E-learning is in turn a subset of distance learning,
which is in turn a subset of flexible learning.
“M-learning is e-learning through mobile
computational devices”
Quin (2001)
m-learning vs e-learning
Flexible Learning
Contact Learning
(residential/face-to-face)
Distance Learning
E-learning
Online
Learning
M-learning
Paper-based
Distance Learning
Diagram 1: The subsets of flexible learning (Brown, 2004)
Approaches to m-learning
Approaches to
the use of m-learning
technologies
Content
approach
Communication
approach
Focus on communication
“Communication is the source from which
m-learning emerges. ”
Nyiri (2002)
Current m-learning
activities and projects
in Africa?
Target markets for M-learning
We can differentiate between two ideal target
markets for m-learning:
• learners that are either without (fixed line/wired)
infrastructure and access to the Internet, or
• learners that require mobility, flexibility and access
anywhere, anytime.
In other words:
• 3rd world rural or remote area learners who have
basic mobile phones, and
• 1st world learners with state of the art mobile
devices.
Brown (2004)
Current activities in Africa
The use of PDAs – two examples at UP:
• M-learning project in the Faculty of Health
Sciences. Using PDAs in clinical assessment
sessions of medical students: assessing
performance and providing automated results and
feedback. (Ina Treadwell) Project software funded
by HaPerT software in Vienna, Austria.
• M-learning project in the Faculty of Engineering.
Students in a 4th year course have been issued
with PDAs to use within a pilot wireless e-learning
environment. PDAs are used for queries, content
delivery, interactive distributed simulations,
notices, database access, collaboration, etc.
(Etienne Barnard) Project funded by Hewlett
Packard.
Current activities in Africa
The use of PDAs in clinical assessment
sessions of health science students:
Research is being done on:
• Impact of PDA use on assessment quality
• Impact of PDA use on student performance
• Impact on efficiency and effectiveness
(impact on administrative load, time, paper work,
human errors, calculation errors, record keeping,
duplication, costs, etc)
Success:
• Project still in progress. No official results yet.
Feedback already very positive regarding
efficiency, effectiveness and cost savings.
Current activities in Africa
The use of PDAs in wireless environments:
•
•
•
•
•
Research is being done on:
HLT (Human Language Technologies):
- speech recognition and speech-to-text
- voice user interfaces
Ability to stimulate collaboration with PDAs
Mobile sharing of software and resources
Multi-user applications and resources (multiplayer
games are very popular)
Wireless VoIP
Success:
• Project still in progress. No official results yet.
Examples - Part 1
The use of bulk SMS
for administrative
support
SMS for admin support
• For example: The University of Pretoria started
using mobile phone support during 2002 in three
paper-based distance education programmes
because more than 99% of the “rural students” had
mobile phones (still the case – currently 98%).
• The profile of these students:
Majority live in rural areas
100% are full-time employees (teaching)
77.4% are English second language speakers
83.8% are between the age of 31 – 50
66.4% are women
0.4% have access to e-mail
99.4% have a mobile phone
M-learning in rural Africa
M-learning in rural Africa
M-learning in rural Africa
M-learning in rural Africa
M-learning in rural Africa
M-learning in rural Africa
SMS for admin support
What does the mobile phone support
entail?
• Bulk SMS (pre-planned) to all students or students
of a specific programme for general administrative
support as well as motivational support
• Customised group SMS to specific groups of
students extracted from the data-base for specific
administrative support
• Customised small group or individual SMS to
specific students extracted from the data-base on
an individual basis for specific admin support
• Project team: Johan Hendrikz, Anita van der Bank,
Tom Brown, Jeanne-Marie Viljoen
SMS examples
Dear Student. Your study material was posted to
you today. Enquire in time, quote your tracking
number: PE123456789ZA, at your post office. UP
Purpose:
• Students do not visit their rural post offices very
often and this leads to many returned packages
• If students know about a dispatch, they make an
effort to fetch packages timely
Success:
• Significant drop in returned packages and
accompanying costs
SMS examples
If you have not submitted Assignment 2, due to
late dispatch of study material, you may submit
before 19 Sept. Do this urgently to help you pass
your exam. UP
Purpose:
• Extension of assignment submission date due to a
late dispatch of study material
• Encouragement to complete the assignment
Success:
• Normal assignment submission statistics
SMS examples
ACE Edu Management contact session block 1
from 7-9 July for modules EDM 401 EDO 401
ONLY, changed to Town Hall Main Street
KOKSTAD. New letter posted. UP
Purpose:
• Urgent notification of a venue change for a specific
contact session
Success:
• All the students arrived at the correct venue (as far
as we know)
SMS examples
Dear Student. We have not received your
registration for the Oct exam. Please fax
registration form or letter not later than Thursday
31 July. UP
Purpose:
• Encouragement for exam registration
• Notification of the deadline for exam registration
Success:
• Increase in the number of exam registrations
compared to previous exams
SMS examples
April exam proved that students attending
contact sessions are more successful. Please
attend July contact session. Register per fax
before or on Friday 6 July. UP
Purpose:
• Encouragement for contact session registration
• Notification of the deadline for contact session
registration
Success:
• 58% of the learners registered before the closing
date vs the normal rate of below 40%.
SMS for admin support
From a logistical and financial point of view,
the successes are also significant:
• Using print and the postal service to distribute the
necessary information to students would have
been more than 20 times the cost of the bulk
SMSs.
• While the SMSs provide immediate and JIT (justin-time) information, the posted information would
have taken between 3 to 18 days (depending on
the remoteness of the student) to reach all the
students.
Examples - Part 2
The use of SMS for
academic learning
support
SMS for academic support
• The University of Pretoria started using SMS for
academic learning support in November 2004 in
one module of one of the three paper-based
distance education programmes in the Faculty of
Education, namely:
* ACE: Special Needs Education:
Module LPO402
• The pilot project comprises four categories of
asynchronous academic interventions during the
six-month cycle of this module from October 2004 April 2005.
• Project team: Jeanne-Marie Viljoen, Carl du Preez,
Johan Hendrikz, Anita van der Bank, Tom Brown
SMS for academic support
The four categories are:
• Academic instructional message (regular bulk
type SMS messages)
• IVR (interactive voice response) system for
FAQs (students phone in to a “FAQ number” and
receive answers from the programmed system)
• SMS quizzes (MCQ’s are send to students and a
simple answer choice is replied via SMS. Answers
and feedback are provided on each quiz)
• SMS question-answer system (students ask
questions via SMS regarding a given pre-selected
topic and then they are answered automatically by
the system via a comprehensive programmed
matching system [text database])
Category 1 example: Instruction
LPO 402 student: study section on Assets p43-44
in Tutorial Letters 1 booklet before answering 1.4
of Assign 1. This is also important for your Project
& Assign 2. UP
Purpose:
• Study tip for a difficult assignment question that
normally gets answered incorrectly by students
• Preparation for contact sessions
• Hint for the project and follow-up assignment
Envisaged outcome/success:
• Increase in the quality of assignment answers
• Increase in the quality of contact session interaction
Category 2 example: IVR
LPO 402 student: phone 0825557777 to hear
more about the most import concept in this
module, the asset-based approach. UP
Voice message when reaching 082555666:
Hello LPO 402 student. We will now discuss
some frequently asked questions on the assetbased approach that will enhance your
understanding of this important concept. Press 1
to hear what the asset-based approach is. Press
2 to hear what makes it so unique. Press 3 to
hear why you should use it.
Category 3 example: Quizzes
First question: Asset-based initiatives are clarified
on
a) p14 of learning guide
b) p14 of Assets textbook
c) p5 of tutorial letter 1
Reply by pressing a, b, or c & send. UP
Correct! The asset-based approach is ecosystemic. Eco-systemic approaches emphasize:
a)interrelatedness
b)individuality
c)neither
Press and send answer
Category 3 example: Quizzes
Correct! The asset-based approach is ecosystemic. Eco-systemic approaches emphasize:
a) interrelatedness
b) individuality
c) neither
Press and send answer
A needs-based approach emphasizes individuality
and a asset-based approach emphasizes
interrelatedness.
Press C & send
Category 3 example: Quizzes
Correct! You are on your way to reaching the 2nd
and 3rd outcomes of this unit. Now read p 15-18
in learning guide. Good luck! Bye.
Purpose:
• Review important content
• Tutoring the reach of desired learning outcomes
• Remedial support on identified learning
shortcomings
Envisaged outcome/success:
• Improve the quality of assignment answers and the
achievement of the desired learning outcomes
• Other successes is not yet determined - needs
further research
Category 4 example: SMS Q&A
Dear student: SMS your questions (1 question
per 1 SMS) on the topic: ECO-SYSTEMIC
APPROACHES to: 0825558888
Purpose:
• Provide students the opportunity to clarify issues
and questions without the high cost of a lengthy
telephone call
• Provide asynchronous learning support
• Lessen the impact on the call centre or the
faculty’s telephone tutoring
Envisaged outcome/success:
• Enhance achieving the desired learning outcomes
• Other not yet determined - needs further research
m-learning in Africa
Premises for m-learning in Africa - lessons
learnt from the pilot study:
• M-learning is a supportive mode of education and
not a primary mode of education.
• The most appropriate mobile device for learners in
rural Africa is a mobile phone.
• Possibilities and latest developments in mobile
technologies must be tested against practicality,
usability and cost-effectiveness.
• The use of multimedia on mobile phones must be
tested against the envisaged learning outcomes.
• The major focus of m-learning should be more on
communication and interaction than on content.
Summary of current
m-learning activities
across the world?
Current activities and projects
The use of mobile phones and SMS:
Administrative learning support:
• Bulk SMS for administrative information
• Access to examination and test marks via mobile
service number or m-portal
• Access to financial statements and registration
data via mobile service number or m-portal
Academic learning support:
• Communication and interaction (bulk SMS / IVR)
• Assessment (MCQs / Quizzes)
• Feedback on assignments and tasks
• Motivational and instructional messages
Current activities and projects
•
•
•
•
•
Integration of m-learning with established elearning environments:
M-portals and SMS-gateways (SMS-portal
integrated with the LMS/LCMS [e.g. WebCT])
Mobile tutoring
Mobile blogging
M-assessment (e-assessment on mobile devices)
Collaborative learning, discussion groups
Wireless environments:
• Pilot wireless classrooms
• Hot spots and wireless LANs on campus
Current activities and projects
The use of PDAs, smartphones and Pocket
PCs:
• Classroom “tools” (note taking, scheduling, etc)
• Beaming (via bluetooth) in classrooms (sharing
notes, handing in assignments, etc)
• Assessment: assessing performance and providing
automated results and feedback
• Coursework, scheduling and assignments in
wirless environments
Current activities and projects
• Language learning through SMS
• JIT on-the-spot (e.g. medical) resources on PDA
• ME-learning (personalised, appreciation for own
learning process)
• Mobile composing (music composition on PDAs)
• Contextual and locational awareness (e.g. at
museums)
• Mobile tutoring
• Mobile blogging (moblogging)
• Courseware and multimedia on PDAs (including
distribution and streaming)
• Assessment
• Experiential learning and fieldwork
• Collaborative learning, discussion groups
Noteable EU-funded projects
• MOBIlearn
(context awareness, adaptive human interfaces,
mLCMS, mobile media delivery, collaborative
learning, etc)
• M-Learning Project
(platforms & systems, learning materials for small
screens [various devices], collaborative tools, etc)
• Ericsson Leonardo Da Vinci Project
(mLCMS, courses and courseware, tools, etc)
Test_hp.exe
Latest and future developments
•
•
•
•
•
•
•
•
•
•
Moblogging
Instant messaging (IM)
Wireless Google
Collapse-to-Zoom and Popouts
Ambient technology and intelligence
Personalised learning with dynamic adaptation of
learning resources to individual preferences
Text to speech & speech recognition for mobile
devices
Multi-user applications and resources
Multi-technology interaction
Podcasting (broadcasting of audio to iPods)
Lessons learnt from
e-learning and
current m-learning
projects
Lessons learnt from e-learning
• Learning is a deeply personal act that is facilitated
when learning experiences are relevant, reliable,
and engaging.
• Different kinds of learning demand appropriate
strategies, tools, and resources.
• Technology in and of itself may not guarantee
better learning.
• The better the experience and the more intentional
the results, the greater is the likelihood that
learning will occur.
• Interoperability, standards, digital rights
management and content repositories is very
important.
Wagner (2005)
Why learners prefer paper
Why (contemporary) learners prefer content on
paper rather than on a PDA:
• Internal memory is limited and sometimes causes
the PDA to be too slow.
• Making notes on the PDAs is an effort and the
keyboard is too small.
• Reading a document requires too much scrolling.
• Pictures in the .doc or .pdf documents are not
always of high quality.
• It takes too much effort and more time to read
documents on a PDA.
• A printed version of the documents is easier to use.
Wentzel, P. et al (2005)
Could be a different story when future sophisticated PDAs offer similar functionalities as laptops or PCs.
Issues for m-learning
research
IBM’s m-learning questions
•
•
•
•
•
Some areas of IBM’s focus in m-learning as
described by Chris von Koschembar:
Repurposing existing e-learning content for PDA
and other applications
Re-engineering content development tools and
process so that information can be re-combined for
a host of possibilities
Facilitate “Take it with Me”
The “Urgent” Scenario, in which information is
provided to learners wherever they are the instant
the information changes
The “Downtime” Scenario, in which “chunks” of
learning can occur during the learner’s downtime
periods (e.g., at airports)
Some research questions
•
•
•
•
•
•
Student and faculty issues:
Which students (profile) would benefit most from mlearning?
What issues surround learner requirements and
usability from a pedagogical point of view?
Understanding your m-learners (student
characteristics)
Understanding your m-teachers (faculty
characteristics and competencies)
M-teaching: What are the skills and competencies
required as well as strategies and methodologies
required to facilitate m-learning?
What are the patterns and contexts of mobile use,
use patterns across disciplines and across
pedagogical approaches?
Some research questions
•
•
•
•
•
•
Teaching and learning strategies:
What educational activities and components are
most suitable for m-learning environments and
benefit most from m-learning devices?
How can we foster higher order learning through
m-learning?
How do the educators’ and tutors’ roles change
when we introduce m-learning?
How to engage students?
What type of communication / interaction between
students and teachers and among students can be
improved?
How can we optimise collaborative learning and the
growth of COLs (Communities of Learning)?
Some research questions
•
•
•
•
Content:
How will m-learning impact the education portfolio
universities offer?
Which subject disciplines and courses would
benefit most from m-learning?
How to design and develop m-content suitable for
multiple devices, including issues such as
transcoding and interoperability?
How to determine what and how much of content or
chunks of content should be designed and
developed or m-learning environments?
Some research questions
•
•
•
•
Student and learning support:
How will support mechanisms change with
widespread use?
What support will be needed to provide the
necessary level of support to m-learning
deployment?
How to cater for students with special needs
(assistive technologies)?
What should the role of moblogging (mobile
blogging) play in m-learning?
Some research questions
•
•
•
•
•
•
Organisational/institutional:
What are the financial implications for the institution
as well as the students?
What are the implications for facilities such as
classrooms, meeting rooms, computer labs,
libraries, etc?
What type of new facilities are needed, how will
they be used and what features will they have?
How to use m-learning to improve success rates as
well as recruitment and retention rates?
How to mainstream m-learning and some of the
research into standard practice at the institution?
How to ensure stable funding for continued
research in m-learning and m-teaching?
Some research questions
•
•
•
•
•
•
•
Technology:
What is/are the ideal mobile learning device(s)?
What are learners’ reasons for selecting particular
technologies? Why are specific devices most
valued?
How can mobile devices be best used as
communication tools in education and training?
What types of interface will optimise learning?
How can key usability features be added to the
current generation of mobile devices?
How can technical developments be communicated
to educators quickly and effectively?
How to ensure that faculty are mobile literate with
regards to m-learning and m-teaching?
Some research questions
•
•
•
•
•
Future:
What should content for future m-learning devices
and learning paradigms look like?
Are we ready and planning for the evolution of
communication skills and practices?
What will be the new ways of recording learning
experiences, sharing content and annotations
between mobile users?
How will portable speech recognition and text-tospeech software and technologies evolve and
impact our educational practice?
How will social and technical innovation influence
e- and m-learning?
Rich mobile environments
•
•
•
•
•
•
•
•
A rich mobile Internet experience includes the
following attributes:
Ubiquity: Availability of applications and devices
Access: Widely available wireless network
Richness: Speed, smoothness and seamlessness
Efficiency: Economies of scale
Flexibility: Usability for a variety of devices
Security: Viruses, access control, integrity,
interceptions, privacy, copyright, etc.
Reliability: Consistency regardless of browser,
device or screen size
Interactivity: Interaction with display and content
Wagner (2005)
Beware of the Pitfalls
Pitfalls
•
•
•
•
•
•
•
Beware of the following:
Technology (not pedagogy) driving learning
Imposing new technological tools onto old teaching
methods
Transferring learning materials without rethinking
their use
Suitability of technologies and devices for particular
teaching and learning activities
Loss of personalization benefits may lead to
rejection (e.g. access to and ownership of devices)
Informality of m-learning may lead to abandonment
because the learning is not taken as seriously
It takes time for m-learning to evolve (adoption,
adaptation, learner and faculty issues)
Pitfalls
Adoption Curve
The technology adoption lifecycle (Moore, 1991)
Pitfalls
•
•
•
•
•
Beware of the following:
Anytime, anywhere activities outside the classroom
may interfere with curriculum intentions. It may
also interfere with private, social life which may
lead to a resistance to m-learning.
Possibilities and latest developments in mobile
technologies must be tested against practicality,
usability and cost-effectiveness.
Little or inappropriate focus on pedagogy.
The implementation of m-learning technologies
must be tested against the envisaged learning
outcomes.
NB: Provision for students with disabilities!
2010 envisaged
• Wireless is “big” and mobile devices are “small”
• Ambient technology and intelligence
• Always-on wireless connections and ubiquitous
computing
• Wearable mobile technologies
• Bio-informatics a commercial reality
• New methods and approaches to learning and
collaborating with ICT
• Personalised learning with dynamic adaptation of
learning resources to individual preferences
• From courseware to performanceware
• m-LMSs and m-LCMSs
• Platforms supporting multi-user interaction on
software, applications and equipment
Thank you
References
Brown, J.S. (1999). Learning, Working & Playing in the Digital
Age. Paper delivered at the 1999 Conference on Higher
Education of the American Association for Higher Education,
March 1999, Washington, USA.
Brown, T.H. (2004) The role of m-learning in the future of elearning in Africa? In Distance Education and Technology:
Issues and Practice. Hong Kong.
Gartner (2002). SMS bigger than email in Europe. Taken from
Nua Internet Surveys, November 11, 2002. Cited in TAD
Consortium December 2002 Information Update No. 4,
Telematics for African Development Consortium,
Johannesburg, South Africa.
Gartner (2003). Emerging Technology Scenario. Paper
delivered by Gartner analyst Nick Jones at the Gartner
Symposium and ITxpo, 4 – 6 August 2003, Cape Town,
South Africa.
References
Gartner (2004) Predicts 2005: Higher Education Evolves,
Gartner Inc, Stamford, USA.
Keegan, D. (2003). The future of learning: From eLearning to
mLearning. Hagen: Fernstudienforchung, Germany.
Moore G.A. (1991) Crossing the chasm. Marketing and selling
high tech products to mainstream customers. Harper
Business.
Nyiri, K. (2002). Towards a philosophy of m-learning. Paper
delivered at the IEEE international workshop on wireless and
mobile technologies in education. August 29-30, 2002, Växjö
University, Sweden.
Oliveira, C. (2003). Towards a knowledge society. Keynote
address delivered at the IEEE international conference on
advanced learning technologies (ICALT). July 2003, Athens,
Greece.
References
Quin, C. (2001). mLearning: Mobile, Wireless, In-Your-Pocket
Learning. LiNE Zine [On-line], Fall 2002.
Soloway, E. (2003). Handheld computing: Right time, right
place, right idea. Paper delivered at the IEEE international
conference on advanced learning technologies (ICALT). July
2003, Athens, Greece.
Wagner, E.D. (2005) Enabling Mobile Learning. EDUCAUSE
Review, vol. 40, no. 3 (May/June 2005): 40–53.
Wentzel, P. et al (2005) Using Mobile Technology to Enhance
Students’ Educational Experiences. Case Study from the
EDUCAUSE Center for Applied Research, Boulder,
Colorado, USA.