Transcript Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities Vijit Prabhu Computer Science & Engineering Indian School of Mines Dhanbad, Jharkhand, India Girijesh Prasad Intelligent Systems Research.

Designing a Virtual Keyboard with
Multi-Modal Access for People with
Disabilities
Vijit Prabhu
Computer Science & Engineering
Indian School of Mines
Dhanbad, Jharkhand,
India
Girijesh Prasad
Intelligent Systems Research Centre (ISRC)
University of Ulster, Magee Campus,
Derry, N. Ireland,
United Kingdom
1
Outline
I. INTRODUCTION
II. PROPOSED VIRTUAL KEYBOARD
A. The working of the VK
B. Modes of input accesss
C. The layout design
III.EVALUATION AND TESTING
A. Evaluation of the layout design
B. User trials
1. Types of errors
2. Testing methodology
IV. RESULTS AND DISCUSSIONS
V. CONCLUSION
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
2
Introduction
•Persons with speech and motor disorders face
problems in expressing themselves in an easy and
intelligible way.
•Computer based augmentative and alternate
communication (AAC) systems are developed to
assist them.
•A Virtual Keyboard (VK), also called as On Screen
Assistive Keyboard is a commonly used AAC system.
•A VK is characterized by the input modalities and
the layout.
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
3
Introduction
…Continued.
•Scanning and access switches require lesser motor
control.
• Brain-computer interface (BCI) translates the intent
of a subject measured from brain signals directly into
control command.
•Eye tracking which uses fixed gaze or blinking to
activate the switch or triggers.
•“Hex-o-Spell” [1] virtual keyboard is an EEG based
Brain Computer Interface that uses machine learning
techniques to identify brain signals.
•Dasher[2] is an information-efficient text-entry
interface, driven by natural continuous pointing
gestures.
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
4
The Proposed Virtual Keyboard: The Working
Fig. 1(a): Virtual Keyboard: pointer pointing at the sub circle
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
5
The Proposed Virtual Keyboard: The Working
Fig. 1(b) Expanded Sub Circle: pointer points at the character F
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
6
The Proposed Virtual Keyboard: Modes of Input
Any one or the combination of three incorporated
technologies can be used for access.
•The Brain Computer Interfacing (BCI) based on EEG
uses ‘vivid imagination’ of a motor activity as a
trigger.
•The Eye Tracker technology uses prolonged gaze
and/or eye blink as a trigger.
•Access switches use any active body part such as
hand, foot, mouth or head as a trigger
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
7
The Proposed Virtual Keyboard: Layout Design
Different alphabets have different frequency of
occurrence in English Text. [3] [4]
TABLE I. RELATIVE FREQUENCY OF THE ALPHABETS IN ENGLISH TEXT
Character
A
B
C
D
E
F
G
H
I
J
K
L
M
Vijit Prabhu & Girijesh Prasad
Frequency
8.167%
1.492%
2.782%
4.253%
12.702%
2.228%
2.015%
6.094%
6.966%
0.153%
0.772%
4.025%
2.406%
Character
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
Frequency
6.749%
7.507%
1.929%
0.095%
5.987%
6.327%
9.056%
2.758%
0.978%
2.360%
0.150%
1.974%
0.074%
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
8
The Proposed Virtual Keyboard: Layout Design
•Different positions of characters in the layout ,
require different amount of access activity.
•Design Principle- higher the frequency of occurrence
of character, lower should be the amount of activity
required to access it.
•Clashes in positioning of character were resolved
based on the probability of the character being the
first letter of the word. [5]
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
9
The Proposed Virtual Keyboard: Layout Design
More probable of being the first letter of word assigned the lower sub-circle.
TABLE II: RELATIVE FREQUENCIES OF THE FIRST LETTERS OF A WORD IN
THE ENGLISH LANGUAGE
Character
A
B
C
D
E
F
G
H
I
J
K
L
M
Vijit Prabhu & Girijesh Prasad
Frequency
11.602%
4.702%
3.511%
2.670%
2.000%
3.779%
1.950%
7.232%
6.286%
0.631%
0.690%
2.705%
4.374%
Character
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
Frequency
2.365%
6.264%
2.545%
0.173%
1.653%
7.755%
16.671%
1.487%
0.619%
6.661%
0.005%
1.620%
0.050%
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
10
The Proposed Virtual Keyboard: Layout Design
Position
Access
Activity
Layout
Alphabetical
Position
Access
Activity
Layout
Alphabetical
1,1
2
space
a
3,1
4
i
o
1,2
3
e
b
3,2
5
r
p
1,3
4
a
c
3,3
6
u
q
1,4
5
n
d
3,4
7
y
r
1,5
6
d
e
3,5
8
b
s
1,6
7
m
f
3,6
9
x
t
1,7
8
g
g
3,7
10
j
u
2,1
3
t
h
4,1
5
h
v
2,2
4
o
i
4,2
6
c
w
2,3
5
s
j
4,3
7
w
x
2,4
6
l
k
4,4
8
v
y
2,5
7
f
l
4,5
9
q
z
2,6
8
p
m
4,6
10
z
space
2,7
9
k
n
4,7
11
del
del
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
11
Evaluation & Testing: Evaluation of Layout Design
Given a random variable x, with the probability of
occurrence p(x), the expected (mean) value of x:
E(x) = Σi xi p(xi)
x -‘total activity’ associated with a character
p(x) -probability of the occurrence of the character in
the text based on their relative frequencies
Alphabetical layout, the expected amount of activity
required is computed to be 6.47.
For the proposed layout, the same activity drops
down 4.55.
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
12
Evaluation & Testing: User Trials
Types of Errors :
Wrong Character Selection (WCS): user types a
wrong character-needs to use backspace to remove
the incorrect character -then type in the correct
character.
Wrong Sub Circle Selection (WSCS): user selects the
wrong sub-circle-needs to use the ‘Go Back’ option
provided in the sub circle
Failed Selection (FS): user realizes that the pointer
(arrow) has already moved beyond the correct sub
circle /character
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
13
Evaluation & Testing: User Trials
Testing Methodology
•Users given the layout design in advance in print
•Explained the working of the VK
•One sentence to type using the soft switch at four
different scan periods 2100ms, 1800ms, 1500ms and
1200ms.
•Just soft switch to select button as the pointer
rotated and no prediction help was allowed.
•Only error free words and sentences with full-stops
were considered for evaluation
•At the end of each typing session an informal chat
THE QUICK BROWN FOX JUMPS OVER A LAZY
DOG.
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
14
Results and Discussions:
TABLE IV: RESULT OF USER TRIALS.
Subject
S1
S2
S3
S4
S5
S6
S7
Scan time 2100ms
Wrong Sub Circle
0
0
0
0
1
0
0
Wrong Character
1
5
0
0
2
2
0
Failed Selection
Total Errors
Total Time Taken
(min:s)
0
1
10:38
6
11
14:58
1
1
10:19
0
0
10:04
1
4
12:14
1
3
11:36
0
0
10:04
Wrong Sub Circle
0
0
1
0
0
0
0
Wrong Character
1
0
0
1
0
0
0
Failed Selection
Total Errors
Total Time Taken
(min:s)
0
1
09:18
2
2
09:51
0
1
08:57
0
1
09:15
0
0
08:49
1
1
08:58
1
1
09:04
Wrong Sub Circle
0
4
0
1
5
0
1
Wrong Character
1
0
0
1
0
0
0
Failed Selection
Total Errors
Total Time Taken
(min:s)
0
1
07:48
0
4
07:56
0
0
07:22
0
2
8:08
0
5
09:53
0
0
07:22
0
0
07:47
Wrong Sub Circle
0
1
0
1
0
0
0
Wrong Character
0
1
0
1
1
0
0
Failed Selection
Total Errors
Total Time Taken
(min:s)
2
2
06:30
2
4
07:48
0
0
06:01
0
2
06:39
0
1
06:25
0
0
06:02
0
0
06:08
Scan time 1800ms
Scan time 1500ms
Scan time 1200ms
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
15
Conclusion:
•The VK developed has multiple modes of access,
hence people suffering from different levels of
disability may be able to use one or more of the access
methods
•Evaluation of the layout design of the proposed VK
clearly establishes the superior performance
measured in terms of average per character activity
required for composing text
•The user trials not only helped prove the robustness
of the VK but also helped in performance evaluation
of the layout
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
16
Conclusion:
Continued
•The various scan periods were also evaluated and an
attempt was made to determine the most suitable
scan times.
•User trials with the actual intended users making use
of all three modalities is an utmost requirement.
•Comparative study with other existing VK like the
flat VK will help in better evaluation of design
•Work is ongoing to combine BCI and eye-tracker
modalities together to create a hybrid modality and
evaluate its performance.
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
17
References
•[1] Blankertz et al., The Berlin Brain-Computer Interface Presents The
Novel Mental Typewriter Hex-O-Spell, In: Proc. of the 3rd International
Brain-Computer Interface Workshop and Training Course 2006,
September 2 1-24 2006, Verlag der Technischen Universität Graz..
•[2] Dasher-Information-Efficient Text Entry-Hanna Wallach,
University
of
Cambridge/University
of
Pennsylvania.
http://www.mendeley.com/research/dasher-an-efficient-keyboardalternative/
•[3] Lewand, Robert (2000). Cryptological Mathematics. The
Mathematical Association of America. p. 36. ISBN 978-0883857199
•[4] Lee, E. Stewart; Essays about Computer Security; University of
Cambridge Computer Laboratory, p. 181
•[5] Letter Frequency, Wikipedia,
http://en.wikipedia.org/wiki/Letter_frequency
Vijit Prabhu & Girijesh Prasad
Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities
18
19