Transcript quill: A UI Design Case Study A. Chris Long Postdoctoral Research Fellow

quill: A UI Design Case Study
A. Chris Long
Postdoctoral Research Fellow
HCI Institute
Administrivia
 Turn in Homework #1
 Homework #2 assignment on course website
2
Outline
 Gestures
 quill Design
 quill Architecture & Implementation
 Object-Oriented Design
 Long-running Tasks and Interfaces
 Recognition-based UIs
 How Gesture Recognition Works
 More quill Implementation
3
Gestures
 Mark that invokes a command
4
Gestures
 Mark that invokes a command
delete me
5
Gestures
 Mark that invokes a command
delete me
 Benefits
* Fast
* Save screen space
* Avoid “aerobic interface”
* Specify operation and operand(s)
simultaneously
6
Problems With Gestures
 For users
* Computer often misrecognizes
gestures
* Difficult to learn and remember
gestures
 For designers
* Gestures are hard to design
7
quill’s Place in the World
 Who: Designer of
Design
Test
Prototype
pen-based UI
 When: During design
phase
 What: Designing
gestures
 Why: To design
better gestures
8
Another View of quill
User Interface
Application “back end”
OS
Pen Input
Display
9
Designing quill
 Design low-fi prototype
 Pilot test low-fi
 Redesign
 Implement Java version
10
Low-fi Prototype
11
Current Version
 Tree view
 Expanded
view
 Suggestion
for
improvement
12
Recognition Problem
13
Gestures, Packages, and Groups,
Oh My!
Gesture
package
MyApplication
Gesture
group
Gesture
category
Gesture
Edit
cut
copy
View
…
…
…
14
quill Architecture
 JDK Core
 Widgets
 Gesture Objects
* Gesture
* Gesture Category
* …
 Recognizer
 Gesture Analyses
15
quill Architecture
Widgets
Gesture Analyses
Gesture &
Recognition
Objects
Swing
Core JDK
16
Implementation Overview
Clipboard
Desktop
windows
Tree display
Main window
Notice
management
 Java/Swing
 93 classes, 26,000 lines
17
Object Oriented Design
 Gesture Objects
*
*
*
*
Gesture
GestureCategory
GestureGroup
GesturePackage
 Gesture Display Widgets
*
*
*
*
*
*
GestureDisplay
GestureInteractor
GestureCategoryDisplay
GestureGroupDisplay
GesturePackageDisplay
GestureCategoryThumbnail
Display
18
OO Design
GestureObject
DefaultGestureObject
GestureContainer
AbstractGestureContainer
GestureCategory
GestureGroup
GesturePackage
19
Analyzing Gestures
 Problem: Analysis may take a long time
 Solutions?
20
Analyzing Gestures
 Problem: Analysis may take a long time
 Strategies for analyzing gestures
* Run analysis in foreground, disable all actions
* Analyze in background
- Disable all conflicting actions
- Allow anything, cancel analyses
- Allow anything
21
Multithreading
 Benefits
* ?
 Drawbacks
* ?
22
Multithreading
 Benefits
* Computation in background
- Can do very complex tasks
* User can perform any action,
any time
 Drawbacks
* Must synchronize data
across threads
* Deadlock
* Race conditions
23
Multiple Threads in Java
 Thread class
* Start separate thread that runs any method
 synchronized keyword
* Prevents a method from being called or object
from being accessed by more than one thread
at once
 SwingWorker class
* Easy to run background thread that produces a
result
24
Recognition-Based UIs
 Examples
* Pen gesture
* Handwriting
* Speech
 Advantages
* ?
 Disadvantages
* ?
25
Recognition-Based UIs
 Advantages
* Natural input
* Informal
* Multiple interpretations
 Disadvantages
* No clear separation of data and commands
* Ambiguous commands
* Individual differences
* Multiple interpretations
26
Dealing with Recognition
 Confirm dangerous operations
 Multiple modes may help disambiguate
* Speech + pen, for example
 Provide undo/redo
27
How Does Recognition Work?
 Training
* Measure geometric
features of gestures
* Compute average
feature values for each
type of gesture
* Compute how important
each feature is
 Recognition (a.k.a.
Classification)
* Measure geometric
features of unknown
gesture
* Compare with features
of known gestures
* Pick closest known
gesture
28
Rubine’s Feature Set
 Initial angle
 Length of bounding





box diagonal
Angle of bounding box
diagonal
Distance between
first and last points
Angle between first
and last points
Length
Total angle traversed
29
Rubine’s Feature Set (cont.)
 Sum of absolute value
of angle at each point
 Sum of squared angle
at each point
(sharpness)
 Square of maximum
speed
 Duration
30
Weakness of Feature-based
Recognition
 Feature set has to be right
31
Feature-based Training
 Mean feature vector
1
f cˆi 
Ecˆ
Ecˆ 1
f
e 0
cˆei
0i F
 Per-class covariance matrix
Ecˆ 1
cˆij   ( f cˆei  fcˆi )( f cˆej  fcˆj )
e 0
32
Training (cont.)
 Average covariance matrix
 ij 
 Weights estimation
C 1
 cˆij
c 0
cˆ
C 1
E
1
 C   Ecˆ
c 0
F
wcˆj   ( 1 ) ij f cˆi 1  j  F
i 1
33
Feature-based Classification
 For all gesture classes c, compute
F
vcˆ  wcˆ 0   wcˆi f i
0cC
i 1
 Return class with max vc
34
quill Implementation Issues
(cont’d)
 Suggestions (cont’d)
* When do they disappear?
- User says to ignore
- No longer apply
* When are two notices the same?
-
At time A, gestures X and Y are too similar
At time B, something happens and they aren’t similar
At time C, they are similar again
Display notice again?
35
Implementation Issues (cont’d)
 Annotations
* Special purpose vs.
generic
* How to compose?
 Gesture display
* Desktop windows
- Tiled
- Standard MDI
* Display factory
36
Implementation issues (cont’d)
 Generic properties
void GestureObject.setProperty(String
name, Object value)
Object GestureObject.getProperty(String
name)
* What happens to properties like
“isMisrecognized” when copy & paste?
* Persistent vs. transient
37
Implementation issues (cont’d)
 Generic properties
void GestureObject.setProperty(String
name, Object value)
Object GestureObject.getProperty(String
name)
* What happens to properties like
“isMisrecognized” when copy & paste?
* Persistent vs. transient
38
Implementation Issues (cont’d)
 Menus: enable/disable
items, and dispatching
commands
* Context-dependent
menu items
* Context-dependent
gesture drawing
* Solution: “Chain of
command” pattern that
follows selection
39
Questions?
 Other topics
* Grad school
* Berkeley (city or university)
* Speech UIs
* More on pen-based UIs
40
The End
Design Patterns: Elements of Reusable
Object-Oriented Software
by Erich Gamma, Richard Helm, Ralph
Johnson, and John Vlissides
http://www.cs.cmu.edu/~chrisl/
http://guir.berkeley.edu/projects/quill/
Extras
Criticisms
 Recognition is due to poor recognizers, which will
eventually be perfect
* Not for a long time, if ever
* Learning and memory predictions still useful
 Recognition will never be good enough
* Already good enough for many users
* Is improving over time
 Typing is faster and more accurate than writing
* Pens won’t replace keyboards, but are better in many
situations
43
Criticisms (cont.)
 Gestures will always be too hard to learn and
remember, and will never be useful enough
* Gestures very often used on paper in many domains
 Design tool doesn’t guarantee good gestures
* Will provide useful advice
* Will shorten design cycle
 There are only a small number of common gestures
* Application/domain-specific gestures
* Personal shorthand
44
Criticisms (cont.)
 Marking menus are better than free-form
gestures
* Can’t specify additional information with marking-menus
* Shapes are arbitrary
* All straight lines and 45/90º angles
45
Criticisms (cont.)
 What’s wrong with traditional GUI interaction
techniques?
* Pen  mouse
* Pen
- Finer control
- Has other dimensions (e.g., pressure, tilt)
* Mouse
- More buttons (sometimes)
46
Pen vs. Mouse
 Kato, et al
* Pen significantly faster & more accurate than mouse for
precise dragging
* Pen significantly faster for pointing
* Pen has directional dependencies; mouse doesn’t
 Mackenzie, et al
* Pen slightly faster for pointing
* Mouse slightly faster for dragging
47