Shift:

A Technique for Operating Pen-Based Interfaces Using Touch

Daniel Vogel University of Toronto Patrick Baudisch Microsoft Research

Motivation

Motivation

Motivation

Small Targets

Small Targets

Advantages of the Pen Pen Finger

unique contact point remove hand from screen ambiguous contact point finger occludes target

Possible Solutions …

Offset Cursor

(Potter et al. 1988)

Pen Offset Cursor

Offset Cursor

(Potter et al. 1988)

Offset Cursor

(Potter et al. 1988)

Disadvantages

1. no visual feedback until contact, need to estimate offset 2. makes some display areas inaccessible 3. unexpected offset affects walk-up-and-use scenarios

Shift

Benefit 1: Aim for the Target

Users expect to click on the target itself.

 allows switching between pen and touch  walk-up and use with kiosk

Benefit 1: Aim for the Target

Users expect to click on the target itself.

 allows switching between pen and touch  walk-up and use with kiosk

Benefit 2: All Areas Accessible

Callout is relative to finger, so it can go anywhere.

 no edge problems

Benefit 2: All Areas Accessible

Callout is relative to finger, so it can go anywhere.

 no edge problems

Benefit 3: Fast For Large Targets

Callout only used when necessary  same speed as unaided touch screen for large targets

Design Iterations

Model

Performance Model

First Prototype

Revision and Visuals

Escalation

Based on selection ambiguity with fallback to hesitation.

S F 1.2

S T 0 -25 S T 0 S F

S T

 = Target Size,

S T << S F



S F

= Finger occlusion threshold high selection ambiguity  no delay 

S T >> S F

 no selection ambiguity  long delay 

S T ≈ S F

 “ambiguous selection ambiguity”  short delay 25

Escalation

Based on selection ambiguity with fallback to hesitation.

S F 1.2

S T 0 -25 S T 0 S F

S T

 = Target Size,

S T << S F



S F

= Finger occlusion threshold high selection ambiguity  no delay 

S T >> S F

 no selection ambiguity  long delay 

S T ≈ S F

 “ambiguous selection ambiguity”  short delay 25

Perceived Input Point Correction

Users expect selection point to be higher.

input point

user’s view

input point

hardware’s view

Iterative estimate for a correction vector

V

using difference between initial contact point P 1 and final lift off point P 2 

V t+1 = V t + w(P 2 - P 1 )

Experiment

(a) (b) (c)

Experiment

(a) (b) (c)

Experimental Design

3 techniques (Shift, Touch, Offset Cursor) 2 finger styles (nail, tip) x 3 blocks x 6 target sizes (6, 12, 18, 24, 48, 96px) 4 target directions (NW, NE, SW, SE) x x

Error

100 80 60 40 20 0 Tip Nail Touch Screen Offset Cursor Shift 6 12 18 24

Square Target Size (px)

48 96

Time

2.5

2.0

1.5

1.0

0.5

0.0

6

Selection Time: Fingernail

Touch Screen Offset Cursor Shift 12 18 24 Square Target Size (px) 48 96

Time

2.5

2.0

1.5

1.0

0.5

0.0

6

Selection Time: Finger Tip

Touch Screen Offset Cursor Shift 12 18 24 Square Target Size (px) 48 96

Corrective Movements

(a)

FingerTip

corrective movements by target size 20 Touch Offset 10 0 6 12 18 24

Square Target Size (px)

48 (b)

FingerTip

net correction distance by target size Shift 96

Corrective Movements

(a)

FingerTip

corrective movements by target size 20 Touch Offset 10 0 6 12 18 24

Square Target Size (px)

48 (b)

FingerTip

net correction distance by target size Shift 96

Discussion

Able to select small targets reliably (like Offset Cursor) Fast for large targets (like unaided Touch Screen) However, biggest benefit may be simpler mental model:  “Just aim for the target”

High Accuracy Enhancements

Added Zooming and CD-Ratio Manipulation

High Accuracy Enhancements

Added Zooming and CD-Ratio Manipulation

Thanks to members of the ASI and VIBE groups at MSR, special thanks to Raman Sarin, Ed Cutrell, and David Thiel.

Appendix

Estimating Occlusion Threshold

Don’t know actual finger size, so estimate it over time    when

S T ≈ S F

 short delay … means user can choose to use escalation by hesitating or not if they hesitate and use escalation  if they just click without escalation  make

S F

make

S F

larger smaller S F S T

Prototypes