Transcript No Slide Title

Relative influence of
sensory cues in a
multi-modal virtual
SIGHT
environment
(visual)
Claudia Hendrix
Pi-Ming Cheng
William Durfee
Department of Mechanical Engineering
SOUND
(aural)
TOUCH
(haptic)
Take-home messages





Virtual environments can do a good job
emulating soft materials
Haptic cues are important for identifying
materials
Adding visual or sound cues improves ability to
discriminate among materials
Of haptic, visual and sound, sound least
important
Virtual prototyping using sight, sound and touch
is feasible, but effects of sensory interaction are
complex and vary from person to person
Some prior multi-modal VR work

Marks [1978]: Reviews psychophysics of sensory interaction

Richard and Coiffet [1995]: Adding substitute sensory haptic
feedback improves performance on grasping and place tasks

Hendrix and Barfield [1995]: Adding synthesized sound sources
to a virtual world increases sense of “presence”…..but not sense
of “realism”

Srinivasan et al [1996] and Durfee et al [1997]: Visual displays
influence perception of haptic stiffness

DiFranco et al [1997]: Audio cues influence perception of haptic
stiffness
Virtual product prototyping


Apply virtual reality technology to create product prototypes
Move beyond CAD-based visual rendering
See
Hear
Touch
?
?
?
?
Panel controls: simplified
paradigm for research
•Fixed in space, single d-o-f, low-force, simple graphics
•Sufficiently complex to enable exploration of research questions
With panel controls paradigm...
“Head-related
transfer function”
system
Visual
Aural
Haptic
Experiment system
to SGI
camera
haptic display
visual display
motion, force, torque
motor
PC
SGI
speaker
servoamp
aural display
Ethernet
green
screen
Haptic display
Haptic display control
command out
servoamp
motor
DAC
DSP
board
Pentium 90MHz
host computer
shared memory
encoder
encoder
interface
accelerometer
ADC
ADC
conditioning
electronics
torque
transducer
Materials modeled as spring-damper, PD impedance controller, 60 u-sec update rate
Aural display



Record sound clips for
different strike velocities
Synchronized playback
(Synthesized sound based
on physical models too
complex…for now)
Match virtual to one of 3 real

EXPERIMENT #1
“BEST” MATCH
Material properties:
V = visual
H = haptic
A = aural
– VVR = V1
– HVR = H1
– AVR = A1

VVR
HVR
AVR
EXPERIMENT #2
SENSORY CONFLICT
– VVR = V2
– HVR = H1
– AVR = A3
V1
H1
A1
metal
VIRTUAL
V2
H2
A2
V3
H3
A3
hard foam
soft foam
REAL
Experiment setup
sp eakers
green screen
motor
tes t material
T V i t or
n
mo
prob e
referen ce materials
camera
subj ect
to SGI
Metal
Metal
35
Hard foam
7.2
Hard foam
Soft foam
3.2
Soft foam
Stiffness (Nm/rad)
Ability of haptic
interface to mimic
material stiffness
REAL
VIRTUAL
Protocol, Expt #1 (“best” VR)
Test conditions
TEST
MATERIAL
12 subjects
MATERIAL TYPE
Metal
Hard
foam
Soft
foam
Virtual
X
X
X
Real
X
X
X
PROCEDURE:
Probe test and reference
materials, then answer:
1. Which material best
matches test?
2. What is the quality of the
match? (1-5)
6 test conditions, 4 replications =
24 trials/session
Matching results
Test material was real
When test
was...
Test material was virtual
Subjects matched to...
When test
was...
METAL
M 100 (4.6)
HF
0
SF
0
HARD
FOAM
M
HF
SF
SOFT
FOAM
M
HF
SF
Subjects matched to...
METAL
M
HF
SF
60 (3.5)
40 (3.0)
0
0
85 (4.3)
15 (3.7)
HARD
FOAM
M
HF
SF
0
83 (3.9)
17 (2.7)
0
2 (3.0)
98 (4.5)
SOFT
FOAM
M
HF
SF
0
2 (2.0)
98 (4.2)
Material, matches (%), quality rating (1-5)
Quality ratings
5
Quality rating
4
Real correct
3
Virtual correct
Real incorrect
2
Virtual incorrect
1
0
Metal
Hard foam
Soft foam
Protocol, Expt #2 (sensory conflict)
TEST MATERIAL
Aural
Haptic
soft
foam
Vsoft foam
Hmetal
Asoft foam
hard
foam
metal
27 test conditions
(always virtual), 3
replications = 81
trials per session
soft
foam
metal
hard
foam
soft
foam
metal
hard
foam
Visual
Looks and sounds
like soft foam, feels
like metal
Matching with sensory conflict
HAPTIC
1-WAY
TEST
2-WAY
VISUAL
MATCH
TEST
M
M
HF
SF
11
79
10
HF
M
HF
SF
SF
M
HF
SF
MATCH
TEST
M
M
HF
SF
8
26
65
14
58
28
HF
M
HF
SF
13
11
76
SF
M
HF
SF
HAPTIC + AURAL
TEST
AURAL
MATCH
M
M
HF
SF
31
62
7
HF
M
HF
SF
4
47
49
SF
M
HF
SF
4
13
83
M
M
HF
SF
18
43
39
14
50
36
HF
M
HF
SF
10
40
50
15
72
13
SF
M
HF
SF
10
65
25
HAPTIC + VISUAL
TEST
MATCH
MATCH
AURAL + VISUAL
TEST
M
M
HF
SF
38
61
1
HF
M
HF
SF
8
76
15
SF
M
HF
SF
7
7
86
MATCH
M
M
HF
SF
22
32
46
HF
M
HF
SF
11
50
39
SF
M
HF
SF
8
68
24
3-way match
HAPTIC + VISUAL + AURAL
TEST
MATCH
M
M
HF
SF
47
44
8
HF
M
HF
SF
0
78
22
SF
M
HF
SF
0
3
97
Quality when matched
5
Quality rating
4
3
H
2
V
A
1
H
+
A
H
+
V
0
one
two
Modalities matched
A
+
V
Cues used by the subjects
“What strategy did you use to make your matches?”
Number of
Subjects
8
1
1
1
1
Most important
characteristics
Haptic then visual
Visual then sound
Visual then haptic
Haptic alone
Haptic then sound
What we learned




Virtual environment does well when emulating soft
materials
Haptic cues are important for matching
Adding visual or sound cues to haptics improves
perceived quality of match
Subjects tended not to use sound cues for matching
Future work:
1. Use 2 out of 3 cues (V,H,A) to determine relative influence
2. Mechanical design task using virtual and real prototypes
Work supported by:
National Science Foundation (NSF/MIP-9420394, and a graduate training grant,
Cognitive Sciences Center, University of Minnesota, DMC8857851)