Transcript Cross-modal integration: Synchronization of auditory and

Cross-modal integration:
Synchronization of auditory and visual
components in simple and complex media
Dr. Scott D. Lipscomb
Institute for Music Research
University of Texas at San Antonio
March 18, 1999
Acoustical Society of America Berlin '99
1
Film Music Literature
• past studies deal almost exclusively with the
referential aspect of musical sound
– “cognitive congruency” (Marshall & Cohen, 1988)
– selected others: Tannenbaum (1956), Thayer &
Levenson (1984)
– special issue of Psychomusicology (vol. 13, 1994)
• vs. accent structure alignment
– i.e., how often important events in the music coincide
with important events in the visual scene
March 18, 1999
Acoustical Society of America Berlin '99
2
Film Music
Paradigm
Visual stimulus
Aural stimulusPerception
(Lipscomb & Kendall, 1995)
Perception
IMPLICIT
PROCESSES
Association
Judgment
NO
YES
Accent Structure
Relationship
NO
YES
No Shift of
Attentional
Focus
March 18, 1999
Acoustical Society of America Berlin '99
Shift of
Attentional
Focus
3
Alignment Conditions
after Yeston (1975)
Consonant
a)
Out-of-phase
b)
Dissonant
c)
March 18, 1999
Acoustical Society of America Berlin '99
4
Subject Ratings
• synchronization: “… how often important
events in the music coincide with important
events in the visual scene”
• effectiveness: “… simply concerns [the
subject’s] subjective evaluation of how well
the two go together”
March 18, 1999
Acoustical Society of America Berlin '99
5
Results of Previous Study (1997)
presented at Penn State ASA Conference
Synchronization Ratings--All Experiments
Exp 1
Effectiveness Ratings--All Experiments
Exp 1
Exp 2
Exp 2
100.00
Exp 3
90.00
90.00
80.00
80.00
70.00
Mean Scores
Mean Scores
100.00
60.00
50.00
40.00
Exp 3-Untrained
Exp 3-Mod &
Trained
70.00
60.00
50.00
40.00
30.00
30.00
20.00
Consonant
Out-of-Phase
Dissonant
20.00
Consonant
Alignm ent Condition
March 18, 1999
Out-of-Phase
Dissonant
Alignm ent Condition
Acoustical Society of America Berlin '99
6
Problems with Dissonant Stimuli
Solution Found
March 18, 1999
Acoustical Society of America Berlin '99
7
Revised Stimulus Preparation
• A-V alignment conditions were created
using Media 100 software on a Macintosh
G3 computer
• sound files were manipulated in Sonic
Foundry’s Sound Forge 4.0, using the Time
Compress/Expand Sonic Foundry plug-in
• completed A-V composites were recorded
directly from Media 100 onto VHS tape
March 18, 1999
Acoustical Society of America Berlin '99
8
Consonant Alignment Condition
aligned as intended by the composer
March 18, 1999
Acoustical Society of America Berlin '99
9
Out-of-phase Alignment Condition
alignment conditions based on Preliminary Study
A-V Scene
misalignment of audio
"Dots"
-100 ms
"Canon"
532 ms
"Synchromy"
532 ms
"Portrait of Elizabeth"
672 ms
"Flashback"
-890 ms
"Reunion"
425 ms
March 18, 1999
Acoustical Society of America Berlin '99
10
Dissonant Alignment Conditions
• audio tracks for the McLaren animations
were “time expanded” by 115%
• audio tracks for the “Obsession” excerpts
were “time expanded” by 110%
– since these excerpts were longer, the gradual
misalignment could occur at a slower pace
March 18, 1999
Acoustical Society of America Berlin '99
11
Video excerpts
March 18, 1999
Acoustical Society of America Berlin '99
12
Subject Info & Method
• N = 20 UTSA students taking music classes
– stimuli presented in groups of 4 to 6
– VAME ratings were provided on a continuous line response
anchored by either “not synchronized-synchronized” or
“ineffective-effective”
synchronized
not synchronized
– each subject was assigned to one of three random stimulus
presentation orders
– response forms were generated so that order of VAME responses
was also randomized
– stimuli were presented to subjects using a Samsung VR 5855 video
cassette recorder and an RCA F27676BC 24” television
March 18, 1999
Acoustical Society of America Berlin '99
13
Statistical Analysis
(MANOVA)
• repeated measures fully-factorial analysis of variance
– repeated measures: two VAME ratings for each of 18 A-V
combinations
• no significant between-groups variation
– musical training: p = 0.512; f(2,17) = 0.696
– gender: p = 0.508; f(1, 18) = 0.457
• within-groups, ratings were significantly different
depending upon the A-V combinations
– p < .0005; f(17) = 11.582
March 18, 1999
Acoustical Society of America Berlin '99
14
Experimental Results
Mean Rating
McLaren Animations
100
90
80
70
60
50
40
30
20
Synchronization
Effectiveness
p _x
n
y _p
p _x
x
ts
o
_
_
_
m
o
n
n
n
y
y
ts o ts
D
o
o hr o m
a
o
n
n
C
o
om
D
D
a
a
c
r
r
n ch
h
C
C
y
c
n
S yn
S
Sy
A-V Stimuli
March 18, 1999
Acoustical Society of America Berlin '99
15
Experimental Results
Mean Rating
"Obsession" Excerpts
100
90
80
70
60
50
40
30
20
Synchronization
Effectiveness
it
p
x
a
ck k_ p k_ x ion n _p n _x
r
t_
t_
i
i
a
tr
a tra
b
c ac
un nio nio
r
o
h
a
t
e
r
r
P
s
b
b
u eu
R
e
Po Po Fla ash ash
R
R
Fl
Fl
A-V Stimuli
March 18, 1999
Acoustical Society of America Berlin '99
16
Experimental Results
McLaren Animations
"Obsession"
Subject Ratings Collapsed
Subject Ratings Collapsed
100.00
100.00
90.00
90.00
80.00
80.00
Mean Rating
Mean Rating
Subject Ratings Collapsed (n=3)
70.00
60.00
50.00
70.00
60.00
50.00
40.00
40.00
30.00
30.00
20.00
20.00
Consonant
Out-of-phase
Dissonant
Out-of-phase
Dissonant
Alignment Condition
Alignment Condition
March 18, 1999
Consonant
Acoustical Society of America Berlin '99
17
Contact Info
Dr. Scott D. Lipscomb
UTSA Division of Music
6900 N. Loop 1604 West
San Antonio, TX 78249
(210) 458-4354
(210) 458-4381 FAX
[email protected]
http://music.utsa.edu/~lipscomb
March 18, 1999
Acoustical Society of America Berlin '99
18