Transcript Каналы мультимодальной коммуникации: от

Каналы мультимодальной
коммуникации:
относительный вклад в
понимание дискурса
«Мультимодальная коммуникация»
15 ноября 2013
А.А. Кибрик (ИЯз РАН и МГУ)
Н.Б. Молчанова (BearingPoint)
[email protected]
What is the contribution of
different communication channels?
 Traditional approach of mainstream linguistics:
the verbal channel is so central that prosody and the
visual channel are at best downgraded as
“paralinguistics”
 Applied psychology
It is often stated that (figures go back to Mehrabian 1971):
• body language conveys 55% of information
• prosody conveys 38% of information
• the verbal component conveys 7% of information
 Who is right?
2
Relative contribution of three
communication channels?
DISCOURSE
Vocal channels
Verbal channel
Visual channel
Prosodic channel
3
Experimental design
 Isolate the three communication channels
 Present a sample discourse in all possible



variants (23=8)
Present each of the eight variants to a group of
subjects
Assess the degree of understanding in each
case
Such assessment may lead to estimates of the
contributions of communication channels
4
Studies in this line of
research
 Èl’bert 2006, year paper
 Èl’bert 2007, diploma thesis
 Reinterpreted and refined in Kibrik and Èl’bert 2008
 Molchanova 2008, year paper
 Molchanova 2009, year paper
 Molchanova 2010, diploma thesis
 Reinterpreted and refined in Kibrik and Molchanova
2013
5
Èl’bert 2007, Kibrik and
Èl’bert 2008





Russian TV serial “Tajny sledstvija” – “Mysteries of the
investigation”
Context excerpt: 8 minutes
Experimental excerpt: 3 min. 20 sec.
 consisting of conversation alone, to ensure that we are testing
the understanding of discourse rather than of the film in general
Two vocal channels have been separated:
 Verbal: running subtitles
 Prosodic: superimposed filter creating the “behind a wall” effect
Participants:
 Native speakers of Russian
 Eight groups of 10 to 17 participants
6
Eight experimental groups
 Group 0: only the context excerpt
 Groups 1 (one communication channel)
 Verbal: subtitles, temporally aligned
 Prosodic: filtered sound
 Visual: video
 Groups 2 (two communication channels):
 Verbal + prosodic = original sound
 Verbal + visual: subtitles and video
 Prosodic + visual: filtered sound and video
 Group 3: original material
7
Verbal + visual
8
Procedure



The context and the experimental excerpts were shown
to a group of subjects on a large screen
Each subject answered 23 multiple-choice questions
concerned with the experimental excerpt alone
What Tamara Stepanovna offers Masha before the
beginning of the conversation:

a. to take off her coat

b. to have a cup of tea
  c. to have a seat

d. to have a drink

Percentage of correct answers is used as an assessment
of a subject’s degree of understanding
9
Results
 All three channels
100,00%
80,00%
60,00%
40,00%
20,00%
0,00%
V V V G
G
ro Ve Pro Vi s er er i s+ ro
r
up b
so ua +P +v pr up
a
r o is
o 3
di l
l
0
c
are substantially
informative
 Verbal > visual >
prosodic
 Integration of visual
and prosodic
channels is difficult
10
Molchanova 2010
Kibrik and Molchanova 2013
 Methodological issues
 The following aspects of the prior study
have been changed (improved)




Stimulus material
Methods of isolating the channels
Questionnaire
Participants and interviewing procedure
11
Stimulus material: discourse
type
 Shortcomings of movies
 Plot facilitates guessing
 Possible familiarity with the movie
 Quasi-natural behavior of actors
 Solution: natural dialogue
 Guessing game
original.avi, 0:19 – 0:57
12
Stimulus material: speakers
 Shortcomings of the prior studies
 Same-sex speakers  indistinguishable in the
prosody-only version
 Solution:
 Different sexes: F0 range is different
13
Methods of isolating the channels:
Verbal channel
 Shortcomings of subtitles
 Subtitles belong to the visual mode
 Hard to read without punctuation
• Especially at the rate of speech
• And especially in the “verbal + visual” condition
 Solution: spoken prosody-free signal
 Each word in transcript is recorded
individually from the corresponding person
 All thus elicited words are glued together in
the right order
14
Visual + verbal
(the robot condition)
15
Verbal channel
 Remaining problem
 Unnatural input
• No reduction
• No intonation
• etc.
16
Methods of isolating the channels:
Prosodic channel
 Shortcomings of the prosodic material as used
in previous studies
 Excessive noise
 Solution:
 Loudness is decreased radically at all frequencies
except for the speaker’s average F0 frequency
 This has led to a more satisfactory “behind the wall”
(or “behind the glass”) effect
17
Visual + prosodic
(the mermaid condition)
18
Questionnaire
 Shortcomings of prior studies
 Èl’bert 2007: gap between Group 0 (38.3%) and
Group 3 (87.4%) is insufficient
 Solution
 Testing stage
• Identify trivial questions (high Group 0)
–5
• Identify unfortunate questions (low Group 3) –2
• 30  23
 Group 0: 34.5% correct answers
 Group 3: 88.0% correct answers
19
Participants and interviewing
procedure


Shortcomings of prior studies
 Uncontrolled social status and geographical origin of participants
 Multiple participants in one room may affect each other’s
performance
 Need for a big screen
Solutions
 Control for social status and geographical origin; homogeneous
group
 Comparable, independent, and comfortable conditions
• Detailed guidelines
 Remote implementation
• Stimulus materials at Youtube.com
• Questionnaire at Googledocs
20
Kibrik and Molchanova 2013:
Results
100%

90%
80%
70%
60%
50%
Each individual channel is
substantially informative and
prevails over the null condition
(34.5%)
F-test: verbal and visual: p<0.05,
prosodic: p=0.127
40%
30%

20%
10%
0%
Null
Prosodic
Verbal+prosodic
Visual+prosodic
Verbal
Visual
Verbal+visual
Original
Verbal (58.8%) > visual
(52.2%) > prosodic (40.2%)
F-test: verbal > prosodic,
visual > prosodic: p<0.05,
verbal > visual: p=0.071
21
Kibrik and Molchanova 2013:
Results
100%

90%
80%
70%
60%
Two-channel conditions prevail over
the one-channel conditions much more
clearly than in the previous experiment
(verbal+prosodic – 73.5%,
verbal+visual – 88.2%)
F-test: all pairwise comparisons but
“visual+prosodic > visual”: p<0.05;
all two-channel conditions > all onechannel conditions: p<0.0001
50%
40%
30%

20%
10%
A dramatic dip in the visual+prosodic
condition is even clearer
F-test: significant difference from the two other
two-channel conditions, p<0.0001
0%
Null
Prosodic
Verbal+prosodic
Visual+prosodic
Verbal
Visual
Verbal+visual
Original
22
Kibrik and Èl’bert 2008 vs.
Kibrik and Molchanova 2013
100%
100,00%
90%
80,00%
80%
60,00%
70%
60%
50%
40,00%
40%
30%
20,00%
0,00%
20%
10%
V V V G
G
ro Ve Pr Vis er er is ro
up rb os u +P +v +p up
od al
r o is ro 3
0 al
ic
0%
Null
Prosodic
Verbal+prosodic
Visual+prosodic
General picture is remarkably similar
In the new study all effects are clearer
Verbal
Visual
Verbal+visual
Original
23
Normalized contribution of
three channels
 Suppose the three channels are
independent
 Sum up all percentages of individual
channel contributions and normalize to
100%
 Identify normalized contribution
24
Normalized contribution of
three channels
Summed percentages
Verbal
Normalized
Prosodic
contributions
Visual
Kibrik and Èl’bert 2008
Kibrik and Molchanova
2013
72+51+62=185
59+52+40=151
72%:1.85≈39% 59%:1.51≈39%
51%:1.85≈28% 46%:1.51≈30%
62%:1.85≈33% 49%:1.51≈32%
25
Gender differences
 Molchanova 2010: gender advantages
 Percentages of correct answers
Condition
Men
Women
Advantage
Verbal only
59.1
69.9
Women: +10.7
Visual +
prosodic
66.1
51.6
Men: +14.5
26
Conclusions

All communicatioin channels are highly significant
 the traditional linguistic viewpoint is incorrect

The verbal channel is the leading one
 the viewpoint popular in applied psychology is incorrect

Information from the prosodic and the visual channels is
primarily used through integration with the verbal
channel

Very similar results have been attained in different
studies, in spite of very different methodological details
27
Further questions
 Auditory or graphic presentation of the
“verbal alone” channel?
 Explore different discourse types, such as
monologic discourse
 …and: Other suggestions on this
approach?
28
Acknowledgements
 Olga Fedorova
 Anna Laurinavičiute
 Andriy Myachykov
 RGNF #11-04-00153
29
Thanks for your attention
visual channel
language
verbal channel
prosodic channel
30