Transcript Lectures

Multimedia Department
K.Marasek
05.07.2005
Spoken dialogs with computers
Krzysztof Marasek
Man-Machine communication: How can it work?
Multimedia Department
 Man-machine interaction by graphic
 Man-machine communication by speech
Lecture’s topic
 multi-modal man-machine communication
 is that all? - No: haptic, structure changes (Logitech mouse)
K.Marasek
05.07.2005
Multimedia Department
Lecture overview















K.Marasek
05.07.2005
encapsulated phonetics
sound propagation
acoustic properties of speech
basic signal forms and distinctive features of speech sounds
phonemes and allophones, grapheme-to-phoneme conversion
speech parameterization
speech data collection
speech modeling
language modeling and basics of natural language processing
speech recognition techniques- Viterbi decoding
model and pronunciation adaptation
speech synthesis methods and Text-To-Speech synthesis
speech understanding
dialog design
applications and challenges of speech technology
Multimedia Department
Architecture of spoken dialog
De Mori, 99
K.Marasek
05.07.2005
Multimedia Department
Human Language Technology worldwide
 USA
Oregon
Cornegie Mellon
MIT
 Japan
Kyoto
Tokyo
 Europe
Germany
France
Great Britain
Scandinavia
Italy
wild East
K.Marasek
05.07.2005
Multimedia Department
Wszystko jasne
 Bdanaia na pweynm anelgiksim uneruwstytecie wyzakały, że nie ma
znczeania, w jaikej kloejności napsziemy lietry wenątrz wryazu, blye
tlkyo pirwesza i otstaina lreita błyy na soiwch mijsecach. Rtszea mżoe
być dolnwoie poszamienina, a mmio to bedzięmy w stniae pczyrzetać
tkest bez wikszęego prleobmu. Diezje się tak dlteago, że nie cztaymy
kżdeaj z lteir odelndziie, ale wrayz jkao cłoaść.
Eric Campbell
K.Marasek
05.07.2005
Speech signal characterization
Multimedia Department
time signal
energy
spectrogram
pitch
duration
K.Marasek
05.07.2005
Man-Machine communication by speech
 Man-machine communication: exchange of information coded in a way
suitable for transmission through a physical medium
Multimedia Department
 Coding: the process of producing a representation of what has to be
communicated
 knowledge sources: constraints for building symbolic version of the
message and transmission through a physical channel
 decoding: models of KSs used by computer: deterministic but often
imprecise
Czesc!
coding
phrases
sentences
words
letters
decoding
K.Marasek
05.07.2005
Knowledge sources: acoustic models
 Coding of acoustic speech events:
 phones: representation of basic speech
coding
Czesc!
units
phrases
words
Multimedia Department
syllables
phones
 coding alphabet: e.g. IPA codes, but also other exist (SAMPA)
K.Marasek
05.07.2005
Speech Recognition: A simple decoder model
Information
Multimedia Department
source
W
Information
X
channel
 Modern systems are based on probabilistic scores for candidate
hypotheses
Utterance -> Speech signal -> PCM -> window ->coefficients -> X
 model of hypotheses scoring: let the sequence of acoustic observations
X=x1..xN be the output of the information channel. If the intention of the
speaker was the sequence of words W=W1…WK then X is coded version
of W
 The objective of recognition is to reconstruct W based on the
observation of X
K.Marasek
05.07.2005
Knowledge sources: acoustic models
 How phones can be modeled?
Multimedia Department
Let’ s assume that speech signal is parameterized as a
sequence of feature vectors computed for equally spaced
speech frames
parameters are statistically independent
Hidden Markov Models:
easy training
robust modeling
also other models are used: ANN, kernel methods, hybrid
systems, but HMM are currently state-of-the-art
K.Marasek
05.07.2005
Knowledge sources: HMMs
Multimedia Department
HMM
 Nodes of graph correspond to states of Markov chain, while directed arcs
correspond to allowed transtions aij
 A sequence of observation is regarded as an emission of the system which at
each time instant makes a transition form one to another node randomly chosen
according to a node-specific probability density and generates a random vector
according to arc-specific probability density. A number of states and set of arcs
is usually called model topology.
 In ASR it is common to have left-to-right topologies, in which aij=0 for j<I
 also usually first and last states are not-emitting, i.e. source and final states are
for setting initial and final probabilities,
Typical phone
model topology
K.Marasek
05.07.2005
Multimedia Department
Knowledge sources: HMM linking
K.Marasek
05.07.2005
Multimedia Department
Knowledge sources: language model
 Language model: set of constraints on the sequence of words acceptable for a
given language
 rules of generative grammar G to produce sentences of a language LG(G)
 4-tuple G  (s ,VT, ,where
VN , P) Vt is a set of all words of LG(G), VN is a set of nonterminal symbols representing abstractions of language components (ex.
syntax), s - category of all sentences in LG(G), P-set of rules a->b, where a is
sequnce of symbols with at least one belongs to VN and b  (VT u Vn)
 if a is only one symbol in VN then grammar G is context-free
 for natural languages it is impossible to conceive a grammar G capable of
generating all and only sentences of a language: no formal models of NL
 heuristic solution - stochastic finite state automata: over-generating
grammar for word pairs plus probabilities of generated sentences (bigrams) HMMs
 integrated network: automata for each word combined of lexical and acoustic
models describing pronunciation variants (phonemes) and distribution of
acoustic parameters of phonemes
K.Marasek
05.07.2005
Statistical modeling approach for ASR
computed as a distance to trained models
Multimedia Department
A priori probability of the acoustic
sequence A given a word sequence W
A priori probability of the word
string W
P ( A | W ) P (W )
P (W | A) 
P ( A)
Most probable word sequence W given
the acoustic input A
Traditional HMM
Output:
(W -word model)
K.Marasek
05.07.2005
Bayesian approach
A priori probability of the acoustic
sequence A
W* 
arg max
p( A,W )
W
Multimedia Department
What can be recognized?
K.Marasek
05.07.2005
(!ENTER{_SIL_}(
Kutno
|
Sopot
|
Pozna\\361
|
Lubin
|
£uk\\363w
|
aleja
Solidarnoœci
|
Beskidy
|
Rzesz\\363w
)(!ENTER{_SIL_}
I=172
I=173
I=174
J=0
J=1
J=2
J=3
J=4
J=5
J=6
J=7
J=8
J=9
J=10
Vocabulary
Kalisz k a l i S
Kamienna k a m j e n n a
Kaszuby k a S u b I
Katowice k a t o v i ts e
Kazimierz k a zi i m j e Z
Kielce k j e l ts e
Klakson k l a k s o n
Kolor k o l o r
Konopnickiej k o n o p ni i ts k j e j
Konstytucji k o n s t I t u ts j i
Koszalin k o S a l i n
Kościuszki k o si tsi u S k i
Krakowska k r a k o f s k a
Krakowsko k r a k o f s k o
Kraków k r a k u f
Krzyki k S I k i
Kujaw k u j a f
Dictionary
Kutno k u t n
W=Jana
W=Jura
W=Kazimierz
S=1
E=0
S=1
E=1
S=2
E=0
S=2
E=1
S=3
E=0
S=3
E=1
S=4
E=0
S=4
E=1
S=5
E=0
S=5
E=1
S=6
E=0
Lattice of
models
o
Multimedia Department
Speaker-independent, continuous-speech ASR now possible
 Digit recognition over the
telephone with word error rate of
0.3%
 Error rate cut in half every two
years for moderate vocabulary
tasks
 Error for spontaneous speech are
more than twice that of read
speech
 Conversational speech, involving
multiple speakers and poor
acoustic environment, remains a
challenge
 Tens of hours of training data to
port to a different domain
 Statistical modelling using
automatic training achieves
significant advances
MIT,2005
K.Marasek
05.07.2005
digits
2k spontaneous
64 k broadcast
100
10
1
0.1
1k read
20k read
10k conversational
Multimedia Department
Text-To-Speech
 Festival Speech Synthesis - steps to synthesize a sentence
 Text
Text preprocessing
 Token_POS
 Token
 POS
Word descriptions
 Word
 Phrasify
Prosody generation
 Pauses
 Intonation
 PostLex
 Duration
Acoustic output
 Int_Targets
 Wave_Synth
K.Marasek
05.07.2005
Speech synthesis
Multimedia Department
 Acoustic output:
pre-recorded speech
articulatory synthesis (formant synthesis) - tries to mimic
human voice generation
Frankfurt
concatenative synthesis - build utterances using stored
units
phonems
diphones: trasitions between two phonemes
Festival
unit selection: units of different length, context depended
selection (maximum length of natural speech sequence)
RealSpeak
ATR Japan
K.Marasek
05.07.2005
Multimedia Department
Example: Tokens, syllables and phones
K.Marasek
05.07.2005
Multimedia Department
Phrasing and Intonation
K.Marasek
05.07.2005
Multimedia Department
Dialog system
MIT 2005
K.Marasek
05.07.2005
Multimedia Department
Dialog systems
 Application dependent
(dialog structure and content)
 finite-state dialog system
(usually domain-dependent)
 chatter-bot systems
(domain-independent?)
 initiative possession
(machine, human, mixed)
 concept detection and spotting
(find important staff in the utterance
and conclude)
 concept and text generation
(generate context-dependent answer)
 Examples: IVR, reservation systems,
but sometimes still not perfect...
K.Marasek
05.07.2005
Multimedia Department
Going beyond…
 Add new dimensions to MMI (para- and extralinguistic
features)
avatars
personality of the dialog partners
speaker`s profile
reaction on speaker`s emotion and emotional synthesis

(rad)
K.Marasek
05.07.2005
Multimedia Department
Generation of word hypotheses: Speech
recognition
De Mori, 99
K.Marasek
05.07.2005
Multimedia Department
Part II
K.Marasek
05.07.2005
HUMAN TO HUMAN COMMUNICATION
Multimedia Department
Dialog architecture
De Mori, 99
K.Marasek
05.07.2005
Domains of verbal communication
Multimedia Department
SPEAKER
Utterance
forming
Articulation
SPEECH
PSYCHOLINGUSITCS
PHYSILOGY
Speech acoustics
LISTENER
Understanding
Hearing
Psychoacoustics
ACOUSTICS
Generation
of speech
K.Marasek
05.07.2005
Perception
of speech
What and how
Eyes – visual information
Multimedia Department
Ears – sound information
Nose –smell information
Tongue –taste information
Skin, muscles, touch receptors – touch and
proprio-kinesthetic information
Proprio-kinesthetic feedback mechanisms include awareness of the movement and location of the fingers in space,
internal monitoring of rhythm and rate, and a grip
K.Marasek
05.07.2005
Organs involved in the production of infomation by humans
Multimedia Department
Articulatory organs – sounds (speech)
 Movement and action organs –
gestures, writing, mimics, mechanic
actions etc.
K.Marasek
05.07.2005
Multimedia Department
Organs which may be involved in human-to-human
communication
Articulatory organs – hearing: speech
Articulatory organs –seeing : lips reading
Move and action organs –seeing: writing, gestures, sign
language, Braille’ writing
K.Marasek
05.07.2005
Levels of communication
Lingustic information
Multimedia Department
Articulatory infos
(phonetics)
Emotional information
Personal information
Information on organic speech disorders
Information on neurogenic speech disorders
Culture, habitats, social information
K.Marasek
05.07.2005
Language is a basis for all human to human communication
Multimedia Department
Language – a system of charaters and phonological, semantic and syntactic rules
which allow to combine this characters
Speech – spoken language
Writings – written language
Signs – sign language (polish, german, english etc.)
K.Marasek
05.07.2005
Sentence generation scheme
Multimedia Department
Syntax
component
K.Marasek
05.07.2005
Phrasing rules
Deep
Semantic
Lexicon
structure
component
Transformation
Surface
Phonological
rules
structure
component
Semantic
sentence
interpretation
Phonological
sentence
interpretation
Multimedia Department
Example of meaning changes by changed phonological structure of the word
Beep
Keep
Leep
Phonology, part of natural language processing,
describes phonemes and relation between phonemes
K.Marasek
05.07.2005
Multimedia Department
K.Marasek
05.07.2005
Articulation
Main articulatory elements of the vocal tract
Multimedia Department
nasal cavity
lips
tongue
glottis
K.Marasek
05.07.2005
Types of sounds
 Sound classification is based on manner and place of articulation – where
the consttriction in the vocal tract is and where the sound is generated
Multimedia Department
 Manner of articulation:
K.Marasek
05.07.2005
 Vowels
 Plosives /p/, /g/
Nasals - /m/, /n/
Taps or trills /r/
Fricatives -/s/, /f/, /v/
Approximants - /j/ /w/
 Place of articulation:
Bilabial
Labiodental
Dental, Alveolar, Postalveolar
Retroflex
Palatal
Velar
Uvular
Pharyngeal
Glottal
-- resonants
– obstruents
-- affricates
-- diphtongs
Multimedia Department
Typical vocal tract configurations
vowel
articulation
front -back
plosives
front (of the tongue)
K.Marasek
05.07.2005
high - low
articulation
back (of the tongue)
Multimedia Department
Typical vocal tract configurations
Front fricative
K.Marasek
05.07.2005
Front lateral
approximant
Multimedia Department
Comparison of airflow over nose and mouth
K.Marasek
05.07.2005
Multimedia Department
Tongue profiles
K.Marasek
05.07.2005
Multimedia Department
Tongue profiles
K.Marasek
05.07.2005
Multimedia Department
Tongue profiles
K.Marasek
05.07.2005
Multimedia Department
Tongue profiles
K.Marasek
05.07.2005
Multimedia Department
K.Marasek
05.07.2005
Phonetic transcription
Multimedia Department
IPA code - vowels
K.Marasek
05.07.2005
Multimedia Department
IPA consonants
K.Marasek
05.07.2005
SAMPA American English
Consonants: 24
Multimedia Department
Symbol Word
p
b
t
d
k
g
tS
dZ
f
v
T
D
s
z
S
Z
h
m
n
N
r
l
w
j
K.Marasek
05.07.2005
Transcription
pin
bin
tin
din
kin
give
chin
gin
fin
vim
thin
this
sin
zing
shin
measure
hit
mock
knock
thing
wrong
long
wasp
yacht
pIn
bIn
tIn
dIn
kIn
gIv
tSIn
dZIn
fIn
vIm
TIn
DIs
sIn
zIN
SIn
"mEZ@`
hIt
mAk
nAk
TIN
rON
lON
wAsp
jAt
Vowels:
I
E
{
A
V
U
i
e
u
o
O
aI
OI
aU
3`
@
@`
17
pit
pet
pat
pot
cut
put
ease
raise
lose
nose
cause
rise
noise
rouse
furs
allow
corner
pIt
pEt
p{t
pAt
kVt
pUt
iz
rez
luz
noz
kOz
raIz
nOIz
raUz
f3`z
@"laU
"kOrn@`
Multimedia Department
Transcription by hand
K.Marasek
05.07.2005
Grapheme to phoneme conversion
Multimedia Department
Task of converting word‘s given orthographic
representation into a sequence of phonemes
This is
Tis
K.Marasek
05.07.2005
IS
just
a
dZVst @
cute demo
kju:t di:m@U
Multimedia Department
Applications
 Speech synthesis (TTS)
 Pronunciation lookup
 Out of vocabulary words (OOV): online conversion
 Multilingual TTS
 Speech Recognition
 Pronunciation lookup
 Acqusition of new words
 Pronunciation dictionary compression
G2p was born with the idea in mind of a dynamic speech recognition
system: OOV words should be online converted.
Goal:
A high quality grapheme to phoneme conversion tool, which could handle
information for which a pronunciation transcription was not available (music
or movie titles, street names, new commands etc.).
g2p
K.Marasek
05.07.2005
Grapheme to phoneme conversion
Multimedia Department
 Is that difficult?
 An example from English
“cough”
“dough”
“plough”



[kQf]
[d@U]
[plaU]
 Best g2p methods result in 67-70% correct words for English [Damper
et. al. 1999]
 German: 75% [Müller et. al. 2000]
 Mistrust g2p results for these languages over 80%:
Wordforms with the same stem might be in training & testing data
 German with morphology: 94% [Möbius 1999]
 Results depend on language: 99% for Spanish, 99.5% for Polish
 Additional problems: foreign words, additional markers (e.g. word
stress), syllables recognition
K.Marasek
05.07.2005
Design Issues
Multimedia Department
Graphemic
representation
language
independent
language
dependent
Classificator
(C5.0)
Tree /
Rules
Phonemic
representation
Post-processing
Phonetic
representations
K.Marasek
05.07.2005
Post-processing
rules
Benefits
Multimedia Department
Variants (e.g. for French)
language
independent
language
dependent
“amis”
Classificator
Classificator
(C5.0)
Tree /
Rules
/a m i %z/
Post-processing
[a m i]
[a m i z]
K.Marasek
05.07.2005
Post-processing
rules
Features
Multimedia Department
 Mapping of output
For phoneme set conversion
e.g. SAMPA-to-MRPA (i:ii, 3:@@, A:aa, …)
Removing non-native phonemes
e.g. remove nasals in German (i~E, …)
 Output rules and trees
Trees are 2-4 times bigger but can be ~10 times faster
 “All in one binary format”
Reduces the size of rules down to 30%
 Methods used: CART trees, decision trees (C5.0, Quinlan), but also
ANN, genetic algorithms
K.Marasek
05.07.2005
Challenges
Multimedia Department
 There is not a one-to-one mapping from letters to phonemes:
K.Marasek
05.07.2005
Th i s
T I s
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
Challenges
Multimedia Department
 There is not a one-to-one mapping from letters to phonemes:
K.Marasek
05.07.2005
Th i s
T I s
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
Challenges
Multimedia Department
 There is not a one-to-one mapping from letters to phonemes:
Th i s
T I s
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
 Enforce one-to-one-mapping by insertion of null phonemes (and null
graphemes)
K.Marasek
05.07.2005
Challenges
Multimedia Department
 There is not a one-to-one mapping from letters to phonemes:
Th i s
T I s
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
 Enforce one-to-one-mapping by insertion of null phonemes (and null
graphemes)
Th i s
T I s
K.Marasek
05.07.2005
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
Challenges
Multimedia Department
 There is not a one-to-one mapping from letters to phonemes:
Th i s
T I s
i s
j u s t
a
c u t e
d emo
I s dZ V s t
@
k j u: t
d i: m@U
 Enforce one-to-one-mapping by insertion of null phonemes (and null
graphemes)
K.Marasek
05.07.2005
Th i s
i s
j u s t
a
c - u t e
d emo
- T I s
I s dZ V s t
@
k j u: t -
d i: m@U
Workflow for new languages
Multimedia Department
Pronunciation
dictionary
Adaptation of
the dictionary
Understanding the phoneme set
Convert phoneme set
Correct errors
Prepare training / test set (correctly!)
C5.0
Tree/Rules
K.Marasek
05.07.2005
Workflow for new languages
Implementation of
Multimedia Department
Post-processing rules
G2p for new
language
LIMITATIONS
• Quality of the corpus
Variants
Extra information
(syllable boundaries, stress marks)
consistency
• Phonetic expertise in the language available
K.Marasek
05.07.2005
Multimedia Department
End here and go to WEB tutorium
K.Marasek
05.07.2005