4/30/2020

Spoken Dialogue Systems

Julia Hirschberg CS 4706 1

Today

• Some Swedish examples • Controlling the dialogue flow – State prediction • Controlling lexical choice • Learning from human-human dialogue – User feedback • Evaluating systems 4/30/2020 2

4/30/2020

The Waxholm Project at KTH

• tourist information • Stockholm archipelago • time-tables, hotels, hostels, camping and dining possibilities.

• mixed initiative dialogue • speech recognition • multimodal synthesis • graphic information • pictures, maps, charts and time-tables • Demos at http://www.speech.kth.se/multimodal 3

4/30/2020

The Waxholm system

4

Today

• Some Swedish examples • Controlling the dialogue flow – State prediction • Controlling lexical choice • Learning from human-human dialogue – User feedback • Evaluating systems 4/30/2020 5

Dialogue control state prediction

Dialog grammar specified by a number of states Each state associated with an action database search, system question… … Probable state determined from semantic features Transition probability from one state to state Dialog control design tool with a graphic interface 4/30/2020 6

Waxholm Topics

TIME_TABLE

Task: get a time-table.

Example: När går båten? (When does the boat leave?)

SHOW_MAP

Task : get a chart or a map displayed.

Example: Var ligger Vaxholm? (Where is Vaxholm located?)

EXIST

Task : display lodging and dining possibilities.

Example: Var finns det vandrarhem? (Where are there hostels?)

OUT_OF_DOMAIN

Task : the subject is out of the domain.

Example: Kan jag boka rum. (Can I book a room?)

NO_UNDERSTANDING

Task : no understanding of user intentions.

Example: Jag heter Olle. (My name is Olle)

END_SCENARIO

Task : end a dialog.

Example: Tack. (Thank you.) 4/30/2020 7

Topic selection

4/30/2020 FEATURES TIME TABLE TOPIC EXAMPLES SHOW FACILITY NO UNDER- OUT OF MAP STANDING DOMAIN OBJECT QUEST-WHEN QUEST-WHERE FROM-PLACE AT-PLACE

.062

.188

.062

.250

.062

TIME PLACE OOD END HOTEL HOSTEL ISLAND PORT MOVE

.312

.091

.062

.062

.062

.062

.333

.125

.875

.312

.031

.688

.031

.219

.031

.200

.031

.031

.031

.031

.556

.750

.031

.073

.024

.390

.024

.293

.024

.500

.122

.024

.488

.122

.062

.244

.098

.091

.091

.091

.091

.091

.091

.091

.091

.091

.091

.091

.091

.091

.091

.067

.067

.067

.067

.067

.067

.067

.933

.067

.067

.067

.067

.067

.067

END

.091

.091

.091

.091

.091

.091

.091

.091

.909

.091

.091

.091

.091

.091

argmax i

{ p(t

i

| F )}

8

Topic prediction results

4/30/2020

15 10 12,9 8,8 12,7 8,5 5 3,1 2,9 0 complete parse raw data no extra linguistic sounds All “no understanding” excluded

9

Today

• Some Swedish examples • Controlling the dialogue flow – State prediction • Controlling lexical choice • Learning from human-human dialogue – User feedback • Evaluating systems 4/30/2020 10

4/30/2020

User answers to questions?

The answers to the question: “

What weekday do you want to go?

” (Vilken veckodag vill du åka ?) • 22% • 11% • 11% • 7% • 6%

Friday

(fredag)

I want to go on Friday

(jag vill åka på fredag)

I want to go today

(jag vill åka idag)

on Friday

(på fredag)

I want to go a Friday

(jag vill åka en fredag) • -

are there any hotels in Vaxholm?

(finns det några hotell i Vaxholm) 11

Examples of questions and answers

Hur ofta åker du utomlands på semestern?

Hur ofta reser du utomlands på semestern?

jag åker jag åker en gång om året kanske ganska sällan utomlands på semester jag åker nästan alltid utomlands under min semester jag åker ungefär 2 gånger per år utomlands på semester jag åker utomlands nästan varje år jag åker utomlands på semestern varje år jag åker utomlands ungefär en gång om året jag är nästan aldrig utomlands en eller två gånger om året en gång per semester kanske en gång per år ungefär en gång per år åtminståne en gång om året nästan aldrig jag reser jag reser en gång om året utomlands inte ofta utomlands på semester det blir mera i arbetet jag reser reser jag reser utomlands på semestern vartannat år utomlands en gång per semester jag reser utomlands på semester ungefär en gång per år jag brukar resa utomlands på semestern åtminståne en gång i året en gång per år kanske en gång vart annat år varje år vart tredje år ungefär nu för tiden inte så ofta varje år brukar jag åka utomlands

4/30/2020 12

other 24%

Results

no reuse 4% 2% no answer reuse 52% 4/30/2020 18% ellipse 13

Today

• Some Swedish examples • Controlling the dialogue flow – State prediction • Controlling lexical choice • Learning from human-human dialogue – User feedback • Evaluating systems 4/30/2020 14

The August system

4/30/2020 Do you like it here?

should not throw stones 15

Evidence from Human Performance

• Users provide explicit positive and negative feedback • Corpus-based vs. laboratory experiments – do these tell us different things?

4/30/2020 16

4/30/2020

Adapt – demonstration of ”complete” system

17

Feedback and ‘Grounding’: Bell & Gustafson ’00 • Positive and negative – Previous corpora: August system • 18% of users gave pos or neg feedback in subcorpus • Push-to-talk • Corpus: Adapt system – 50 dialogues, 33 subjects, 1845 utterances – Feedback utterances labeled w/ • Positive or negative • Explicit or implicit • Attention/Attitude • Results: – 18% of utterances contained feedback – 94% of users provided 4/30/2020 18

– 65% positive, 2/3 explicit, equal amounts of attention vs. attitude – Large variation • Some subjects provided at almost every turn • Some never did • Utility of study: – Use positive feedback to model the user better (preferences) – Use negative feedback in error detection 4/30/2020 19

This is a 3D test environment

The HIGGINS domain • • The primary domain of HIGGINS is city navigation for pedestrians. Secondarily, HIGGINS is intended to provide simple information about the immediate surroundings.

4/30/2020 20

Initial experiments

• Studies on human-human conversation • The Higgins domain (similar to Map Task) • Using ASR in one direction to elicit error handling behaviour

User

4/30/2020

Speaks Listens ASR Vocoder Reads Speaks Operator

21

Non-Understanding Error Recovery (Skantze ’03)

• Humans tend not to signal non-understanding: – O: you?

Do you see a wooden house in front of – U: NOW ASR: YES CROSSING ADDRESS (I pass the wooden house now) – O: Can you see a restaurant sign?

• This leads to – Increased experience of task success – Faster recovery from non-understanding 4/30/2020 22

Today

• Some Swedish examples • Controlling the dialogue flow – State prediction • Controlling lexical choice • Learning from human-human dialogue – User feedback • Evaluating systems 4/30/2020 23

Evaluating Dialogue Systems

• PARADISE framework (Walker et al ’00) • “ Performance ” of a dialogue system is affected both by

what

gets accomplished by the user and the dialogue agent and

how

it gets accomplished

Maximize Task Success

4/30/2020

Minimize Costs Efficiency Measures Qualitative Measures

24

Task Success

•

Task goals seen as Attribute-Value Matrix

ELVIS e-mail retrieval task

(Walker et al ‘97) “Find the Kim .” time and place of your meeting Attribute Selection Criterion Time Place Value Kim or 10:30 a.m.

2D516 Meeting with

•

Task success defined by match between AVM values at end of with “true” values for AVM

4/30/2020 25

Metrics

• Efficiency of the Interaction:User Turns, System Turns, Elapsed Time • Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Mean Recognition Score (concept accuracy), Cancellation Requests • User Satisfaction • Task Success : perceived completion, information extracted 4/30/2020 26

Experimental Procedures

• Subjects given specified tasks • Spoken dialogues recorded • Cost factors, states, dialog acts automatically logged; ASR accuracy,barge-in hand-labeled • Users specify task solution via web page • Users complete User Satisfaction surveys • Use multiple linear regression to model User Satisfaction as a function of Task Success and Costs; test for significant predictive factors 4/30/2020 27

User

Satisfaction

:

Sum of Many Measures

• Was Annie easy to understand in this conversation? ( TTS Performance ) • In this conversation, did Annie understand what you said? ( ASR Performance ) • In this conversation, was it easy to find the message you wanted? ( Task Ease ) • Was the pace of interaction with Annie appropriate in this conversation? ( Interaction Pace ) • In this conversation, did you know what you could say at each point of the dialog?

(

User Expertise

) • How often was Annie sluggish and slow to reply to you in this conversation? ( System Response ) • Did Annie work the way you expected her to in this conversation? ( Expected Behavior ) • From your current experience with using Annie to get your email, do you think you'd use Annie regularly to access your mail when you are away from your desk? ( Future Use ) 4/30/2020 28

Performance Functions from Three Systems

• ELVIS User Sat.= .21* COMP + .47 * MRS - .15 * ET • TOOT User Sat.= .35* COMP + .45* MRS - .14*ET • ANNIE User Sat.= .33*COMP + .25* MRS +.33* Help – COMP: User perception of task completion (task success) – MRS: Mean recognition accuracy (cost) – ET: Elapsed time (cost) – Help: Help requests (cost) 4/30/2020 29

Performance Model

• Perceived task completion and mean recognition score are consistently significant predictors of User Satisfaction • Performance model useful for system development – Making predictions about system modifications – Distinguishing ‘ good ’ dialogues from ‘ bad ’ dialogues • But can we also tell on-line when a dialogue is ‘ going wrong ’ 4/30/2020 30

Next Class

• Turn-taking (J&M, Link to conversational analysis description, Beattie on Margaret Thatcher) 4/30/2020 31