Asking Questions to Limited Domain Virtual Characters: How Good Does Speech Recognition Have to Be?

Dr. Anton Leuski, 2LT Brandon Kennedy, Ronak Patel, Dr. David Traum

Outline

• Question-Answering Characters • Sgt Blackwell • Answer selection mechanisms • Research questions – How good are the responses?

– What is the impact of imperfect ASR?

• Experiment and Results • Summary, Future Work, & Final Thoughts

Question-answering characters

• Q&A dialogue – Focus on information and social interaction • Simulate person answering question, e.g.: – From reporter – From interviewer – From police interrogator – Different from Question-answering system • Give appropriate answer rather than correct information – Different from believable characters • Focus on simulation of question-answering process rather than Turing test • Uses for Q&A characters – Simulation – Training – Games

Examples of ICT Question-answering Characters

Sgt Blackwell C3IT/TACQ: Raed Be a Reporter QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Sgt Blackwell

• Technology demo for ASC 24 • Highlights: – Life-sized, mixed reality • Trans-screen – High-production quality • Rendering (> 60K polygons) • Voice • Authored Text • Robust responsiveness – Speech recognition – speech and non-verbal reply – Limited domain of interaction: responding to interview/Q&A QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Virtual Character Creation: Data-driven method

1. Collect data (questions) a) Scripted b) Paraphrases c) Wizard of Oz d) System 2. Annotate data – – Pick appropriate answers Rate level of appropriateness 3. Train statistical algorithms 4. Integrate in system 5. Evaluate = (1d)

WOZ Data collection

Sgt Blackwell Dialogue Model

• Set of pre-constructed answers – In domain – Off-topic – Prompt • Local history • IR-based classification – Given (possibly previously unseen) question, map to best answer

Methods for Computing Responses to Questions

• Classification-Response Selection • Extraction-Template selection template filling • Parsing/Interpretation - Inference Generation

Some Word-based Classification Approaches

• • • • • • Key-word spotting Bayesian Classification Latent Semantic Analysis (LSA) Support-Vector Machines (SVM) Relevance Model Retrieval Cross-language Relevance Model

Sgt Blackwell Evaluation

Questions: 1. How good are the responses to expected questions?

2. How good are the responses to open questions?

3. How much do speech recognition errors affect performance?

Experiment

• Data: – 20 users asked Sgt Blackwell 20 questions each: • 10 given, in domain • 10 composed by user • Metrics – Accuracy of Speech recognizer & classifier – Appropriateness of replies (including to unseen and out of domain questions) • Answers rated for relevance (scale from 1-6)

Example Session:user questions

Q Who created you (Who created you) Tell you the truth, I have spent a lot of time with the smart folks at ICT, testing and tweaking. I am about ready to get out there and train with real soldiers. I am thinking first stop is going to be Fort Sill. But my ultimate goal is to get forward deployed...

Q Why should we spend money on you (What should we spend money on you) Why do you need me?! What you should be saying is "How did you get along without me?" I will show you how to be a leader, how to make critical decisions under stress... And I am glad to be here, because I can help. I know I can. Look... I have one mission: to make better soldiers, make better warfighters. And if I can ... can save one life... bring one loved one back to his family... Well, then, I am definitely worth it. You be the judge.

Q What's your value (What's the value in) I can do "Virtually" anything. I do mission rehearsals, Urban Combat training, language training - I speak Farsi, Pashtun and Arabic. "Mar Haba shabab! That means how are you in Arabic. Q Do you really work or is this just a demo (G. really worker is this just a demo) Sorry. That's outside my AO.

Q How could you be better (How could you be better) Stop mumbling. Just kidding. I didn't get that.

Q What do you wish you could do (When you wish you could do) Career Army. That is what I am thinking. Just call me a lifer.

Q What kind of people have you trained (What other people in the turn in) I teach cultural awareness, function as a training mentor and can call in the full range of Joint fires and effects.

Q What's the hardest part of your job (He was the hardest are you job) I am not authorized to comment on that Q You have any virtual friends (The universal friends) Sorry, I can't hear you Q Are you hard of hearing (Are you had a hearing) I can't understand you.

1.

2.

3.

4.

5.

6.

Response coherence coding

Gandhe et al ASC 2004 Response is not related in any way to the question Response contains some discussion of people or objects mentioned in the question, but does not really address the question itself.

Response partially addresses the question, but little or no coherence between the question and response.

Response does mostly address the question, but with major problems in the coherence between question and response; seems like the response is really addressing a different question than the one asked.

Response does address the question, but the transition is somewhat awkward.

Response answers the question in a perfectly fluent manner.

Avg Ans Score

Performance on in-domain questions

% data ASR WER

Avg Ans Score

Performance on user-selected questions

% data ASR WER

Summary

• Question Answering Characters • How important is Speech recognition accuracy?

– Not very • getting some correct words is good enough – Even a moderate quality recognizer is good enough, and worth convenience factor of speech

Future Work

• More use of context – Information transfer – Mood of character • New domains – Extended Blackwell (Cooper-Hewitt Museum) – Tactical questioning • ELECT BiLAT character Hassan • C3IT character Raed

Closing thought: NL Dialogue Processing what are the best techniques for a task?

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

QuickTime™ and a TIFF (LZW) decompressor are needed to see this picture.

Sgt. Blackwell, RAED BAR, TACQ, C3IT Understand language Text classification Radiobot Information extraction MRE, SASO (Doctor Perez) Semantic parsing Manage dialog Keep history Generate language Recorded answers Follow Protocol Template based Rule-based reasoning Statistical & Grammar-based generation