Human-Computer Interaction with an Intelligence Virtual Analyst

Denis Gouin Valérie Lavigne Alexandre Bergeron-Guyard Innovative Interfaces and Interactions Group Intelligence and Information Section DRDC Valcartier KSCO 2012 February 2012

1

iVAC initiative

• Intelligent Software Assistant (ISA) identified by MIT’s Technology Review as one of 2009’s most promising emerging technologies • Conversational, computer-generated characters capable of providing guidance to a user in the conduct of his tasks • Intelligence Virtual Analyst Capability (iVAC) R&D initiative at Defence R&D Canada • 3-year initiative started in 2011 • Knowledge System with important HCI component

Virtual Assistant / Virtual Advisor

2 DARPA’s CALO Cognitive Assistant that Learns and Organizes IBM's Watson supercomputer SIRI Apps on IPhone 4s

DARPA’s Personal Assistant that Learns (PAL)

Video Clip presentation 3 http://www.youtube.com/watch?v=BF-KNFlOocQ&feature=youtu.be

4

Cognitive Assistant that Learns and Organizes (CALO)

5

HCI Requirements - General

• HCI dialogue between iVAC and the analyst(s) – How to input knowledge into the system – How to raise questions – How to task the IVA – How does the IVA ask for precisions on the questions and the way of presenting information • Optimization of the presentation of the results • One and/or multiple virtual analysts can serve multiple users • Virtual analysts can interact with each other within and across meeting spaces

6

Enabling HCI Technologies

• Smart Room Environments • Multimodal Interaction • Information Presentation and Adaptive Interfaces • Augmented Cognition • ISA Representation / Avatars • Storytelling

Smart Room Environments

• Instrumented environments (e.g. cameras, microphones, biometry) • Ubiquitous computing. The user can access the system from various locations • Track users location • Track users activities 7 US Navy Command Center of the Future (CCoF) DSTO LiveSpaces

Multimodal Interaction

• Allow the user(s) to employ multimodal interaction (e.g. voice, pointing, gesture, eye/gaze, neural/brain interfaces, emotion detection) • Resolve ambiguities – These are my priorities – We should move that milestone back 6 months Xbox Kinect 8 AFRL Interactive Data Wall Emotion Detection, Valenti et al. 2007

9

Information Presentation

• In support of various tasks – Present tools and information – Answer questions – Ask for clarification – Remind the user of some tasks or procedures – Propose alternative possibilities – Give a briefing

10

Information Presentation

• Means – Voice output and/or information display – Highlight information elements – Display information in a new window, organized as the user wants to see it – Gather a set of documents – Filter information based on user requests – Capture information about a meeting (automatic transcription and semantic analysis of multiparty meetings )

11

Adaptive Interfaces

• Customized to the role and current tasks of the user • Present tools and information based on the user preferences • Suggest sequences of events based on learned tasks and preferences • Understand the context / tasks • Observe what the user is doing and his mental state • Do not disturb • The user is recognized using biometry • The interface adapts to the distance of the user to the (large) display (text fonts and granularity of information)

Augmented Cognition

• Track the sensory and cognitive overload of the users • Use of neural/brain interfaces, emotion detection • Employ computational strategies to restore operational effectiveness – Intelligent interruption to improve limited working memory – Attention management to improve focus during complex tasks – Cued memory retrieval to improve situational awareness and context recovery – Modality switching (i.e., audio, visual) to increase information throughput – When not to disturb 12 DARPA (2011a), DARPA Defense Science Office, Improving Warfighter Information Intake Under Stress (AugCog), http://www.darpa.mil/dso/archives/warfighter/index.htm

13

ISA Representation / Avatars

• Various representations – Speech output (voice) – A simple icon – A two- or three-dimensional representation of a character (avatar) PAL

ISA Representation / Avatars

• Adaptation of the avatar to the task and socio-cultural context – Facial and voice features of the avatar (serious / smiling, tone of the voice ) should communicate emotions, danger, risk, uncertainty – Consider socio-cultural factors (Gender / age / ethnicity / profession) – Use different avatars to support different ISA tasks (weather prediction or course of action) • Embedding avatars in a video conferencing • Be careful of the Spectre of the Uncanny Valley • Need for experiences with avatar representations 14 www.postgenomic.com/faces DSTO’s FOCAL

Storytelling

• Guide audience through a preconstructed narrative • Use of tangible graphics and videos (TV news like) • Richness of the storytelling approach • Convey situation awareness about complex events 15 Logo – VisWeek -Telling Stories with Data

16

ISA for Coalition Operations

• Improved interoperability between coalition forces in terms of disseminating information • Translate information between languages • Share differences in tactics, techniques and procedures (TTPs) • Synchronizing coalition activities • Improve cognitive assistance by providing shared coalition awareness and task support • Enable coalition organizations to learn by managing a knowledge base • Provide better and faster decision making through access to comprehensive knowledge developed through time

17

Conclusion

• ISA, a knowledge system component • Solve the human cognitive overload • Conducting a wide variety of tasks – Assign tasks – Search and organize information – Manage schedules – Summarize documents – Learn procedures and preferences – Mediate interactions – Guide and remind the user – Track people • HCI is Key – Multimodal interaction – Avatar is not mandatory

18