Transcript User-System Interaction: from gesture to action

User-System Interaction:
from gesture to action
Prof. dr. Matthias Rauterberg
IPO - Center for User-System Interaction
TU/e Eindhoven University of Technology
The optimization problem
costs
user
system
optimum
© M. Rauterberg, TU/e
technical complexity
of the user interface
What are the research topics of USI?
• Communication / Co-operation
• Home / Office Environment
• Financial / Medical Sector
• Knowledge Management
• Product / Process Industry
• Transportation / Logistic
• Teaching and Learning
Working domain
Working system
Goal:
user
Science
• perception
• cognition
• action
© M. Rauterberg, TU/e
interactive system
Application
• user-centered
design
Engineering
• audio interfaces
• computer vision based input
• speech input / output
• tactile input / output
Pa => Pd
Two Trends in User Interface Technology
Ubiquitous and Mobile computing
© M. Rauterberg, TU/e
Ambient rooms and
Cooperative buildings
What are the technical challenges?
• New interaction styles
–
–
–
–
speech input/output
computer vision based input (e.g., gestures)
audio interfaces (e.g., non-speech audio)
tactile and force feedback
• New interface concepts
– adaptive and intelligent software
– natural user interfaces
© M. Rauterberg, TU/e
Human-human communication
interpersonal relation
Non-verbal
facial
expression
© M. Rauterberg, TU/e
verbal
body
movement
gesture
content
action
explicit
emotion
implicit
emotion
Non-facial gestures associated with speech act
• Iconic gestures
– form of this gesture supports a feature of the described action or
event (e.g., “he climbed up the pipe…”)
• Metaphoric gestures
– form of this gesture is representational (e.g., “the meeting went on
and on…”)
• Deictic gestures
– form of this gesture specializes or locates something in the
physical space nearby the narrator (e.g., “this document shows…”)
• Beat gestures
– small baton-like gestures that serve a pragmatic function (e.g., “she
talked first, I mean second…”)
© M. Rauterberg, TU/e
What is the research approach?
Design relevant
knowledge
Empirical
evaluation
The most important design principle
• Perception Space
– The physical space where
the user’s attention is.
• Action Space
– The physical space where
the user acts in.
• Design Principle:
– perception space and action
space must coincide!
© M. Rauterberg, TU/e
Two design principles for natural user
interfaces (NUIs):
1. design principle
No technical equipment inside
to body space of the user!
2. design principle
Perception space and action
space must coincide!
© M. Rauterberg, TU/e
© M. Rauterberg, TU/e
BUILD-IT: a tool for integrative design
• design team with experts
from different disciplines
• integration of
different expertise
• intuitive interaction style
• an example for a natural
user interface
© M. Rauterberg, TU/e
How to meet the challenges?
Co-operation inside TUE
Research in perception, cognition, and action
Research in speech and computer graphics
Research in signal processing and pattern recognition
© M. Rauterberg, TU/e
Co-operation outside TUE