Transcript Interaction, Work and Technology V:

Lecture 17:
Interaction, Work and Technology III:
Affective HCI
Jon Oberlander
1
Introduction

Affect
– Timescales, complexity

Affect in Computer mediated communication
– Especially, detection

Affect in Human computer interaction
– Uses and means
– 12 recent papers in Affective HCI

The future of Affect
UG4: HCI Lecture 17
2
What is Affect?

Variable timescale of affect
– Feelings
• Joyful: second, minutes
– Moods
• Happy: hours, days
– Personalities/Temperaments
• Outgoing: years, decades

Variable complexity of affect
– Basic feelings:
• Happiness, surprise, fear, sadness, disgust, anger
– More complex ones:
• Jealousy, hope, relief, pride, remorse … anxiety, frustration
– Or just:
• Arousal versus valence
UG4: HCI Lecture 17
3
Personality: Five Factor Models

The Big Five factors of personality analysis (OCEAN)
– Extraversion, Energy, Enthusiasm
talkative, assertive, dominant
quiet, reserved, shy, retiring
– Agreeableness, Altruism, Affection
sympathetic, kind, warm, helpful
(IV)
stable, calm, contented
– Openness, Originality, Open-mindedness
imaginative, artistic, inventive
(III)
careless, disorderly, frivolous
– Neuroticism, Negative affectivity, Nervousness
tense, anxious, moody, worrying
(II)
fault-finding, cold, unfriendly
– Conscientiousness, Control, Constraint
organized, thorough, efficient
(I)
(V)
commonplace, simple, shallow
UG4: HCI Lecture 17
4
Affect in computing: What is it good for?


But what can you actually do with affect in CMC and HCI?
Picard 1997:
– Recognise people’s feelings, moods, temperaments
– Express a particular feeling, mood or temperament
– Have a feeling:
• caused by events in the world, or in yourself
• causing—or at least biasing—further actions by you.
UG4: HCI Lecture 17
5
HCI diagnosis of HAL 9000 failure


According to Rosalind Picard, the problem wasn’t that HAL had
emotions.
He could certainly recognise other people’s
– “Look Dave, I can see you’re really upset about this”

He had his own
– “I’m afraid, I’m afraid”

The problem was expressing them to others:
– That level tone of voice betrayed nothing
– And what about his facial expressions?
UG4: HCI Lecture 17
6
HCI diagnosis of HAL 9000 failure: A more expressive HAL?


Might have been more understandable to his crewmates
And left us with a less dramatic film …
UG4: HCI Lecture 17
7
Affect in CMC: Why would it matter?



We’ve seen that CSCW deprives people of cues
to other’s states of mind
What is actually available in CSCW?
Do people use it?
UG4: HCI Lecture 17
8
Affect in CMC: Personality Perception

Computer-mediated communication
– Reduces available cues

Extraversion
– High observability
– Low evaluativeness


Neuroticism: more evaluative and less observable
Previous findings:
– Relatively high agreement for Extraversion
– Even at zero-acquaintance and in CMC environment
UG4: HCI Lecture 17
9
Perception: Methods

Participants
– 30 experienced e-mail users
– Current or recent university students
– Naive raters of personality

Materials
– Six texts representative for range of scores for Extraversion
– Six texts representative for range of scores for Neuroticism
– For a given dimension, other EPQ-R dimensions held constant

Procedure
– Texts subjectively rated using exemplar descriptions
(Eysenck and Eysenck, 1991; Sneed, McCrae and Funder, 1998)
– Ease of judgment rated
– Participants also rated similarity of target
UG4: HCI Lecture 17
10
Perception: Results for Extraversion

Strong target-rater agreement
– Aggregate correlation (McCrae and Costa, 1987):
rs(5)= .89, p=0.019
– Also relatively high inter-rater agreement:
Mean rs = .48 (SD=0.17)

High E judges rate High E targets as more similar
– Interaction effects of text author and rater personality
– However, Mid E rated as more dissimilar than Low E
– … All judges rated High E texts as more similar

High E texts viewed as easier to rate
UG4: HCI Lecture 17
11
Perception: Results for Neuroticism

Poor target-rater agreement
– Aggregate correlation:
rs(5)= –.37, ns
– Somewhat better inter-rater agreement:
Mean rs = .31 (SD=0.16)


No similarity rating effects
No ease of rating effects
UG4: HCI Lecture 17
12
Perception: Discussion

Extraversion is highly salient (observable)
– Even through minimal cues
– Using exemplar description ratings
– Confirms previous literature

But what about Neuroticism?
– Apparently it is not consciously detected in language
– Does this mean it is irrelevant to communication?
– …and therefore irrelevant to language generation?
UG4: HCI Lecture 17
13
Affect in HCI: Why would it matter?

Reeves and Nass:
– Computers Are Social Actors
eg: people are polite to computers

Nass et al. (1995) had subjects use a text-only interface to
solve a problem with help from the computer.
– Simple language variables were manipulated to provide Dominant
and Submissive system versions
• Priority, hedges, confidence, name

They found similarity attraction effects
– Preference extended to estimates of system efficiency, etc.
– Cf. more recent results from Isbister and Nass (2000, 2001).
– If a system can project a consistent—or even better, a convergent—
linguistic personality, this will enhance the user’s experience.
UG4: HCI Lecture 17
14
Affect in HCI: the medium term



In the neaar future, we are unlikely to have machines that Have
emotions.
But, given the Reeves and Nass results, it may be worth working
on the Recognition and Expression of emotions, moods and
temperaments.
Why would this help?
– Recognition: making personal agents sensitive to our feelings or
moods.
• Boredom, inattention, stress
• Content indexing (pain, fear, rage)
• Deception, anxiety detection
– Expression: making computers appear to have feelings, moods or
temperaments, even if they don’t
• More acceptable (?) companions
• Better information transmission
UG4: HCI Lecture 17
15
The uses of recognition

Personal agents
– Software on your mobile, PDA, desktop or dashtop
•
•
•
•
•
•
•
It filters your email or vmail,
shops for bargains for you,
chooses mood music to calm you down or wake you up,
finds places of interest to visit,
finds news snippets,
solves problems
arranges meetings.
– To do this well, it spends a lot of time with you, learning what you
do, and how you like to do it

But it would do much better if:
– It knew when you were interruptible
– It knew how well you liked its suggestions
UG4: HCI Lecture 17
16
Means of recognition


Emotion response, not mental workload …
Physiological (contact):
– Galvanic Skin Response
• How stressed you are
– Blood Volume Pressure
– EEG
• Which areas of your brain are most active
– ECG, ERP, pulse, respiration, pupil dilation …

Visual (remote):
– Facial expression
• Muscle action units (cf. Ekman)
– Posture, gait, gesture

Audio (remote):
– Voice features:
• Volume, rate, pitch range, quality
UG4: HCI Lecture 17
17
The uses of expression


Software agents may be more or less confident in their
recommendations
Presenting a lot of information via voice is a problem, since it
takes time, and doesn’t really use the voice channel to the full
UG4: HCI Lecture 17
18
Means of expression


If an agent has an animated talking head, it should change its
vocal and facial expression if it isn’t sure you’ll like its
suggestion or if it’s unusual
A speech synthesiser should be able
to distinguish
– News items you will or won’t enjoy hearing about
– Email actually requiring an answer—as opposed to all that stuff you
just get cc’d on

Non-human, but predictable, expression
might help too:
– Culture drones have colour fields (Iain M. Banks)
UG4: HCI Lecture 17
19
J Scheirer, R Fernandez, J Klein, RW Picard – 2002


Frustrating the user on purpose: a step toward building an
affective computer - Interacting with Computers, 2002
Timed visual sorting task, with mouse clicks
– Eliciting frustration by adding unpredictable delays to some clicks
• Poor predictability, inconsistent honesty
– Measuring:
• GSR (arousal -> sweat -> higher conductance); frustration & anxiety
• BVP (anxiety -> cold feet -> light absorbtion in capillaries)
– Hidden markov models to learn sequence labels (frustration: 1/0)
• 67% accuracy on test set (50% baseline)
• Four click strategies (relative aggression?)
– Several HCI-affect guidelines, including
• Eliciting affect may require deception, and violation of HCI guidelines
– More pattern recognition research needed;
eliciting pleasure worth considering.
UG4: HCI Lecture 17
20
M Pantic, LJM Rothkrantz – 2003


Toward an affect-sensitive multimodal human-computer interaction Proceedings of the IEEE, 2003
Major survey paper, covering physiology, visual (facial) and auditory
(speech) modalities
– Rapid behavioural signals communicate messages including:
affective states, emblems (wink), manipulators (scratch, put), illustrators
(brow raise), regulators (nods)
– Recognise from 10ms audio, 40ms video
– Notes face features (up to 98% acc), speech features (70-80% acc),
compared with (eg) 17% baseline
– Notes drive toward bimodal recognition

Notes:
– most work assumes clean, posed input,
– ignores task and communicative context
– Accepts “late integration”

Acceptance of affective HCI depends on user-friendliness and
trustworthiness
UG4: HCI Lecture 17
21
FN Egger – 2001



Affective Design of E-Commerce User Interfaces: How to
Maximise Perceived Trustworthiness - Proc … of CAHD2001:
Conference on Affective Human Factors Design, 2001
All about trust in e-commerce (and hence, websites, mostly)
Focus on initial trust
– Before (brand), during (user experience), after (customer service)

Responsiveness, expensiveness
– Induce trust (cf. Fogg et al.)
UG4: HCI Lecture 17
22
F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003


From Greta's mind to her face: modelling the dynamics of
affective states in a conversational … - … - International Journal
of Human-Computer Studies, 2003
Affective Presentation Markup Language
– To drive facial actions (given facial definitions) and voice tone
appropriate to affective status of conversation
– Predicted from status of goals;
• cf. Ortony et al. 1988
– Modelled with dynamic belief networks


Good survey of expressive agents
Problems noted:
–
–
–
–
Spurious variations
Overemotion
Emotion masking
Expression overlapping (and time stability)
UG4: HCI Lecture 17
23
F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003
UG4: HCI Lecture 17
24
C Lisetti, F Nasoz, C LeRouge, O Ozyer, K Alvarez – 2003





Developing multimodal intelligent affective interfaces for telehome health care - International Journal of Human-Computer
Studies, 2003
Doctors and nurses need to know how remote patients feel
BodyMedia SenseWear wireless non-invasive wearable computer
for physiological recognition (GSR, temperature, movement)
Haptek PeoplePutty Avatar (talking head) to elicit information,
reflect empathy
10 participants; 35 minutes of data with elicited emotion states
– K nearest neighbour; discriminant function analysis
– Accuracies: 90% (sadness) 80% (anger, fear), 70% (frustration)
UG4: HCI Lecture 17
25
E Hudlicka – 2003



To feel or not to feel: The role of affect in human–computer
interaction - International Journal of Human-Computer Studies,
2003
Introduction to special issue
Draws together the disciplines:
– Neuroscience (bases), cognitive psychology (uses), AI
(models)

Cites Picard as proponent, Hollnagel as sceptic
– Notes the question: even if you can detect/express affect, does it
actually enhance HCI?

Lots of useful references, and an orienting framework
UG4: HCI Lecture 17
26
T Partala, V Surakka – 2004


The effects of affective interventions in human–computer
interaction - Interacting with Computers, 2004
Measure pupil size, in response to emotional auditory stimuli
UG4: HCI Lecture 17
27
D Johnson, J Wiles – 2003

Effective affective user interface design in games - Ergonomics,
2003
UG4: HCI Lecture 17
28
RW Picard – 2003

Affective computing: challenges - International Journal of
Human-Computer Studies, 2003
UG4: HCI Lecture 17
29
RW Picard, SB Daily – 2005

Evaluating affective interactions: Alternatives to asking what
users feel - CHI Workshop on Evaluating Affective Interfaces:
Innovative …, 2005
UG4: HCI Lecture 17
30
K Hone – 2006

Empathic agents to reduce user frustration: The effects of
varying agent characteristics - Interacting with Computers, 2006
UG4: HCI Lecture 17
31
ME Foster, J Oberlander - 2006

Mary Ellen Foster’s study explores types of variation in emphatic
talking heads and finds that:
– Variation is sometimes preferred to more simple model.
UG4: HCI Lecture 17
32
… a note of caution
Male
subjects
UG4: HCI Lecture 17
33
… a note of caution
Female
Subjects
Cf. J. Hall
1984
UG4: HCI Lecture 17
34
Do the (new) experiment
http://homepages.inf.ed.ac.uk/mef/head-experiments/
UG4: HCI Lecture 17
35
Affect in HCI: the longer term

Norman, in Emotional Design, points out:
– Complex machines with emotions will be more understandable to us
than complex ones without them.
– We predict and explain each other’s behaviour by reference to their
thoughts and feelings.
– People who appear to have no feelings are very hard to predict or
explain.
– So, having understandable emotions would make machines easier
to interact with
UG4: HCI Lecture 17
36
Articles
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
J Scheirer, R Fernandez, J Klein, RW Picard – 2002 - Frustrating the user on purpose:
a step toward building an affective computer - Interacting with Computers
M Pantic, LJM Rothkrantz – 2003 - Toward an affect-sensitive multimodal human-computer
interaction - Proceedings of the IEEE
FN Egger – 2001 - Affective Design of E-Commerce User Interfaces: How to Maximise Perceived
Trustworthiness - Proc … of CAHD2001: Conference on Affective Human Factors Design
F Rosis, C Pelachaud, I Poggi, V Carofiglio, BD … - 2003 - From Greta's mind to her face:
modelling the dynamics of affective states in a conversational … - … - International Journal of
Human-Computer Studies
C Lisetti, F Nasoz, C LeRouge, O Ozyer, K Alvarez – 2003 - Developing multimodal intelligent
affective interfaces for tele-home health care - International Journal of Human-Computer Studies
E Hudlicka – 2003 - To feel or not to feel: The role of affect in human–computer interaction International Journal of Human-Computer Studies
T Partala, V Surakka – 2004 - The effects of affective interventions in human–computer
interaction - Interacting with Computers
D Johnson, J Wiles – 2003 - Effective affective user interface design in games - Ergonomics
RW Picard – 2003 - Affective computing: challenges - International Journal of Human-Computer
Studies
RW Picard, SB Daily – 2005 - Evaluating affective interactions: Alternatives to asking what users
feel - CHI Workshop on Evaluating Affective Interfaces: Innovative …
K Hone – 2006 - Empathic agents to reduce user frustration: The effects of varying agent
characteristics - Interacting with Computers
ME Foster, J Oberlander - 2006 - Data-driven generation of emphatic facial displays. Proceedings
of the 11th European ACL
UG4: HCI Lecture 17
37
Further Reading





Byron Reeves and Cliff Nass “The Media Equation”
Rosalind Picard “Affective Computing”
Don Norman “Emotional Design”
Cynthia Breazeal “Designing Sociable Robots”
Alastair Gill, Jon Oberlander and Elizabeth Austin “Perception of
e-mail personality at zero acquaintance”
UG4: HCI Lecture 17
38