Transcript EmuPlayer Music Recommendation System Based on User Emotion

EmuPlayer
Music Recommendation System
Based on User Emotion
Using Vital-sensor
KMSF- sunny
親：namachanさん
1
Motivation
Music – Emotion: mutual relationship
Users choose songs based on current
feelings
Playlist constantly expanding
difficulty in picking appropriate song
2
Requirement
Track User Emotion
Recommend by Sorting playlist based on
user’s current emotion
Sort songs by 2 factors
Relevancy to User Preference
Effect on User Emotion
3
Related research
 Matching Music Mood
and User Emotion
(-) Emotion: declared by
user
(-) Subjectiveness on
Music Mood?
(-) Music Mood and User
Emotion are always the
same?
 EmuPlayer
Emotion: automatically
detected
User Preference under
each emotion is studied
Recommendation
implying emotional
effect’s feedback
4
EmuPlayer: Example from User A
 Before
NoRPulse = 76.73, NoRTemp = 33.6
Pulse = 70.04, Temp = 34.0  relax/~sleepy
 Song Information
SongNo = 3
SongID = 4, Title = “So Close”
 After
“Like”
Pulse = 79.0,
Score = 1
Temp = 34.39  pleasure
5
Approach
Vital
Information
Emo
Detector
User
Emotion
Recommender
Sorted
Playlist
EmuPlayer
Emotion Recognition
Music Recommendation
6
Emotion Recognition
Merit of Vital-sensor
Requirements
Portability if integrated in Music Player
Continuity of output data
Sensitiveness towards changes in emotion
Vital-sensor meets all those requirement
7
Emotion Recognition
Vital-sensor v.s Other Methods
Portability
if integrated in
Music Player
Continuity
of output data
Sensitiveness
towards
changes in
emotion while
listening to
music
Facial Expression
X
X
△
Speech
△
X
△
Eyes movement
△
△
△
Brainwave
X
○
◎
Gesture
X
X
△
Vital-sensor
○
○
○
Requirement
regard for the use
in MRS
Method
8
Emotion Recognition
Russell model and Two Biosignals
Russell’s model
Horizontal axis:
Pleasure
SkinTemp
Vertical axis:
Arousal
Heart Rate
90°
135°
45°
180°
0°
225°
315°
270°
9
Emotion Recognition
Mapping in EmuPlayer
 Define Emotion Region
Based on Theory of the
Fuzziness of words
8 equal regions
 Mapping
Based on angle
(1,1)45°Excitement
10
Music Recommendation
2 factors to evaluate a song
Relevancy to User Preference
Mental Effect on User Emotion
2 subjects
Study User Preference
Study Emotional Effect of songs
11
Music Recommendation
Study User Preference
Rating Like/Dislike
Record listening history
12
Music Recommendation
Study songs’ emotional effect
 Define emotional
effect: Good-Bad
to avoid potentially
harmful
recommendations to
user emotion
good
bad
13
Music Recommendation
Effect Definition Survey
Matching point = 42/48*100 = 87.5%
14
Music Recommendation
Rating songs
Better songs rank higher
15
System Flow
2
Data
Pre
Processor
Data Receiver
8
3
4
Emo
Detector
9
Recommender
10
7’
11
1
Database
5
5
10’
7
RF-ECG sensor
Evaluator
6
Interface
User
16
Demonstration
17
Evaluation on Emotion Recognition
Number of
Participant
Gender
Average Age
10~
Male
21
 Experiment 1: Testing Accuracy of Emotion
Recognition through arranged situation
Survey: (1) if they experienced the emotion expressed
through the situation,
and (2) if not, what emotions rather than the one in (1)
they experienced.
18
Result of Experiment 1
Output
(Result from
Engine)
Arousal
Exciteme
nt
Pleasure
Relaxatio
n
Sleepine
ss
Disples
ure
Distress
Depressi
on
Input
(Verified
Experimenting Emo)
Relaxation
4.25% 7.96% 22.77
%
55.58
%
9.44% 0
0
0
Excitement
10%
63.34
%
21.66
%
5%
0
0
0
0
Pleasure
1.4%
10.87
%
66.86
%
20.87
%
0
0
0
0
Arousal
81.68
%
5%
0
6.66% 6.66% 0
0
0
Depression
8.33% 0
0
0
10%
55.01
%
Accuracy = 64.5%
6.66% 20%
19
Evaluation on Emotion Recognition
Experiment 2
Change User Emotion by music
Purpose
Verify whether the system can realize user’s
emotional changes
Verify songs’ influence on listeners’ emotions
20
Case
Experiment
Mean
Result
1
Arousal
Pleasure/Rela
xation
Classical
music
Arousal:81.68%1.41%
Pleasure:0%54.91%
Relaxation:6.66%38.94%
2
Normal
Pleasure
Music
participants
like
Pleasure: 66.86%  93.33%
3
Normal
Excitement
Fast beat
Music
Excitement: 10.87%62.12%
4
Normal
Depression
Loud Heavy Pleasure:66.86%Depression:
Music played 80.02%
in long time
21
Emotion Recognition
Conclusion from 2 experiments
Accuracy of Extracting Emotion: 64.5%
Strong at detecting bad emotions
Detect precisely regarding to changes of
user emotion
Hypothesis of music influencing on user
emotion is true
22
Evaluation: EmuPlayer Performance
Observing high-rating songs
% being “dislike” after being listened
% paying “bad influence” on user’s emotion
after being listened
% being reduced in score after being listened
Observing “like” song
Emotional change
23
EmuPlayer Performance
Observing high-rating songs
24
EmuPlayer Performance
Observing “like” song
Emo Before
Emo After
Percentage
Arousal
Relaxation
1%
Arousal
Sleepiness
1%
Excitement
Arousal
1%
Excitement
Pleasure
2%
Pleasure
Excitement
2%
Pleasure
Pleasure
77%
Relaxation
Arousal
1%
Relaxation
Pleasure
2%
Relaxation
Relaxation
7%
Sleepiness
Sleepiness
4%
Displeasure
Displeasure
2%
 No song influencing badly on users’ emotion
25
Conclusion of EmuPlayer Performance
EmuPlayer algorithm ensures
recommendation of songs meeting
proposed two requirements
Songs influencing badly on user emotion: 0%
Songs being “dislike” in later listening time:
6.66%
26
Overall survey
 Are you interested in such a MRS system?
 Yes: 90%
 Scale your satisfaction of EmuPlayer’s work
 Average point = 3.6/5
 Do you feel uncomfortable wearing RF-ECG?
 Yes: 40%
 Did you experience bad emotion after listening to highrating song
 Yes: 10%
 Reflect the truth: proposing ER method responds to only clear
and strong emotions. Slight changes in emotion felt by users
may not be recognized by the system
27
Conclusion
 Concept of EmuPlayer is essential
Evaluate song through 2 factors
Employ User Emotion as crucial input for MRS
 Accuracy of extracting emotion is not very high:
64.5%
 Strong at detecting bad emotion
Applicable in giving alert when playing music influences
badly on listener’s emotional state
 EmuPlayer’s efficiency in suggesting songs
meeting the two requirements
28
Future works
 Enhance the accuracy of detecting emotion by
Employing other means than Heart Rate and Skin
Temperature
Alternate RF-ECG
 Enhance the work of Recommending music by
combining proposed method with songs’ content
analyzing
 Enhance reasoning user’s state by combining
User Emotion with context analyzing
29
Thank you for listening
30