TMSG NuActiv: Recognizing Unseen New Activities Using Semantic Attribute Based Learning

實驗室: 先進網路技術與服務實驗室 報告者: 黃福銘 (Angus F.M. Huang)

2013.07.10

Publication

• MobiSys 2013 • Session 8: Behavior and Activity Recognition • NuActiv: Recognizing Unseen New Activities Using Semantic Attribute Based Learning • SocioPhone: Everyday Face-To-Face Interaction Monitoring Platform Using Multi-Phone Sensor Fusion • MoodScope: Building a Mood Sensor from Smartphone Usage Patterns • Auditeur: A Mobile-Cloud Service Platform for Acoustic Event Detection on Smartphones Angus F.M. Huang

2

Authors

• Heng-Tze Cheng • Feng-Tso Sun • Martin Griss – Electrical and Computer Engineering – Carnegie Mellon University • Paul Davis • Jianguo Li • Di You – Applied Research Center – Motorola Mobility Angus F.M. Huang

3

• To recognize a new human activity – No such training example • Essential for – User-centric & Context-aware applications • Most methods – Only recognize known activities • NuActiv – Semantic attributes for representation – Two-layer zero-shot learning algorithm – User feedback for reinforcing accuracy

Abstract

• Evaluation – 10-exercise-activity dataset collected – Public dataset of 34 daily life activities – Achieved up to 79% accuracy Angus F.M. Huang

4

Outline

• • •

INTRODUCTION NUACTIV SYSTEM OVERVIEW

– Scenarios and Design Considerations – System Architecture of NuActiv •

SYSTEM DESIGN AND ALGORITHMS

– Feature Extraction – Semantic Attribute-Based Activity • Recognition – Active Learning: Reinforcing Activity Recognition Using Minimal User Feedback

EVALUATION

– System Implementation – Datasets – Evaluation Methodology – Case Study I: Exercise Activities – Case Study II: Daily-Life Activities – Active Learning Experiments

CONCLUSION

Angus F.M. Huang

5

Introduction

• Diversified people daily activities – 462 different activities – American Time Use Survey • Two steps of existing approaches – (1) Collect and label – (2) Classify Angus F.M. Huang

6

Two Research Questions

• Q1. How to recognize a previously unseen new activity class when we have no training data from users?

• Q2. If we have the opportunity to ask users for labeled training data, how to reinforce the recognition accuracy using minimal help from users?

Angus F.M. Huang

7

Two Inspirations

• Many human activities and context types share the same underlying semantic attributes – Sitting vs...

• Having lunch in the cafeteria • Working at a desk • The limit of supervised learning can be overcome by incorporating human knowledge – Office working vs...

• Motion-related attributes: Sitting, HandsOnTable, • Sound-related attributes: PrinterSound, KeyboardSound, Conversations Angus F.M. Huang

8

Challenges – Q1

• Zero-shot learning – To learn a classifier that can recognize new classes that have never appeared in the training dataset • Which attributes are useful • From Static image data to Sequential sensor data • Decompose high-level activities into combinations of semantic attributes – Human readable term – Two-layer attribute-based learning algorithm Angus F.M. Huang

9

Challenges – Q2

• Accuracy reinforcement by user feedback • Outlier-aware active learning algorithm • Hybrid stream/pool-based sampling scheme Angus F.M. Huang

10

Activity Recognition

Angus F.M. Huang

11

NuActiv System Overview

• Two scenarios of activity domain • Daily life activities – ex. ReadingAtHome & Driving –

→

ReadingOnTrain – It is also arguably of much larger variation because different people do the same things differently (time, situation, …) • Exercise activities – Various same underlying attributes Angus F.M. Huang

12

System Architecture of NuActiv

Angus F.M. Huang

13

System Design and Algorithms

• Feature Extraction • Semantic Attribute-Based Activity Recognition • Active Learning: Reinforcing Activity Recognition using Minimal User Feedback Angus F.M. Huang

14

Feature Extraction

• The mean and standard deviation of sensor data in dimension x, y, and z • Pairwise correlation between each pair of dimensions x, y, and z • Local slope of sensor data in dimension x, y, and z in using 1 st -order linear regression • Zero-crossing rate in dimension x, y, and z Angus F.M. Huang

15

Examples of features extracted from acceleration data for each exercise activity

Angus F.M. Huang

16

Semantic Attribute-Based Activity Recognition

• Activity class space, y = {y 1 ,y 2 , …,y k } • Feature space, x = [X 1 ,X 2 , …,X d ] • Classifier function f: x→y • y = {{y 1 ,y 2 , …,y s },{y s+1 , …y s+u }} = y S – y S , seen classes – y U , unseen classes U y U • Problem: How to recognize an unseen class y ∈ y U ?

• Semantic attribute space, a = [A 1 ,A 2 , …,A m ] Angus F.M. Huang

17

Graphical representation of semantic attribute-based activity recognition

Angus F.M. Huang

18

Activity-Attribute Matrix

• • Encode the human knowledge

M

x

N

matrix; M activities, N attributes • Common-sense knowledge, domain knowledge, web text mining, crowdsourcing platforms … Angus F.M. Huang

19

Attribute Detection

• To train a set of attribute detectors so that we are able to infer the presence/absence of an attribute from the sensor data features • Training data – what we need is one set of positive samples and another set of negative samples • Classifier – Support Vector Machine (SVM) • After training phase, we have a trained attribute detector for each attribute specified in the activity-attribute matrix Angus F.M. Huang

20

Attribute-Based Activity Classification

• A nearest-neighbor classifier is used to recognize the high-level activity – given an attribute vector generated from attribute detectors – in the attribute space • The activity recognizer – takes an attribute vector a = [A 1 ,A 2 , …,A m ] as input – returns the closest high-level activity

y*

Angus F.M. Huang

21

Hybrid Feature/Attribute-Based Activity Recognition

• Transforming low-level features to mid-level attributes has the benefit for unseen class recognition – to keep the advantages of both feature-based and attribute-based • How do we know if a sample belongs to a seen class or an unseen class?

– a sample from seen class -> similar to others – a sample from unseen class -> like anomaly • We first train an unseen class detector using the one-class SVM classifier – where only the positive samples are given to the classifier • After using the unseen class detector, we then do a hybrid feature/attribute-based activity recognition using the Algorithm-1 Angus F.M. Huang

22

Angus F.M. Huang

23

Active Learning: Reinforcing Activity Recognition using Minimal User F eedback

• Our idea is simple: – We ask a user for labels only when we are highly uncertain about our recognition result • To achieve this: – We used the idea of uncertainty sampling in the field of active learning • The idea of active learning algorithms – a machine learning algorithm can perform better with less training data if it is allowed to choose the data from which it learns Angus F.M. Huang

24

• Hybrid Sampling Scheme – Stream-based sampling – Pool-based sampling • Uncertainty Sampling Metrics – Least Confident – Minimum Margin – Maximum Entropy • But!

– Outliers get higher uncertainty scores – Outliers do not help training a classifier • Outlier-Aware Uncertainty Sampling – To select samples that are uncertain but not outliers – Mean similarity between this sample and all the other samples – Algorithm-2 Angus F.M. Huang

25

Angus F.M. Huang

26

Evaluation

• System Implementation – Nexus S 4G phones & MotoACTV wristwatches • Accelerometer and gyroscope – Android app • JAVA, SVM classifier, LibSVM library Angus F.M. Huang

27

Angus F.M. Huang

28

• Datasets – Exercise activity dataset • 20 subjects, 10 exercise activities, 10 iterations, dumbbell • phone-arm, watch-wrist, watch-hip • 30 Hz sampling rate, 1 second time window size with 50% overlap – Public dataset on daily-life activities • A published dataset, Technische Universität Darmstadt • 34 classes, one subject, 7 days • Wearable sensor-wrist&hip, 100Hz, 30 seconds with 50% overlap • 17 attributes Angus F.M. Huang

29

Evaluation Methodology

• Leave-two-class-out validation – Each time we first train our system on (N-2) classes, and then test the discriminative capability of the classifier on the remaining 2 classes that were "unseen" by the system during the training process – True positive (TP), True negative (TN), False positive (FP), False negative (FN) • Precision – the percentage of times that a recognition result made by the system is correct • Recall – the percentage of times that an activity performed by a user is detected by the system • F 1 -score – a integrated measure that combines both Angus F.M. Huang

30

Unseen Activity Recognition Result

• What is the overall precision/recall of unseen activity recognition using NuActiv? How does the performance vary among classes?

• Average accuracy – 79%, for overall activities – 80-90%, for five activities Angus F.M. Huang

31

The Impact of Number of Unseen Classes And Comparison with Baseline

• How does the recognition accuracy change with the number of unseen classes?

• Baseline approach – Random-guess prediction Angus F.M. Huang

32

Comparison of Different Attribute Detecto rs

• How does the performance vary with the use of different classification algorithms for attribute detectors?

– Decision Tree classifier, Naive Bayes classifier, and k-Nearest Neighbor (k-NN) classifier – k-NN is comparable to SVM • but requires the storage and access to all the training data Angus F.M. Huang

33

Evaluation of The Importance of Attributes

• How to select attributes based on their importance to unseen activity recognition?

– Discriminability • how well can an attribute discriminate between different high-level classes • ArmUp, ArmDown, ArmFwd – Detectability • how accurately can we detect the presence or absence of an attribute Angus F.M. Huang

34

Cross-User Activity Recognition Results

• What is the cross-user performance, i.e. when the users in the training set are different from those in the testing set? Is the system able to generalize from one or a few users to many new users?

– 70-80% for all – 5 seen users – 65% for 1 seen Angus F.M. Huang

35

Impact of Device Position on Attribute Detection Accuracy

• An attribute is often inherently associated with a characteristic of a human activity or a motion of a specific part of human body – to understand how the position or the set of positions at which the sensors are placed affects the attribute detection accuracy • The upper arm sensor – achieves better and stable Angus F.M. Huang

36

Recognizing Unseen New Daily Life Activi ty

• How does NuActiv perform on recognizing unseen daily life activities?

– 60-70% precision and recall rate – Some classes have High recall and Low precision, and vice versa – Some classes do not have clear difference • Sitting-desk-activities vs. Sitting-talking-on-phone • Low precision rate Angus F.M. Huang

37

Comparison of Active Learning Algorithms

• How efficiently can the system reinforce its performance using active learning? How does the performance vary with different active learning algorithms?

Angus F.M. Huang

38

Outlier-Aware Uncertainty Sampling Results

• What is the effect of outlier-aware uncertainty sampling on active learning algorithms?

Angus F.M. Huang

39

Conclusion

• NuActiv – Uses semantic attribute-based learning to recognize unseen new activity classes by reusing and generalizing the attribute model learned for other seen activities • Outlier-aware active learning algorithm – To efficiently improves the recognition accuracy of the system using minimal user feedback • 79% recognition accuracy on the unseen activity recognition Angus F.M. Huang

40

Angus Comments

• Very depend on domain knowledge • Pity: it is unseen, not unknown • Can apply it to PLASH..

– Unseen activities recognition – Unseen trip recommendation – Composite mobility-activity recognition • car-driver / car-passenger, ...

• Important for adaptive service provision • Important attributes evaluation for PLASH’s

Urban Routing

&

Friend Matchmaking

Angus F.M. Huang

41

Dumbbell Fly Dumbbell Curl Angus F.M. Huang

42

Bent-Over Row Chest Press Angus F.M. Huang

43

Daily Life Activities Angus F.M. Huang

44

Angus F.M. Huang

45