Managing Context Information in Mobile Devices

2011. 3. 28.

Panu Korpipää, Jani Mäntyjärvi,Juha Kela, Heikki Keränen, and Esko-Juhani Malm VTT Technical Research Centre of Finland 0

Outline

• A context management framework – Information sources – Ontology for sensor-based context information – Inference mechanism concept • Bayesian reasoning for higher level contexts • Application programming interface – Adding context – Requests and responses – Subscriptions and indications – Example application • Discussion 1

A context management framework

• Motivation – Mobile device users’ desire • Access and manipulation of the users’ information • Services specific to their location, time, and environment – Gathering context information from various sources • Sensors, networks, device status, user profiles, and other sources – Enhancing mobile applications’ usability • Letting the applications adapt to the users’ conditions • Problems to achieve true context awareness – Uncertain, rapidly changing, partially true data from multiple heterogeneous sources – Extracting relevant context information by fusing data from several sensors – Many factors such as noise, faulty connections, drift, miscalibration, wear and tear, humidity to degrade data acquisition 2

A context management framework

• Problems to achieve true context awareness – Overlapping extracted contexts, changing contexts with time, only partially reliable approximations • Mobile devices’ dynamic environment – Learning from multidimensional data • Impossible easy generalization beyond training data – Occurrence of problems using even sufficiently reliable contexts • Users with different ideas of “context” • Proposed approach to address the challenges – A uniform mobile terminal software framework • Providing systematic methods for acquiring and processing useful context information from a user’s surroundings • Giving the context to applications 3

A context management framework

• Framework recognizing semantic contexts in real time – Dealing with uncertain, noisy, and rapidly changing information – Delivering contexts for the terminal applications in an event-based manner – Application programming interface (API) for using semantic context information – Using an expandable context ontology to define contexts • Framework properties – Blackboard-based approach underlying communication paradigm between framework entities – Focusing on mobile terminal capabilities rather than infrastructure – Framework for the Symbian platform to achieve true device mobility, high performance, and a broad user base 4

A context management framework

• Four main functional entities of the context framework – Context manager – Resource server – Context recognition service – Application • Context manager – A central server for providing services to other entities – Running on the mobile device itself • Black board-based context manager – Central node storing context information from any source – Serving context to clients in three ways • Direct querying the manager to gain context data • Subscription to various context change notification services • Using higher-level (composite) contexts transparently 5

A context management framework

• Resource server – Connecting to any context data source – Posting context information to the context manager’s blackboard – Delivering context to the clients according to their subscriptions • Delivered data’s abstraction level → high enough • Delivered data’s frequency → low enough (not overwhelming) – Performing low-level recognition before delivering the data • Low-level context recognition process flow in the resource server • Measurement phase – Reading the sensors and output raw data • The preprocessing phase builds measurement data arrays that contain a certain number of samples (quantization of time dimension) and calculates generic features for each time interval.

6

A context management framework

• Resource server – Performing low-level recognition before delivering the data • Preprocessing phase – Building measurement data arrays that contain a certain number of samples (quantization of time dimension) – Calculation of generic features for each time interval • Feature extraction – Calculation of more specific context features • Quantization phase – Binding the feature values to the real-world context which has a meaning for a person or an application according to a predefined ontology – Using fuzzy sets or crisp limits for quantizing extracted features – Outputs context atoms (base context units) that applications can use or recognition services can refine 7

A context management framework

• Resource server – Two methods for quantization • Set crisp limits in a true-false labeling with fuzzy membership functions denoted as μ(x) • Quantizing environment sound intensity – Three quantities (Silent, Moderate, and Loud) corresponding to the three membership functions • If one of these is true, the others are false.

8

A context management framework

• Resource server – Two methods for quantization • Apply a fuzzy set for features, resulting in continuous valued fuzzy labeling • μL(x) = 0.7 / Silent + 0.3 / Moderate + 0 / Loud 9

A context management framework

• Context recognition services – Sharing recognized higher-level contexts – Addition and removal of recognition services by developers or processes – Using a set at a certain time instant or a time series of context atoms as input – Returning single higher-level contexts for the context manager • Summary of framework – Resource server and recognition service • Converting an unstructured raw measurement data flow into a representation defined in the context ontology (human-interpretable context information) for the applications in an event-based manner – Framework providing a semantic interface that enables more systematic and rapid application development – Efficient reuse of context information compared to using raw measurement data 10

A context management framework

• Blocks for change detection and security – Not focusing on the functionalities – Any change detection services to detect context change – Security module for checking the trustworthiness of incoming contexts 11

Information sources

• Acquiring context information from many possible sources – Using a microphone, three accelerometers, two channels for light, and sensors for temperature, humidity, and touch – Sensor data provided the starting point • Device processes that represent mobile devices’ internal information, such as applications currently running • The Internet and other networked resources • Global Positioning System, among the most important sources of location information • Internal device processes, which provide explicit information on device application use and users’ scheduled tasks, preferences, and social networks • Time information, used to associate certain events with others and form event sequences that might relate to a current higher-level context or predict a future event – Providing continuous and ad hoc connectivity to local and remote data • Supporting various wireless networks, such as GSM and GPRS, and short-range ad hoc networks, such as Bluetooth 12

Ontology for sensor-based context information

• A common structure for representing information – Managing the context information systematically – Design of an ontology for representing sensor-based context information • A schema for ontology – A client-usable, extendable vocabulary that presents the terms for describing context information – Using the Resource Description Framework (www.w3c.org) as the description syntax 13

Ontology for sensor-based context information

• Description of each context using six properties – Containing at least Context type and Context value – Context type • Referring to the context category • Primary parameter of all subscriptions and queries – Context value • Referring to the semantic or absolute “value” of context type • Usually being used together with context type • Containing an absolute numerical value or feature describing context in some cases – Confidence • An optional property which describes the context’s uncertainty • A probability or a fuzzy membership of context depending on the source – Source • Description of the context’s (semantic) source 14

Ontology for sensor-based context information

• Description of each context using six properties – Timestamp • Denoting the latest time the context occurred – Attributes • Specifying the context expression freely • Containing any additional details not included in the other properties • Higher-level context and context atoms links to a recognition service – Containing the Input context type set for recognizing the higher-level context • Three properties for describing recognition services: – Context type • Using the category of the higher-level context as a normal context atom to request the recognition for the type 15

Ontology for sensor-based context information

• Three properties for describing recognition services: – Context recognition service • Receiving input defined in Input context type set • Performing the classification, and returning the higher-level context – Input context type set • Subscription of the input context type set to be notified of the changes • Inferring the higher level context upon change in input type set • Seven context values in higher-level contexts from a set of 47 audio based context atoms 16

Ontology for sensor-based context information

• Management of uncertainty of sensor-based information – Confidence property • Description of context uncertainty: a context derived from multiple sensory input only holds true with a certain probability – Partly true context • Boundary between two contexts is not clear and discrete 17

Inference mechanism concept

• The conceptual blackboard-based system – Use of any inference framework for reasoning new contexts or actions • Context uncertainty described in the confidence property – Useful in such inference frameworks as probabilistic networks, clustering, or case based reasoning • Recognition services – Input : information from the common data space – Output : higher-level contexts from either a set of contexts at a certain moment or a context history 18

Bayesian reasoning for higher level contexts

• Using a naïve Bayes classifier to recognize higher-level contexts • Naïve Bayes classifier for online context inference and sensor fusion – Robust method with missing, uncertain, and incomplete information • Possible to use context data described by the ontology, a vector of context atom confidence values • Treating fuzzy membership values as virtual evidence – Computationally efficient • Training and inference both have a linear complexity in input data size – No requirement of background information for modeling • Exception for choosing the relevant network inputs • Examples of contexts to be recognized by a naive Bayes classifier 19

Bayesian reasoning for higher level contexts

20

Bayesian reasoning for higher level contexts

• Higher-level context Outdoors from the context atoms • Indoor/Outdoor Bayesian network – Classification of the current situation’s context atom confidence values (vector) – Conditional probability indicating each input’s probability given the output – Learning conditional probabilities from the training data • Using naïve Bayesian networks to classify 9 contexts of a mobile device user in their normal daily activities – Nine contexts measured from a continuous real-life scenario containing different activities such as driving a car, running, walking, using an elevator, listening to different kinds of music, and speaking – The measurement system hardware • A small sensor box attached to the shoulder strap of a backpack containing a laptop 21

Bayesian reasoning for higher level contexts

• The classification results – Indication that naïve Bayes classifier can extract situations fairly well – Valid only in a restricted scenario • Experiments under controlled scenarios – Nine different contexts, and measured by four people – Car, Elevator, and Tapwater with nearly 100 percent accuracy – Supervised learning approach • which defines the target classes and their inputs before training 22

Application programming interface

• Context manager API – Implemented on the Symbian platform 23

Adding context

• Processing and collecting the context properties by resource servers • Method for adding context: ContextAdd( ContextObject ) 24

Requests and responses

• Request of context information from the context manager database – Exposition of context ontology to application developer – Request context information by the context ontology tree (sub branch) – Response containing all contexts under the sub branch – Denoting responses to requests with an arrow (→) ContextRequest( Environment:Light:Intensity ) →Bright ContextRequest( Environment:Light ) →{Bright, Natural, NotAvailable} ContextSetRequest( {Environment:Humidity, Device:Activity:Position} →{Dry, AntennaUp} ContextSetRecognitionRequest( Environment:Sound:Type ) →Car 25

Subscriptions and indications

• Giving applications the required context information in an event-based manner – Subscription to context change notifications • Examples illustrate how applications use this service ContextChangeSubscribe(Device:Activity:Placement) →AtHand ContextChangeSubscribe( Environment:Temperature:Absolute ) →21 ContextStartSubscribe( Location:Facility, MovieTheatre) →MovieTheatre ContextChangeSubscribe( Environment:Sound:Type ) →Elevator 26

Example application

• Using the context ontology to demonstrate the use of sensor-based context information in mobile terminal applications – Such as browsing and information presentation in the user interface – Environment:Light:Intensity, Environment:Sound:Intensity, and User:Activity:PeriodicMovement • Environment sound intensity – Control of the volume of operating tones • User activity and ambient light level – Adaptation of font size, screen brightness, and service content • Context-based information representations of a bus timetable service 27

Example application

• Using fuzzy rules to help guide the application adaptation • Fuzzy rule definition for adapting font size – Using the logical operation AND • Intersection between two fuzzy sets • Environment:Light:Intensity and User:Activity:PeriodicMovement: μ FONTSIZE (x) = min(μ USER:ACTIVITY:PERIODICMOVEMENT(x), μ ENVIRONMENT:LIGHT:INTENSITY (x)).

– Using the logical operations OR and NOT 28

Discussion

• Framework and context ontology – Providing a semantic application programming interface for handling imprecise information from multiple sources – Many remaining obstacles to the rapid use of many contexts derived from sensor data – Need for more sensors and a large data collection to gain enough information • Several advantages of high-level context – Event based communication of high-abstraction-level data – Communicating low-level data up to a client continuously • Consumption of more processing power and the limited battery resources. – Optimal use of the context manager storage 29