The PAM project Personalised Ambient Monitoring: aiding those with Bipolar Disorder Sally Brailsford John Crowe Christopher James Evan Magill The PAM project The PAM project Enabling health, independence and wellbeing.
Download ReportTranscript The PAM project Personalised Ambient Monitoring: aiding those with Bipolar Disorder Sally Brailsford John Crowe Christopher James Evan Magill The PAM project The PAM project Enabling health, independence and wellbeing.
Slide 1
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 2
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 3
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 4
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 5
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 6
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 7
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 8
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 9
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 10
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 11
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 12
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 13
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 14
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 15
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 16
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 17
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 18
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 19
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 20
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 21
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 22
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 23
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 24
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 25
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 26
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 27
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 28
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 29
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 30
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 31
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 32
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 33
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 34
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 35
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 36
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 37
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 38
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 39
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 40
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 41
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 42
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 43
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 44
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 45
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 46
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 47
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 48
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 49
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 50
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 51
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 52
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 53
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 54
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 55
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 56
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 57
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 58
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 59
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 60
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 61
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 62
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 63
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 64
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 65
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 66
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 67
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 68
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 69
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 70
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 71
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
50.936348, -1.393458, 0.0, 4.0
-0.5083, 1.7986, 0.0782
-0.1173, 1.1339, 0.2346
-0.0782, 0.8993, 0.1173
2.0, 0.0
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 2
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 3
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 4
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 5
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 6
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 7
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 8
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 9
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 10
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 11
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 12
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 13
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 14
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 15
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 16
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 17
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 18
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 19
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 20
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 21
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 22
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 23
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 24
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 25
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 26
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 27
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 28
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 29
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 30
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 31
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 32
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 33
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 34
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 35
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 36
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 37
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 38
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 39
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 40
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 41
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 42
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 43
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 44
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 45
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 46
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 47
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 48
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 49
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 50
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 51
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 52
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 53
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 54
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 55
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 56
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 57
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 58
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 59
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 60
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 61
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 62
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 63
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 64
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 65
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 66
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 67
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 68
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 69
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 70
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72
Slide 71
The PAM project
Personalised Ambient Monitoring:
aiding those with Bipolar Disorder
Sally Brailsford
John Crowe
Christopher James
Evan Magill
The PAM project
The PAM project
Enabling health, independence and
wellbeing for psychiatric patients through
Personalised Ambient Monitoring
2
The PAM project
A sandpit project
• Funded by the Engineering and Physical Sciences Research
Council
• Sandpit theme: “Bringing Care to the Patient”
3
The PAM project
The PAM team
• Sally Brailsford, Southampton
Behavioural
Analysis
• John Crowe, Nottingham
• Christopher James, Southampton (PI)
• Evan Magill, Stirling
Sensors
PAM
Ambient
Monitoring
• plus 4 PhD students
– Syed Mohiuddin, Pawel Prociow,
James Amor and Jesse Blum
Operational
Research
4
The PAM project
PAM external steering group
• Ms Sylvia Wyatt, Future Healthcare Network, NHS Scotland
• Dr Amy Drahota, Research Fellow, University of Portsmouth
• Mr Peter Jones, Community mental health nurse, Lancashire Care NHS
Trust
• Dr Paul Courtney, Consultant Psychiatrist, Hampshire Partnership
Trust
• Mr Richard Barritt, Chief Executive, Solent MIND
• Mr James Stubbs, Service User Representative
5
The PAM project
The aims of PAM
• To build a system of unobtrusive sensors, linked (through a
standard mobile phone) to a remote computer system,
which automatically monitors the activity patterns of
people with mental health problems
• To determine whether it is possible to use such a system to
obtain ‘activity signatures’ in a manner which is acceptable
to the patient and can provide useful information about the
trajectory of their health status
• And if this is so, to determine how this information can best
be used to maintain health and aid independence
6
The PAM project
Bipolar Disorder
• Severely disabling mental illness which affects functionality,
relationships, employment and quality of life; affects 2% of the
UK population (MHF, 2006)
• Bipolar disorder is the 6th most common disabling illness
worldwide (WHO, 2004)
• In 2002, the estimated annual cost to the UK NHS of managing
bipolar disorder was £199M, of which £70M was spent on
hospital admissions (Gupta and Guest, 2002)
• Many pharmacological treatments are available but these can
have unpleasant side-effects and adherence is often poor, leading
to hospital admission
7
The PAM project
How does BD alter lifestyle?
Manic
Depressive
Euphoric behaviour
Low mood
Increased (excessive)
social activity
Lack of interest in social
interaction
Psychomotor agitation
Psychomotor retardation
Sleep deprivation
Insomnia
Flight of ideas
Concentration problems
The PAM project
Managing Bipolar Disorder
• Most patients want to manage their own condition, using
medication only when necessary
• Motivated patients of above-average intelligence, interested
in self care and independence
• Early warning signs or prodromes can be detected while
patient is still “self-aware” and can take action (seek
medical help, start medication etc) to avoid hospital
admission
• Paper-based “mood diaries” shown to be effective in trials
9
The PAM project
Problems with paper-based systems
• Do not provide a sense of control over daily life
• Patients complain about vigilance and energy required
• Problems with accuracy, completeness and honesty of
patient-reported data
• Patients may forget to document important details
• Comorbidity and drug response go unmeasured
• No reduction in depressive relapses (Perry et al, 1999)
10
The PAM project
The aim of PAM
• To use a system of electronic sensors to provide an
automated equivalent of a mood diary, which alerts the
patient to a change in activity pattern which could signal
the onset of a bipolar episode
• Patient would be sent an SMS alerting them to a possible
change, which they could then act on (if they chose)
• PAM is mainly aimed at people who live alone
• Aim is to identify a baseline “activity signature” and then
identify significant deviations from this
11
Device Nodes
• Worn
– Mobile Phone
• Questionnaire
• Gateway Application
– GPS Transceiver
– Wearable Accelerometer
– Wearable Microphone
– Wearable Light Sensor
• Environmental
–
–
–
–
–
–
–
Microphone
Light Sensor
Passive Infrared Sensors
Micro-switches
Bed Sensor
Camera
Infrared Receiver For Remote Control
The PAM project
The PAM project
Wearable sensor set
User input:
• General health
questionnaires
• Mood selfassessment
Wearable Node
•Acceleration
•General light level
•Artificial light level
•Ambient sound
properties
GSM
location
GPS
module
XYZ
accelerometer
Internal
accelerometer
Bluetooth
Encounters*
- Bluetooth
- 3G / GPRS
- User input
-Internal
Environmental sensor set
Environmental processing
unit
•Processing
•Storage
•Backup
•Upload
Wide-angle
Camera
PIR
sensors
Home appliances
monitoring
•Microwave
•Refrigerator
•Oven
Environmental Node
Monitoring of:
•Remote control activity
Main and cupboard doors.
•General light level
•Artificial light level
•Ambient sound.
- Bluetooth
The PAM project
Bed occupancy
sensor
- WiFi
- 433 MHz RF
The PAM project
Example data – wearable light levels
General light
Artificial light
Working
Bus awaiting
Bus awaiting
Commuting
Walking
Home
The PAM project
Threads of research activity
• The four centres collaborated across the project but we
gravitated towards independent themes (as required by the
four PhD students)
• accelerometry & behaviour analysis
• outside the home
• BD modelling
• rule-based sensor network
The PAM project
accelerometry & behaviour analysis
accelerometry & behaviour analysis
The PAM project
accelerometry & behaviour analysis
Accelerometry & Behavioural analysis
• Determine what a person is doing (sleeping, eating,
restlessly pacing around, etc) by feature extraction
algorithms on sensor data (e.g. the “neuroscale” algorithm)
• Develop an “activity signature” for that individual,
describing their normal activity pattern when well
• Develop a set of decision rules which determine whether an
individual’s current activity is “normal” – for them – or
may indicate the potential onset of a prodrome
19
The PAM project
Tri-axial accelerometry
accelerometry & behaviour analysis
Walk
12
Walk
25
walking
10
20
15
8
10
6
5
0
4
-5
2
-10
0
0
0.5
1
1.5
2
2.5
-15
-800
-700
-600
-500
-400
-300
-200
-100
0
100
-200
-100
0
100
4
x 10
Lecture
12
25
at a lecture
10
Lecture
20
15
8
Acceleration
10
6
5
0
4
-5
2
-10
0
0
2000
4000
6000
8000
Samples (1 Hz)
10000
12000
activity over time
14000
-15
-800
-700
-600
-500
-400
-300
clustered activity
20
The PAM project
outside the home
outside the home
The PAM project
Outside the home
Example: tracking movement & position
outside the home
Off-the-shelf GPS module
BT enabled accelerometer
logfile.txt
13
13
13
13
Feb
Feb
Feb
Feb
2010
2010
2010
2010
13:06:41;
13:06:42;
13:06:44;
13:06:45;
G;
A;
A;
A;
5256.0723; -112.181; 0.0;
0.044; -0.888; 0.484;
0.036; -0.892; 0.492;
0.036; -0.892; 0.496;
The PAM project
outside the home
Positional data and pre-processing
The PAM project
outside the home
Identifying meaningful locations
Activity data – Bluetooth
The PAM project
outside the home
• Participants on average encountered more than 1000
unique Bluetooth devices of which:
–
–
–
–
80% were one-off encounters
15% were “occasional” (1-10) encounters
4% were “frequent” (10-40) encounters
1% were “regular” (40 or more) encounters
• This data can be used to monitor social interactions and
enhance location information
The PAM project
BD modelling
BD Modelling
The PAM project
BD modelling
Operational Research modelling of PAM
• Aim is to develop a “natural history” model for BD and use
it to test the sensitivity and specificity of the PAM
algorithms for detecting change in a patient’s health status,
in the context of:– A random (personalised) selection of sensors
– Unknown reliability of the chosen sensors and the
computer network system
– Occasional failure (or deliberate removal) of a sensor
– Variety in patient behaviour, in all states of health
27
The PAM project
BD modelling
Challenges for modelling BD
• No OR modelling approach of BD in the literature, although
some Markov models for depression (Patten et al, 2005)
• No universally accepted staging models for BD found in the
medical literature
• Symptoms vary among patients ; and patients may exhibit
mixed behaviour (manic and depressed)
• Lack of easily measurable criteria
• Took advice from clinical psychiatrist on our Steering
Group
28
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Final state transition model
…
=0
=1
• The parameter represents mental health state: totally
depressed ( = 0), “normal” or healthy ( ≈ 0.5) and totally
manic ( = 1)
• Each day, with a certain probability, the person may either
stay in the same state, or progress to an adjacent state, in
steps of 0.01
30
The PAM project
BD modelling
An illustrative sample path for λ
1.2
1
Lambda values
0.8
0.6
0.4
0.2
0
0
100
200
300
Days
400
500
600
31
Normal
Depressed
Manic
Hours of sleep
6
10
2
Phone calls
4
1
12
The PAM project
BD modelling
Sleep
Phone
14.00
12.00
Hours / Calls
10.00
8.00
6.00
4.00
2.00
0.00
0.00
0.20
0.40
0.60
0.80
1.00
1.20
Lamda
32
The PAM project
BD modelling
PAM-detected physical activity levels
during various mood states
Physical Activity Levels (PAL)
PAM detected PAL
2.6
2.4
Physical Activity Levels
2.2
2
1.8
1.6
1.4
1.2
1
0
100
200
300
Days
400
500
600
33
The PAM project
rule-based sensor network
rule-based sensor networks
The PAM project
rule-based sensor network
Programming sensor networks (PROSEN)
• distribute rules to rule engines embedded in smart sensors
• flexible programming
• support for run-time updating of rules
• aids personalisation and changing mental states
• initial work in a wind farm setting ….
The PAM project
PROSEN & REED
rule-based sensor network
• REED (Rule Execution and Event Distribution):
– supports the distribution of rules and trigger events
– employs a rule-based paradigm :
• allows sensor networks to be programmed at run time
• allows allow sensor network behaviour to be changed
at run time
– allows subscribe-notify service to be constructed
– potential for processing, filtering and collating data
The PAM project
Communications paradigm
rule-based sensor network
• Low-level decision and event driven
• Interact by sending/receiving decisions and events
• Low-level decision:
–
Event received from:
• components in PN
• Neighbour PN
• Policy server
Test of a local
state
Executed if the condition
is true
• manipulate/store local
data
• generate events
• may generate low-level
decisions
The PAM project
REED Middleware architecture
Sensor diagnostics
Low-level AI
(“novelty” filter)
rule-based sensor network
Sensor controller
…
Middleware Interface
Decision Space management
Event
Initial default
decisions
<“power up”, true,
<“temp
sensor
reading
“sending
HELLO
update”,
“temp < -20”,
event”>
“send ‘temp too low’ event
to Policy server”>
Event
Function call
Decision Space
Decision
event
condition
action
…
event
condition action
Decision
Event
Decision
Operation System Interface
Operation
System
Processing
Storage
Communications
The PAM project
Mobile phone-centric
sensor-based care system
rule-based sensor network
39
The PAM project
rule-based sensor network
Backend – Gateway Connection
The PAM project
rule-based sensor network
Network Interface
The PAM project
rule-based sensor network
Mobile Phone Based Body Area Network
The PAM project
rule-based sensor network
PAM Sensor Reading (PSR)
...
The PAM project
rule-based sensor network
MOBILE RULE-BASED APPLICATIONS
• Custom Symbian S60 Java ME applications installed on the
mobile phone interfaced with m-Prolog. (Also Sony-Ericsson)
• PAM-Gateway
– Control data capture from wearable units (such as GPS,
accelerometer, ambient light and sound levels)
• PAM-Transfer
– Perform automatic mobile to PC data transmission
• PAM-Q
– Dynamically adjustable questionnaires
The PAM project
rule-based sensor network
RELIABILITY AND ACCEPTABILITY ISSUES
• Mobile phone battery life
• On-body gateway disconnection
• On-body device form factor issues
• Environmental sensor reliability issues
• Rule coherence
The PAM project
Internal GPS: 5 hours
@ 0.68 w
POWER ISSUES
rule-based sensor network
BT, but no storage:
7.5 hours @ 0.48 w
1.8
1.6
1.4
1
0.8
Profile 1
Profile 2
0.6
Profile 3
0.4
Profile 4
0.2
0
TIMESTAMP (s)
33.25
36.75
40.25
43.75
47.25
50.75
54.25
57.75
61.25
64.75
68.25
71.75
75.25
78.75
82.25
85.75
89.25
92.75
96.25
99.75
103.25
106.75
110.25
113.75
117.25
120.75
124.25
127.75
131.25
134.75
138.25
141.75
145.25
148.75
152.25
155.75
159.25
162.75
166.25
169.75
173.25
176.75
Power (W)
1.2
BT, and storage: 5.5
hours @ 0.67 w
No BT & no user
applications: 9
hours @ 0.41 w
The PAM project
rule-based sensor network
Rule Coherence
• when rules are:
– changing over time
– possibly unique for particular individuals
– originating from different stakeholders
• how can we ensure the integrity of the rules
– in particular the lack of conflicts between rules
47
Example: “traditional” feature interaction
The PAM project
rule-based sensor network
Alice
OCS
Bob
CFx
X
Charlie
• Alice cannot call Charlie
– Originating Call Screening (OCS)
• If Alice calls Bob
– Bob’s Call Forwarding transfers call to Charlie
48
The PAM project
classes of feature interactions
1.
MAI: Two (or more) features
control the same device
(Multiple Action Interaction)
Power
F
Saving
off
rule-based sensor network
heater
D
Env
F
cntrl
2.
STI: One event goes to
different services which
perform different conflicting
actions (Shared Trigger
Interaction)
temp
D
hot
airF
con
wind
F
cntrl
49
The PAM project
classes of feature interactions
3.
SAI: A service performs an action on a device which triggers another
feature. The chain might involve any number of links (Sequential
Action Interaction, Loops)
Env
F
Cntrl
4.
rule-based sensor network
close
blinds
move
D
!
alarm
F
MTI: The existence of one feature prevents the another one from
operating. (Missed Trigger Interaction)
Power
F
Saving
off
temp
D
cold
heat
F
cntrl
50
Conflict Analysis
The PAM project
rule-based sensor network
• Offline and online analysis
looking for conflicts
between device rules
• Like FI for call control
• Searching for 5 types of
conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were
developed to explore the
conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
51
The PAM project
rule-based sensor network
initial results
• 867 tests for combining:
– shared trigger,
– multiple action, and
– sequential action.
• that is; from 17 features against each other and
themselves across the three criteria.
• 410 conflicts detected
• currently being analysed for patterns
{jmb, ehm}@cs.stir.ac.uk
52
The PAM project
in conclusion
in conclusion
The PAM project
in conclusion
PAM in practice
• A short technical trial of PAM was been carried out on the
four PhD students
• NHS ethics approval obtained for a small patient study
(max 4 patients) of PAM: completed in Southampton but
still under way in Scotland
• Many practical aspects highlighted in clinical study!
• Further technical developments under discussion
• Further collaborations planned
54
The PAM project
in conclusion
Moving forward – other PAM like projects
• PSYCHE (Personalised monitoring SYstems for Care in
mental HEalth) project develops a personal, cost-effective,
multi-parametric monitoring system based on textile
platforms and portable sensing devices for the long term
and short term acquisition of data from selected class of
patients affected by mood disorders.
• http://www.psyche-project.org
55
MONARCA
The PAM project
in conclusion
• MONitoring, treAtment and pRediCtion of bipolAr disorder
episodes (Monarca)
• EC funding of near €4m
• An example of recently funded EU projects in this field.
• Monarca is investigating aspects of bipolar disorder disease
by adopting a holistic approach to its assessment, treatment
and self-management. The project focuses on objective
assessment and prediction of bipolar disorder episodes and
aims to advance the discovery of new markers for this
disease.
56
OPTIMI
The PAM project
in conclusion
• Online Predictive Tools for Intervention in Mental Illness
• EU funding
• The Neuroscience Institute at the University of Bristol.
• The aim is to develop tools to perform predictions based on
early identification of the onset of an illness by monitoring
poor coping behaviour.
• The system will study an individual's behaviour patterns
over a sustained period and spot any baseline changes
suggesting they are becoming unwell.
• Will use wearable sensors and sensors fitted to domestic
appliances to measure activity levels. EEG readings, voice
analysis and physical activity analysis will be used.
57
The PAM project
Thank you for listening
58
The PAM project
State-of-the-art Health Sensor Networks
• Wearable Sensor Networks & Body Sensor Networks for medical and
psychiatric monitoring is active an research area:
– Alarm-Net
– CodeBlue
– Care in the Community
– UbiMon
– MobiCare
– LiveNet
The PAM project
BD modelling
Initial conceptual model of BD
“Normal”
Manic
Depressed
The PAM project
BD modelling
Using λ to model behaviour
2 1
4 1 N 2 3 1 D M
2 2
2
N = value of parameter X when normal (λ close to 0.5)
D = value of parameter X when depressed (λ close to 0)
M = value of parameter X when manic (λ close to 1)
where X = number of phone calls made daily, or number of
hours of sleep per 24-hour period
61
The PAM project
BD modelling
Time t
in days
Individual’s activity when
normal, manic or depressed
Mental health
state λ(t)
Actual activity
on day t
PAM-detected
activity on day t
Sensor
accuracy and
reliability
Decision rules
Trigger alert?
The PAM project
BD modelling
Patient types (the P in PAM)
• Different people will accept different levels of monitoring
• Defined on the basis of prodromes rather than sensors
– Patient types 1 to 10 chose a selection of two different prodromes
– Patient types 11 to 19 chose a selection of three different prodromes
– Patient types 20 to 24 chose a selection of four different prodromes
– Patient type 25 chose all five
• Pragmatic choice given vast number of combinations of
actual sensors
63
The PAM project
BD modelling
Model outputs
• True positive alerts (TP) and false positive alerts (FP)
• True negatives (TN) and false negatives (FN)
• Average number of days to detect the onset of a depressive
episode (ODE)
• Average number of days to detect the onset of a manic
episode (OME)
• The ideal would be a very low FP, a very high TP, and very
low ODE and OME
64
The PAM project
BD modelling
Example Results for Dataset
Patient types
Choices of prodromes
Minimum no. of sensors
required
ODE
(days)
OME
(days)
TP (%)
FP (%)
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
5.90
2.64
87.48
2.12
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor
8.10
3.12
85.22
0.85
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Camera; Cupboard door
sensors
8.17
3.54
84.16
1.30
8.37
4.01
83.65
1.05
Activity level
Sleep
Type 25
Talkativeness
Social energy
Appetite
Activity level
Type 20
Sleep
Talkativeness
Social energy
Activity level
Type 22
Sleep
Social energy
Appetite
Activity level
Type 21
Sleep
Talkativeness
Accelerometer; GPS; TV usage sensor;
Pressure mat; Light sensor;
Microphone; Phone sensor; Camera;
Cupboard door sensors
Appetite
65
The PAM project
PAM Infrastructure Vision
The PAM project
BD modelling
Implications
• PAM was found to be inadequate for almost all the personalised choices
of two prodromes only
• PAM was found to be efficient for most choices of three prodromes.
• PAM was found to be less effective for a few specific combinations of
personalised prodromal choices, especially ‘sleep’, ‘talkativeness’ and
‘social energy’, or ‘talkativeness’, ‘social energy’ and ‘appetite’, because
these prodromes were associated with relatively few observable
behaviours
• To be able to effectively offer choices such as these, the PAM system
would need to increase the number of their associated observable
behaviours
67
The PAM project
example: Context Triggering System
1 % respond to changes upon receiving contextual information
2 cds_cts(Trigger,T) :3 T2 is T+1,
4 assert(happens(listen_for_connection,T)),
5 assert(happens(make_connection,T)),
6 assert(happens(receive_data,T)),
7 assert(happens(checks_data,T)),
8 assert(happens(listen_for_connection,T2)),
9 ((
10 holdsAt(message(Trigger), T2),
11 assert(initiates(checks_data,prompt(Trigger),T)),
12 assert(terminates(checks_data,message(Trigger),T))
13 );
{jmb, ehm}@cs.stir.ac.uk
14 assert(terminates(checks_data,message(Trigger),T))).
68
The PAM project
Conflict Analysis
• Offline and online analysis
looking for conflicts between
device rules
• Like FI for call control
• Searching for 5 types of conflict:
– STI, SAI, LI, MAI, MTI
• 12 case studies were developed
to explore the conflicts
Missed Trigger Interaction occurs when the
Context Triggering rules delay the activation
of a home gateway.
{jmb, ehm}@cs.stir.ac.uk
69
The PAM project
Detection Approach
• Prolog-based framework
• Evaluates pairs of feature
rules to determine whether
they are concordant or
conflict
Example diagram describing MTI
70
conflict detection rule
{jmb, ehm}@cs.stir.ac.uk
The PAM project
Device Priority Approach to Resolution
• Allows precedence across devices without their knowledge of each
other
• How it works
1.
Resolver receives a list of conflicts, device priorities and
device rules
•
Priorities are declared as ordered preference lists of
particular properties (such as power efficiency,
bandwidth minimisation, data integrity, etc)
• Rules may be listed for each property
2. Resolver determines rules that should be disabled
{jmb, ehm}@cs.stir.ac.uk
71
The PAM project
Analysis Results
MTI Case Study
Notification suppression
Notification suppression
MTI
Notification suppression
Response prompting
MTI
Response prompting
Notification suppression
MTI
Response prompting
Response prompting
MTI
SAI Case Study
Data Transfer
Data Transfer
SAI
Data Transfer
Data Redirect
SAI
Data Redirect
Data Transfer
SAI
Data Redirect
Data Redirect
Concordance
{jmb, ehm}@cs.stir.ac.uk
72