Transcript ppt - Johnny Chung Lee

Using a Low-Cost
Electroencephalograph for Task
Classification in HCI Research
Johnny C. Lee
Carnegie Mellon University
Desney S. Tan
Microsoft Research
UIST 2006, Montreux Switzerland
National Geographic, March 2005
NY Times Magazine, October 16, 2005
Brain-Computer Interfaces (BCI)
A direct technological interface between a brain and
computer not requiring any motor output from the user
Example Conferences/Journals with BCI interests:
Neural Information Processing Systems (NIPS)
IEEE Transactions on Biomedical Engineering
IEEE Transactions on Neural Systems and Rehabilitation Engineering
Why is this relevant to UIST or HCI?
BCI research traditionally focuses on exploratory
neuroscience and rehabilitation engineering.
Brain sensing could provide valuable data about:
- engagement
- cognitive work load
- surprise
- satisfaction
- frustration
Potentially helpful to
Context Sensitive or Evaluation Systems
Values of BCI
Values of HCI
To use any means necessary to
demonstrate that brain-computer
interaction is possible.
To use reasonable means to
achieve a practical benefit to
many users.
We’d like to:
It is okay to:
use equipment costing
$100K to +$1 million USD
VS
use fairly affordable and
accessible equipment
use highly invasive surgical
procedures
be safe for repeated and
extended use
require hours or days of
operant conditioning
be usable without requiring
significant user training
remove data from poor
performing subjects
use data from all subjects to
evaluate its performance
Where do we start?
Brain Sensing/Imaging Technologies
MRI
CT
ECoG
SPECT
Currently Impractical
PET
for HCI
MEG
fMRI
EROS/fNIR
EEG - Safe, easy, no medical expertise
EEG – Electroencephalograph
the neurophysiological measurement of the electrical activity
of the brain by recording from electrodes placed on the scalp
(skipping the lower level neurophysiology)
- Measures the voltage difference between two locations on the scalp
- Only picks up gross, macroscopic, coordinated, and synchronized firing of
neurons near the surface of the brain with perpendicular orientation to the
scalp. (thus majority of activity is hidden)
Analogous to holding a thermometer up to the side of a PC case
EEG Devices
Manufacturer: EGI Systems
Channels: 128-512
Cost: $100K-$250K USD
Manufacturer: BioSemi
Channels: 64-128
Cost: ~$30K USD
The Brainmaster
•
•
•
•
Lowest cost FDA approved device
Designed for home and small clinical use.
Only $1500 USD
Specs:
– 2-channels
– 8-bit at 4µV resolution
– 256 samples/sec
• Has yet to be validated for BCI research work.
• If it works, it lowers the entry bar for BCI research.
Validating the Device (and ourselves)
1. Validate the device
Can we get useful data from such a low-end device?
2. Validate ourselves
To explore this space, we must be able to
collect our own data.
Validating the Device (and ourselves)
Keirn, Z., “A New Mode of Communication Between Man
and His Surroundings”, IEEE Transactions on
Biomedical Engineering, Vol. 37, No. 12, 1990.
• Data is available for download
• Data has not been reproduced in the past 15 years
– Some computational BCI researchers have just used this data.
– State of the art does is not a great deal better.
Reproducing the Keirn Data
We adapted procedure from Keirn to better control
potential confounds.
3 tasks:
• Rest (Baseline): Relaxation and clearing of mind
• Math: Mental arithmetic, prompted with “7 times 3 8 5”
• Rotation: Mentally rotate an object, prompted with “peacock”
Tasks from the original paper were designed to elicit
hemispheric differences.
Experimental Procedure
User is instructed to keep eyes closed, minimize body
movement, and not to vocalize part of the tasks.
For each task, a computer driven cue is given:
Rest, Math, Rotate
Following Math and Rotate, the experimenter says
either the math problem or object
Experimental Procedure
Block design adapted from Kiern
task (14 seconds)
Rot
Math
Rest
Math
Rest
Rot
Rest
Rot
Math
Rot
Rest
Math
Rest
Math
Rot
Math
Rot
Rest
session
trial
Experimental Procedure
Rot
Math
Rest
Rot
Math
Rest
Rot
Math
Rest
Math
Rest
Rot
Math
Rest
Rot
Math
Rest
Rot
Rest
Rot
Math
Rest
Rot
Math
Rest
Rot
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Rest
Math
Rest
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Rest
3 sessions per subject
Many short tasks prevent correlation with EEG drift
Experimental Procedure
Rot
Math
Rest
Rot
Math
Rest
Rot
Math
Rest
Math
Rest
Rot
Math
Rest
Rot
Math
Rest
Rot
Rest
Rot
Math
Rest
Rot
Math
Rest
Rot
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Rest
Math
Rest
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Math
Rot
Rest
Math
Rot
Rest
Math
Rot
Rest
Subjects:
8 subjects (3 female)
29-58 years of age
All were cognitively and neurologically healthy
All right handed
EEG Setup
International 10-20 EEG electrode
placement system
Two channels placed on P3 and P4 with
both references tied to Cz.
Electrodes are held in place using
conductive paste.
5-10 minute preparation.
Processing the Data
Data Processing
task (14 seconds)
Rot
14 secs
Data Processing
task (14 seconds)
Rot
14 secs
Task Cue
Data Processing
task (14 seconds)
Rot
14 secs
Experimenter Prompt
Data Processing
task (14 seconds)
Rot
14 secs
Task Onset
Data Processing
task (14 seconds)
Rot
14 secs
Performing Task
Data Processing
task (14 seconds)
Rot
14 secs
~4 secs
Performing Task
Data Processing
task (14 seconds)
Rot
10 secs
Performing Task
Removing time for machine learning
Most machine learning algorithms don’t
handle time series data very well.
30
20
10
0
-10
-20
-30
0
500
1000
1500
10 seconds
2000
2500
3000
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec
• Perform signal processing on each of the 9 windows to get our
“time less” feature set
30
20
10
0
-10
-20
-30
0
500
2 secs
1000
1500
2000
2500
3000
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec
• Perform signal processing on each of the 9 windows to get our
“time less” feature set
30
20
10
0
-10
-20
-30
0
500
2 secs
1000
1500
2000
2500
3000
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec
• Perform signal processing on each of the 9 windows to get our
“time less” feature set
30
20
10
0
-10
-20
-30
0
500
1000
2 secs
1500
2000
2500
3000
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec
• Perform signal processing on each of the 9 windows to get our
“time less” feature set
30
20
10
0
-10
-20
-30
0
500
1000
2 secs
1500
2000
2500
3000
Removing time for machine learning
• Divide the 10 seconds into 2 sec windows that overlap by 1 sec
• Perform signal processing on each of the 9 windows to get our
“time less” feature set
30
20
10
0
-10
-20
-30
0
500
1000
1500
2 secs
2000
This provides
486 windows
per participant
2500
3000
Signal features for each window
Generic signal features such as mean power,
peak frequency, peak frequency amplitude, etc.
Features frequently used in EEG signal analysis.
Common EEG Features
30
20
10
Raw EEG
0
-10
-20
-30
0
500
1000
1500
2000
2500
3000
5000
4000
3000
Spectral
Power
2000
1000
0
0
5
10
15
20
25
30
35
40
Theta
Beta Low
Beta High
Gamma
(4Hz-8Hz)
(12Hz-20Hz)
(20Hz-30Hz)
(30Hz-50Hz)
Delta
Alpha
(1Hz-4Hz)
(8Hz-12Hz)
45
50
Feature Processing and Selection
The 39 base features from each window are
mathematically combined to create 1521 total
features.
We used a feature preparation and selection
process similar to [Fogarty CHI’05] to reduce the
number of features:
23 features for 3-task classification (486 examples)
16.4 features for pair-wise classification (324 examples)
Baseline Results – 3 cognitive tasks
BayesNet classifier
3 task
Math v.
Rotate
Rest v.
Math
Rest v.
Rotate
user 1
67.9%
83.3%
88.0%
85.8%
user 2
70.6%
82.7%
91.4%
84.3%
user 3
77.6%
88.3%
93.8%
86.7%
user 4
63.6%
69.4%
84.9%
86.7%
user 5
66.5%
91.0%
81.2%
80.9%
user 6
59.3%
80.6%
80.2%
68.5%
user 7
71.4%
87.3%
90.4%
86.7%
user 8
69.8%
87.7%
82.4%
83.6%
Average
68.3%
83.8%
86.5%
82.9%
Chance:
33.3%
50%
50%
50%
30
20
10
0
-10
-20
-30
0
500
1000
2 secs
1500
2000
2500
3000
We can do better…
30
20
10
0
-10
-20
-30
0
500
1000
???
1500
2000
2500
3000
Throwing time back in…
30
20
10
0
-10
-20
-30
0
500
1000
“Math”
1500
2000
2500
3000
We can average over
temporally adjacent windows
to improve classification
accuracy
Averaging with Task Transitions
Task transitions result in conflicting data in averaging window.
High density of transitions will result in lower accuracy.
Averaging with Task Transitions
Fewer task transitions will yield better classification accuracy.
Averaging with Task Transitions
No transitions and averaging over all data will be the even better.
Classification Accuracy with Averaging
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
100%
95%
90%
85%
80%
+5.1 to +15.7%
for 3-tasks
75%
70%
3 tasks
65%
Math v. Rot
Rest v. Rot
60%
Rest v. Math
55%
50%
Baseline
No Averaging
5 windows
with transitions
5 windows
no transitions
Error bars represent standard deviation
9 windows
no transitions
So, can we really read minds?
Mean Classification Accuracy vs. Averaging Scenarios (Mental Tasks)
100%
95%
90%
85%
80%
75%
Possibly not, we might be 3really
tasks
detecting subtle motor movements….
Math v. Rot
70%
65%
Rest v. Rot
60%
Rest v. Math
55%
50%
Baseline
No Averaging
5 windows
with transitions
5 windows
no transitions
Error bars represent standard deviation
9 windows
no transitions
Cognitive/Motor “Fabric”
Tasks of varying cognitive difficultly are involuntarily
coupled with physiological responses, such as minute
imperceptible motor activity. [Kramer ’91]
Therefore, it is impossible to completely isolate cognitive
activity neurologically intact individuals.
Does this matter to neuroscience? Yes
Does this matter to HCI?
Maybe not
Cognitive/Motor “Fabric”
If motor artifacts are reliably correlated with different types of
tasks or engagement, why not use those to help the classifier?
Requiring users to not move is also very impractical.
Non-Cognitive Artifacts detected by EEG:
–
–
–
–
–
–
–
Blinking
Eye movement
Head movement
Scalpal GSR
Jaw and facial EMG
Gross limb movements
Sensory Response Potentials
Experiment 2 – Game Task
To explore this idea of using non-cognitive artifacts to classify tasks
using EEG, we chose a PC-based video game task.
Halo, a PC-based first person shooter
game produced by Microsoft Game
Studios.
Navigate a 3D environment in an effort to
shoot opponents using various weapons.
Relatively high degree of interaction with
mouse and keyboard input controls.
Game Tasks
Rest – baseline rest task, relax, fixate eyes on cross hairs on center of
screen, do not interact with controls. Game elements do not interact with
participant.
Solo – navigate environment, interact with elements in the scene, and
collect ammunition. Opponent controlled by expert did not interact with
participant.
Play – navigate environment and engage opponent controlled by expert.
Expert instructed to play at a level just slightly above skill of participant to
optimally challenge them.
Game Experimental Procedure
Setup, design, and procedure was similar to first study.
• Participants had tutorial and practice time with game controls.
• 3 tasks repeated 6 times (counterbalanced)
• Tasks were 24 seconds to allow navigation time.
• Only 2 sessions were run for each participant
• Same 8 participants from first study were run in this study.
• Same data preparation and machine learning procedure.
Results – Game Tasks
Mean Classification Accuracy vs. Averaging Scenarios (Game Tasks)
93.1%
100%
95%
90%
85%
80%
75%
3 tasks
70%
Rest v. Play
65%
Rest v. Solo
60%
Solo v. Play
55%
50%
Baseline
No Averaging
5 windows
with transitions
5 windows
no transitions
Error bars represent standard deviation
19 windows
no transitions
Conclusion
This experimental design and data processing procedure can
be applied to a much wider range of applications/tasks. Our
two experiments were just two examples at different ends of
a spectrum.
Compelling results can be achieved with low-cost equipment
and without significant medical expertise or training.
Non-cognitive artifacts (inevitable in realistic computing
scenarios) can be embraced improve classification power.
To make BCI relevant to HCI, we must challenge traditional
assumptions and creatively work with its limitations.
Thanks!
Johnny Chung Lee
[email protected]
Desney Tan
[email protected]
Thanks to MSR and the VIBE
Group for supporting this work.
Cross-user Classifier
3 Cognitive Tasks
100.00%
95.00%
3class
90.00%
MathRotate
85.00%
RestRotate
80.00%
RestMath
75.00%
70.00%
65.00%
60.00%
55.00%
50.00%
45.00%
40.00%
Base
5 win
5win no edges
Known
Cross-user Classifier
3 game tasks
100.00%
95.00%
90.00%
85.00%
3class
80.00%
SoloPlay
RestPlay
75.00%
RestSolo
70.00%
65.00%
60.00%
55.00%
50.00%
Base
5 win
5win no edges
Known
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG
SPECT
PET
MEG
fMRI
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG
fMRI
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR
EEG
Brain Sensing/Imaging Technologies
MRI – only anatomical data
CT – only anatomical data
ECoG – highly invasive surgery
SPECT – radiation exposure
PET – radiation exposure
MEG – extremely expensive
fMRI – extremely expensive
EROS/fNIR – currently expensive, still in infancy
EEG – safe, easy, no medical expertise
Other cool things you can do
with an EEG device…
Event Related Potentials (ERP)
• Electrical activity related to or in response to
the presentation of a stimulus
• Very well studied
• Relatively robust
• Used daily in clinical settings to check sensory
mechanisms, typically in infants
• Requires averaging over 30-100 windows
synchronized with to see response.
ERP - AEP
ERP: Auditory Evoked Potential
• Used in clinics/hospitals to check hearing.
• Response to clicks in the ear
AEP response
AEP response
Bold Lines = no clicks
Thin Line = with clicks
ERP - VEP
ERP: Visual Evoked Potential
• Focusing on a flashing target, the visual cortex will “resonate” at the
stimulus frequency.
Stimulus Frequency
Harmonics
ERP – Auditory and Visual P300
•
•
•
Well known/studied potential related to “attention” or
“surprise”
Presented with 2 stimuli and instructed to count one of the
stimuli
Positive response will follow the stimulus of interest.
Side note: EEG as ECG
ECG - Electrocardiogram
0.4 µV units
• placing an electrode on the chest provides a clear measure of
cardiac activity.
• translation to BPM is a simple autocorrelation
Single Beat period
Heart beats
EEG as EMG
EMG - Electromyography
• Measures muscular activity
Wrist relaxation (return to straight position)
0.4 µV units
Wrist rest state
Tension holding
Wrist inward contraction (toward inner forearm)
NOTE: The magnitude of the spikes seem to be proportional to the acceleration involved with the movement.
EEG as Blink Detector
Blinks
Blinks
200
150
100
50
0
-50
-100
-150
-200
0
0.5
1
1.5
2
2.5
4
x 10
• Electrical activity due to muscle movement involved with eye
blinks propagate through the head.
• Similarly, eye movements also affect the EEG recording
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning
Pattern Recognition
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning
Pattern Recognition
Human learns how
the machine works
Machine learns how
the human works
Task Classification Background
• Previous work is split primarily into two camps:
Operant Conditioning
Pattern Recognition
Human learns how
the machine works
Machine learns how
the human works
Relatively new
Early dabbling in the late-80’s
Most work has happened in last 5 years.
Pattern Recognition
Data Collection &
Signal Processing &
Machine Learning &
Experimental Design
Feature Generation
Improving Accuracy
EEG Setup
International 10-20 EEG electrode
placement system
Two channels placed on P3 and P4 with
both references tied to Cz. Locations
selected based on pilot recordings.
Attaching electrodes: Prepare the site
with a cleaner, use conductive paste
to improve connection and hold
electrode in place.
P
Paste rinses out with water, non-toxic.
5-10 minute preparation.