Transcript BlueEyes Human-Operator Monitoring System

BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
BlueEyes
Human-Operator Monitoring System
•System
design
•System
implementation
Team members
•Future
improvements
•Project work
summary
•System
demonstration
Team mentor
Paweł Kowalik
Piotr Kubiaczyk
Krystian Nowak
Stanisław Osiński
Tomasz Pużak
Jan Kniat, Ph. D.
Poznań University of Technology, Poland
http://www.put.poznan.pl
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Agenda
•
•
•
•
•
•
•
•
Motivation
What is BlueEyes (and what is not) ?
Physiological foundations
System design
System implementation
Future improvements
Project work summary
System demonstration
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Motivation
• Human error - a frequent reason of
catastrophes and ecological disasters
– temporal indisposition
– weariness
– oversight
• Monitoring of conscious
brain involvement
• Capable sensors available
• Lack of complex solution exploiting
wireless link technology, database systems
and AI algorithms
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
What is BlueEyes ?
• The complex solution for human-operator
monitoring:
– conscious brain involvement monitoring
– triggering user-defined actions
– data recording
What is BlueEyes not?
• Doesn’t predict nor interfere with
operator’s thoughts
• Cannot force directly the operator to work
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Who is BlueEyes for ?
• Generic control rooms
– Power station
– Captain bridge
– Flight control centers
– Operating theatres – anesthesiologists
• Common application
– A simpler system version for drivers
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
BlueEyes - benefits
• Prevention from dangerous incidents
• Minimization of
– ecological consequences
– financial loss
– a threat to a human life
• The reconstruction
of the course of
operator’s work
BlueEyes
Human
Operator
Monitoring
System
Physiological foundations
•Motivation
•What is
BlueEyes ?
Human brain
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Preconscious brain
Conscious brain
• alogical
• „automatic”
• manages the
knowledge
• logical reasoning
• makes decisions
BlueEyes
Human
Operator
Monitoring
System
Conscious brain - operation modes
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Exploratory mode
Planning mode
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Eye movements
• Retina - central vision (macula)
• Central vision and
the conscious brain
Saccade - a rapid
eye jump to a new
location assigned
by the conscious
attention process.
Saccades presence  Exploratory mode
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
The eye movement sensor
• Eye position measuring - direct infrared
oculography
• Oxy- and deoxyhemoglobin measurment
• Two axial
accelerometer
• Ambient light
sensor
•System
demonstration
Jazz Multisensor
BlueEyes
Human
Operator
Monitoring
System
Wires, wires, wires...
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
A serious limitation of:
– mobility
– comfort
– safety
BlueEyes
Human
Operator
Monitoring
System
Personal Area Network !
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Solution:
A wireless Personal Area
Network
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•Objectives
•System
overview
•DAU
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Design objectives
• A mobile data acquisition device
• A reliable real-time buffering, processing
and recording
• A clear visualization interface
• Mechanisms for creating custom data
analyzers
• To ensure a possibility of distributing the
processing among a number of computers
BlueEyes
Human
Operator
Monitoring
System
System overview
•Motivation
•What is
BlueEyes ?
Data Acquisition Unit
Central System Unit
•Physiological
foundations
•System
design
•Objectives
Atmel 89C52
microcontroller
•System
overview
Jazz
Multisensor
Bluetooth
Bluetooth
Connection
Manager
Visualization
Module
Data Analysis
Data
Logger
•DAU
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Bluetooth technology provides
means for creating a Personal
Area Network linking the
operators and the central
system.
BlueEyes
Human
Operator
Monitoring
System
DAU - features
•Motivation
•What is
BlueEyes ?
Data Acquisition Unit
•Physiological
foundations
•System
design
•Objectives
Atmel 89C52
microcontroller
•System
overview
Jazz
Multisensor
•DAU
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Bluetooth
• Lightweight
• Runs on batteries - low
power consumption
• Easy to use - does not
disturb the operator
working
• ID cards for operator
authorization
• Voice transmission using
hardware PCM codec
BlueEyes
Human
Operator
Monitoring
System
CSU - features
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•Objectives
•System
overview
•DAU
• Connection
management
• Data processing
• Visualization
• Data recording
Central System Unit
Bluetooth
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
• Access
verification
• System maintenance
Connection
Manager
Visualization
Module
Data Analysis
Data
Logger
BlueEyes
Human
Operator
Monitoring
System
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•Objectives
• Saccade detector and conscious brain
involvement level calculation
Raw eye
movement data
•System
overview
•DAU
•CSU
Differentiation
•System
demonstration
8000
16000
24000
32000
Time [ms]
Visual attention level
60
Attention level (Lva)
•Project work
summary
saccade
250
200
150
100
50
0
-50
-100
-150
-200
-250
0
•System
implementation
•Future
improvements
Eye movement velocity
Velocity
•Motivation
CSU - saccade detector
50
Conscious
brain
involvement
level
40
30
20
10
0
0
8000
16000
Time [ms]
24000
32000
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•Objectives
•System
overview
CSU - Visualization Module
• Raw and processed data visualization
using:
– VU-meters
– pie-charts
– time series
•DAU
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
• audio and video
data playback
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•Objectives
•System
overview
•DAU
•CSU
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
Data security
• Only registered mobile devices
can connect to the system
• Bluetooth connection authentication
• Bluetooth connection encryption
• Access rights restrictions
• Personal and physiological data encryption
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•DAU
•CSU
•Future
improvements
•Project work
summary
•System
demonstration
Implementation - DAU
• Made from scratch by the team
• Bluetooth module inside
• Jazz sensor
interface
• Microphone and
earphone sockets
• 5-key keyboard
• Two lines, 8
characters LCD
• Powered from
6 AA batteries
BlueEyes
Human
Operator
Monitoring
System
Implementation - DAU
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
PCM codec board Microcontroller board
•System
implementation
•DAU
•CSU
•Future
improvements
•Project work
summary
•System
demonstration
Batteries and
module
voltageBluetooth
level monitor
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•DAU
•CSU
•Future
improvements
•Project work
summary
•System
demonstration
Implementation - CSU
• Standard MDI user
interface
• Custom-built
Bluetooth
communication
routines
• Intel USB camera
• ID card programmer
• Developed using Visual C++ 6.0
• Deployment environment:
Windows 2000,
MS SQL Server
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•System
demonstration
•Future
improvements
•Project work
summary
Future improvements
• DAU
– small CMOS camera to monitor the
operator’s point of gaze
– single PCB (SMD technlogy)
– low voltage ICs - LiIO batteries power
• CSU
– data mining algorithms
– advanced database encryption using e.g.
AES algorithm
BlueEyes
Human
Operator
Monitoring
System
Project work summary
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
•System
implementation
•System
demonstration
•Future
improvements
•Project work
summary
• Bluetooth and PAN technology acquired
• New skills on designing software and
hardware systems gained
• A mobile device implemented
• 31 KLOC of C++ code and
4 KLOC of 8051 assembler code written
• All parts of the Project Kit efficiently used
• New approach to system safety proposed
and implemented
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
•System
design
BlueEyes
Human-Operator Monitoring System
•System
implementation
•Future
improvements
•Project work
summary
•System
demonstration
System demonstration
BlueEyes
Human
Operator
Monitoring
System
•Motivation
•What is
BlueEyes ?
•Physiological
foundations
Thank you for your attention
•System
design
•System
implementation
•System
demonstration
•Future
improvements
•Project work
summary
BlueEyes
Human-Operator Monitoring System
Poznań University of Technology, Poland
http://www.put.poznan.pl
BlueEyes
Human
Operator
Monitoring
System
DAU - components
microphone
MC 145483
PCM codec
earphone
Bluetooth
module
Jazz Multisensor
LCD display
Atmel 89C52
microcontroller
UART
Beeper
UART
MAX232
LED indicators
BlueEyes
ID card
interface
ID card
Simple
keyboard
Data Acquisition Unit
BlueEyes
Human
Operator
Monitoring
System
CSU - Connection Manager
Operator
Data
Manager
Bluetooth
...
...
Bluetooth
Bluetooth
Connection
Manager
••Data
Initializes
demultiplexer Bluetooth
•Data
producer queues
modules
• Inquires for new
Separated
Raw Data
devices
in range
data streams
Operator
Buffer
Manager
Alerts
• Establishes
Alert buffer
connections
...
• Supports operator
...
roaming
• Feeds
Operator
Raw Data
Operator
Buffer with
Manager
Manager
Alerts
Alert buffer
physiological
data
BlueEyes
Human
Operator
Monitoring
System
CSU - Data Analysis
• Custom data analyzers - decision tree
based classifiers
Example: Operator gesture recognition
(simplified)
Vertical Acceleration variance
value high
Horizontal
Acceleration variance
high
N
“YES”
Y
N
Horizontal Acceleration
variance high
Y
N
No answer
Y
“NO”
BlueEyes
Human
Operator
Monitoring
System
CSU - Data Analysis
• Custom data analyzers - decision tree
based classifiers
• Automatic decision tree coefficients
generation - C4.5 based algorithm
Data samples
Assigned
category
“Yes”
...
No
answer
...
“Yes”
“No”
No
answer
Decision tree
coefficients
BlueEyes
Human
Operator
Monitoring
System
CSU - intermodule communication
• Uses buffered thread-safe queues and
producer-consumer scheme
register at
Data
consumer
creates
Data
producer
Data
queue
Data
Data
consumer
&
Processing
consumer
producer
Data
queue
Data
consumer
Data
Data
consumer
&
Processing
consumer
producer
Data
queue
BlueEyes
Human
Operator
Monitoring
System
CSU - intermodule communication
Connection
Manager
Operator manager (OM)
Alerts
Data Analysis
...
OM
OM
Separated physiological
data streams
Visualization
Module
Processed data
Recorded (off-line) data
Data
Logger