Transcript Augmented Cognition Brief To CS Students

The Context Engine A Device to Augment
Cognition
Perry McDowell
The MOVES Institute
1
Augment Cognition
Add to your thinking, hopefully improving it
That’s been the whole point of computers
We want to make it completely natural and
automatic

In some ways, similar to:
 An admiral’s aide or efficient secretary
 A senior NCO or subject matter expert
Want to do this with a wearable computer
Move nearer to ubiquitous computing
2
Literary Inspiration for the
Military Context Engine
“Our suits give us better eyes, better ears…
better intelligence… the beauty of it [is] you
don’t have to drive it, fly it, conn it, operate it,
you just wear it… This leaves you with your
whole mind free to handle your weapons and
notice what is going on around you… If you
load a mud foot down with a lot of gadgets
that he has to watch, someone a lot more
simply equipped – say with a stone axe – will
sneak up and bash his head in while he is
trying to read a vernier.”
- Robert A. Heinlein, Starship Troopers
3
Steve Mann’s Six Attributes of
Wearable Computing
1. Unmonopolizing of the user's attention




It does not cut you off from the outside world.
You can attend to other matters while using the
apparatus.
Assumes computing will be a secondary activity.
Provides enhanced sensory capabilities.
2. Unrestrictive to the user

You can do other things while using it
4
Steve Mann’s Six Attributes of
Wearable Computing(cont)
3. Observable by the user


It can get your attention continuously if you desire
The output medium is constantly perceptible by
the wearer.
4. Controllable by the user
5. Attentive to the environment

Environmentally aware, multimodal, multisensory.
6. Communicative to others

Can be used as a communications medium.
5
Steve Mann’s Six Attributes of
Wearable Computing(cont)
Implied by the above six properties is that it
must also be:
Constant: always on, running, and ready.
Personal: human and computer are inextricably
intertwined.





Prosthetic: you can adapt to it so that it acts as a true
extension of mind and body; after time you forget that
you are wearing it.
Assertive: resists, if you wish, prohibition or requests by
others for removal.
Private: others can't observe or control it unless you let
them.
6
Example of Uses Finding a Restaurant
You’re downtown and decide you want to
eat out
The engine makes a recommendation
based upon several factors:
 Location
 Cost
 Favorite
foods
 Favorite restaurants
 Tonight’s
specials
 Friends’ favorites
 Reviews
 Current wait
7
Finding a Restaurant
Let’s take it one step further:

You can tell it you want to take your wife out for
your anniversary - handle it
 It knows your favorite romantic restaurants, so it makes a





reservation at one
It contacts one of the babysitters you use and arranges
services for that night
Orders flowers delivered to your wife’s office
Reminds you to buy a present, or orders one from the web
for you
Schedules it with your wife’s contact engine
Tells you that everything is ready to go
8
Examples of Uses Watch Turnover
At watch relief, transfer all pertinent data to your
relief

List of contacts and their status, current/ pending
orders, equipment status, upcoming events, etc.
The system can’t forget anything, rush so it can
hit the pit, misinterpret what it’s told…
The system would also be able to give you
advice - “No, you don’t want to turn left…”
9
Example of Uses Crossing the River
You are the leader of a platoon of infantry
on a covert mission, currently moving to
conduct a rendezvous across a river

The bridge you were expecting to use is
destroyed
What are you going to do now?
10
Crossing the River
(cont)
Determining your current options,
weighing them, and deciding upon a
course of action takes time
Context engine immediately determines
options and the best course of action
and makes a recommendation

It can explain its reasoning fairly quickly to
you
11
Examples of Uses Summary
In the dinner example, it acts mainly as an
assistant - it knows your likes and searches
data for the best match
In the river example, it acts as an expert
system, giving you a better solution
In the watch example, it combines the two - it
stores data for use later, but can have
intelligence (either the previous watch officer
or from the system) built in
12
Goals
Our goal for this project is to take
Commercially Off the Shelf (COTS)
hardware and software to create a
proof of concept for the Context
Engine.
13
Difficult Problems
Determining the “symbolic content” of the user’s
environment
Once it is symbolized, interpreting the
environment to give information and advice
Presenting the information to the user in an
unobtrusive and helpful manner
14
Breakdown of the Context
Engine’s Processes
People
Visual
Display
Goal 2
Actions
Goal 3
Goal 1
Determine
Course of
Action
Outer World
Abstraction
Aural
Display
Goal 4
Output
to User
Goal n
Location
Objects
User’s
Commands
Preferences
15
Initial Process Recognize the World
People
Actions
Outer World
Abstraction
To course
of action
Location
Objects
16
Symbolic Content
Having the computer determine where it
is, who/what is present and what is
happening


Computer visualization
Wireless recognition
 Bluetooth


Aural clues
GPS
17
Current Systems Which
Convert Data to Abstraction
Face Recognition
 Several companies has fairly effective
facial recognition systems.
 Lau Technology’s daughter company, Viisage,
scanned all the faces at the Super Bowl,
looking for terrorists
 We anticipate getting an SDK from one
of these companies to begin our
research
18
Current Systems Which
Convert Data to Abstraction
Wearable belt for outdoor sports
 Determines events which might indicate
the wearer is incapacitated
 High acceleration
 Vital Signs
 Alerts authorities to
potential injury
19
Determining The
Course of Action
Goal 2
Goal 3
Goal 1
Determine
Course of
Action
From World
Recognition
Goal 4
To Output
to User
Goal n
User’s
Commands
20
Determining the
Course of Action
Once you know what is in the
environment, what does that mean?

Network solution
 Take the dynamic extensibility from NPSNET-V
 Embedded XML information explains what an
object is and how to interact with it

SIMS style smart environment
 Objects broadcast what they can do
21
Determining the
Course of Action
Get input from the user’s body


Affective computing takes body measurements to
determine your mental/ physical state
The context engine will be able to indicate when
medical assistance is required
 Expedite triage also


System knows not to deliver bad news when you’re
angry/ tired/ morose/ etc.
System can tell you’re beginning to panic and
“slaps you in the face”
22
Affective Computing Gear
This affective wearable includes a ProComp sensing system
(upper left corner) surrounded by four sensors, clockwise from
top: respiration, galvanic skin response (GSR) (center, left),
blood volume pressure (BVP), and electromyogram (EMG). This
unit attaches to a PC104 standard based computer (lower right)
which receives data from a Twiddler hand-held keyboard (lower
left). and displays data with the Private Eye (far right, below
respiration sensor.) (J. Healey)
23
Determining the
Course of Action
User can input commands which give the
computer tasks to determine
The user’s commands are evaluated in
light of the current goals
 Goals are determined either explicitly or
implicitly
 Explicitly means that the user has previously
defined this as a goal
 Implicitly means the computer has learned that
this goal is important
24
Determining the
Course of Action
Database sharing

Gives one user the knowledge and insight
of another user
 Can be as simple as identifying someone
 Can be as complex as planning an exercise

Will raise emphasis on training people's
machines closer to the level we currently
spend on training people
 May be possible to train both simultaneously
25
Outputting Information
to the User
Visual
Display
From Course
of Action
Aural
Display
Output
to User
Preferences
26
Presenting Information
to the User
Interface must be:







Light
Small
Unobtrusive
Intuitive
Robust
Easy to use
Cheap
Easier said than done
27
Presenting Information
to the User
“Fitting” the system to specific
users


Everyone wants different
information
User must find the given
information useful

Don’t want the annoying
paperclip scenario
28
29
Conclusion
Context Engine must be easy and natural
to use
The problem has three major areas:
 Recognizing the world
 Determining the course of action
 Presenting information to the user
30
NPS/MOVES Work
Construct a virtual world and build a more
complete and robust engine
 Several options as to the input to the world:
 Joint SIGINT Avionics Family (JSAF) Data
 Plan on using agents:
 To determine context from the symbolic
representation
 To determine recommended course of action
31
Potential Thesis Work
You can work in any part of the project
discussed in this brief



First part will be extremely difficult
Second part will be difficult, but you’ll be
working on a smaller part of the problem
Third part will be very interesting, and is a
great opportunity for someone interested in
human factors work
32
Questions?
Contact Information:
Perry McDowell
Lecturer of Computer Science
[email protected]
http://www.movesinstitute.org/~mcdowell
33