Transcript VUI01: Using Natural User Interface (NUI) Technologies to Improve User Experience Tim Huckaby Founder / Chairman, InterKnowlogy Founder / CEO, Actus Software Microsoft RD & MVP.

VUI01: Using Natural User
Interface (NUI) Technologies to
Improve User Experience
Tim Huckaby
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
About Tim…
InterKnowlogy (www.InterKnowlogy.com)
•
Tim Huckaby, Founder/Chairman ([email protected])
●
Custom Application Development / Consulting / Software & Systems Engineering Firm headquartered in
Carlsbad, CA
●
Design, Architect, Build and Deploy Enterprise Class Applications
●
Industry Experts:
•Most of the employees are published, MVPs, RD, Speakers, etc.
•Microsoft .NET Application development since 2000
•Microsoft .NET Rich Client Pioneers / Industry Leaders
•Microsoft Touch Application Development Leaders (Surface / Windows 7 / Silverlight 3)
•Information Worker Solutions (VSTO & SharePoint)
•Integration / Messaging, B2B / B2C, Wireless / Mobility
•Leading Edge: Surface, WPF, Silverlight, Win7 touch, SharePoint, VSTO
•Cutting Edge Solutions on emerging Microsoft technologies
•Largest Client: Microsoft
Actus Software (www.Actus-Software.com)
•
Tim Huckaby, Founder/CEO ([email protected])
•Software Product Company, Coming soon…
A little bit about me and why I am here…
Presentation Agenda
Touch Capable Computing: Using Natural User Interface (NUI)
Technologies Improve The User Experience
Multi-Touch Software built in Microsoft .NET: WPF, Surface,
Silverlight, & the phone…
Hardware & Software Platforms for Touch Computing
Multi-Touch enabled applications used to improve the
communication, education and user experience across the
software continuum.
Demos, Demos, Demos
About the Computer I am Using Today…
• Acer Aspire 1420P
• Give away “free” to all attendees at the
PDC
• One year old…
• Price in October-2009: $420
• Price in November-2009: $350
• Tim Huckaby
•
•
•
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
Multi-Touch ScatterView Photos
Application
•
•
•
Tim Huckaby
Founder, InterKnowlogy & Actus Software
Microsoft RD & MVP
The InterKnowlogy “Firebrick” Silverlight Multi-Touch
ScatterView Control
• What is it?
●
It’s a Silverlight Control for Intuitive & Comprehensive Multi-Touch Object Manipulation for your Web
Applications.
• Download Site: http://firebrick.interknowlogy.com/
• Example Problem(s) it solves:
●
Silverlight’s Touch API
•
•
●
Is rudimentary at best. Basically the API gives you “I know something has been touched; but, I don’t know what or where.”
Is difficult; takes a ton of programming to implement at any significant level
The Silverlight Multi-Touch ScatterView Control
•
•
•
Makes building a Multi-Touch capable application “Brain-dead simple” in a matter of minutes
Contains all the “heavy lifting”: thousands of lines of calculus and trigonometry
Contains a sample application and detailed “How-to” documentation
• Installation Notes:
●
Free Developer and Low Cost ($99) production versions available
• Usage Notes: Multi-Touch is supported in SL3 and above and currently only on Windows 7
Touch Capable Computing
Devices are Not New
1972: PLATO IV Touch Screen
Terminal
Past
The Microsoft
Surface
Present
Windows 7 Multitouch
Future (Now)
What is Relatively New is:





Inexpensive Touch Capable, Non
Proprietary Hardware
A Multi-Touch Capable Windows OS
A Consistent Multi-Touch API for .NET
across 3 Windows Hardware/Software
Platforms
Developer Productivity in Touch
Capable Apps
The Forecast of Touch Capability in
Every New Computing Device in the
Short Term
Surface WPF 3D:
The Scripps Research Institute
VitruView
• Tim Huckaby
•
•
•
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
History of Surface
The concept of an
interactive table is born
A dedicated group is formed,
and more than 85
prototypes are developed
The first
prototype is
presented to
Bill Gates
The look and feel of
Surface is finalized
Microsoft Surface is unveiled
to the world
Single Touch vs. Multi-Touch
• You get single touch “for free” in Windows 7
●
Almost everywhere in Windows 7, a touch = a
mouse click
• Multi-Touch is the capability to recognize 2
or more fingers on the screen at once.
●
●
The iPhone made the “pinch gesture” a cultural
norm in usability
Vernacular on hardware capability: “4 touches”
TOUCH
HARDWARE IMPLEMENTATIONS
Capacitive
Infrared
Resistive
Sample Hardware
Capacitive
HP Touchsmart tx2
N-trig (2-10)
Capacitive
Dell LatitudeXT2
N-Trig (2-10)
Infrared
HP Touchsmart IQ816
NewWindow(2)
How to touch
Resistive Think Push Down and drag
How to touch
Capacitive Think Electric Impulse
Common hardware implementations
• Capacitive
●
iPhone / iTouch – Capacitive multi-touch cell phone / PDA
●
HP TouchSmart TX2 – Dual touch capacitive laptop screen.
●
Dell Latitude XT – Multi-Touch tablet using N-Trig capacitive
technology
●
MacBook Pro – Capacitive Multi-Touch pad with traditional click
●
N-Trig – Capacitive monitor overlay vendor (Windows 7 Drivers)
How to touch
Infrared Think laser pointer(s)
Common hardware implementations
• Infrared
●
Surface – Infrared diffused illumination multi-touch “table”
device
●
Perceptive Pixel – Frustrated total internal reflection multitouch panels
●
HP TouchSmart – Infrared dual touch bezel all in one computer
●
NextWindow – Infrared Multi-Touch bezel vendor (Windows 7
Drivers)
NUI GESTURES
•
•
•
•
Tap
Flick/Fling/Scroll
Pinch/Spread
Drag
Surface Craps
• Tim Huckaby
•
•
•
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
SOFTWARE IMPLEMENTATIONS
(ON THE MICROSOFT PLATFORM)
Windows Desktop: WPF
Surface: WPF
Browser: Silverlight
Phone: The Windows Mobile 7 phone
NUI
Multi-Touch Scenarios
Touch Scenarios and Windows 7
Developer Platform


Navigating and
consuming the Web

Reading and sorting
email

Viewing photos

Playing casual games

Consuming music
and video

Navigating files and
arranging windows

Using Microsoft Office
Applications
At the root is the touch developer platform
that exposes touch APIs for any application
UI Enhancements

Focusing on core scenarios, many parts of
the core UI optimized for touch experiences
Gestures

Multitouch gestures enable consistent
panning and zooming in most applications
Applications

Powerful Multitouch-focused applications to
ship separately from Windows 7
VitruView – Badge Authentication
Microsoft Surface
VitruView – 3D manipulation
Microsoft Surface
VitruView – Adding annotations with the
Virtual Keyboard in 3D
Patient Management
Microsoft Surface
Bedside Machine Management
Microsoft Surface
Hospital Management
Microsoft Surface
Molecule Viewer
Microsoft Surface
Silverlight… “WPF Light”
Lightweight WPF runtime
• Interactive experiences for devices, PC, & Mac platforms (and
others…)
• Strict subset of Windows Presentation Foundation
• Lightweight download (4.2mbs)
• Programming model uses XAML + Jscript + WPF
• Access to WPF tooling
• Touch Capability in Version 3.0
WPF-based Tools
<XAML>
<Button Width="100px"> OK
<Button.Background>
LightBlue
</Button.Background>
</Button>
</XAML>
Eclipsys Maestro: Hospital
Management
Eclipsys Maestro: Hospital
Management
EMN8 & REZN8
• Tim Huckaby
•
•
•
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
from our Marketing VP Madeline Pantalone:
“It's okay for you to use application for
demonstration as long as you are sure to preface it
with the phrase: "This is test data only. Products
you see may have been created for testing purposes
only."
NUI: Games as a Tool for Learning
• Learning made Fun / Entertaining
• Proven as a learning tool beyond just
Children
• Computer based Games are a significant
part of popular culture
• Educational Video Game Curriculums
●
MIT, UC Irvine, Minnesota, Mich State, etc.
NASA: Mars Bound
• Tim Huckaby
•
•
•
Founder / Chairman, InterKnowlogy
Founder / CEO, Actus Software
Microsoft RD & MVP
XBOX Kinect (Natal)
•
•
•
•
•
Motion Based Interface
$149.99
Availability: November 4th
40 million Xbox 360 owners
RGB camera, three sensors
●
•
•
•
•
(one depth sensor, one motion sensor, and one multi-array microphone)
tracks full-body movement, facial expression, and voice with precision
Navigate the entire Kinect Hub interface on your Xbox 360 without touching a
conventional 360 controller.
While you're watching a movie, you can verbally instruct the Xbox to stop, fastforward, rewind, exit out, etc. by calling out easy commands like, "Xbox. Fastforward." or "Xbox. Pause.“
Kinect sensor bar scans and monitors 1 million points on the body
• XBOX Kinect (Natal)
• Tim Huckaby
• Founder / Chairman, InterKnowlogy
• Founder / CEO, Actus Software
• Microsoft RD & MVP
XBOX Kinect (Natal)
The Bad News:
•
•
•
•
Not many applications at launch
Not really any applications at launch; just games
“little kid games”
No exposed API at launch, plus the
legal restrictions of “hacking it”
●
●
●
●
No building your own apps
No integration with any of the Microsoft Stack
No vertical solutions
….for the immediate future
Neural Interfaces
Conscious and non-conscious
Interfaces
•
•
•
•
High fidelity EG acquisition device
No conductive gel, etc.
Non-invasive
Use Case Examples:
• Conscious: mind manipulation of objects
• Non-Conscious: Ranking music
• First target market: gaming
• $300!!!
• Tim Huckaby
• Founder / Chairman, InterKnowlogy
• Founder / CEO, Actus Software
• Microsoft RD & MVP
Emotiv SDK
• C++ Header files and Import Libraries
…which means…
• Wrap the SDK in managed C++ thereby
exposing a the API to a C# application
• That is very powerful…
NUI
The Natural User Interface
• Multi-Touch Capable
●
Common gestures of touch
• Gesture Capable
●
●
Motion Based Interface
Tracking of full-body movement, facial expression, and voice
with precision
• Neural Capable
●
●
Conscious and non-conscience interfaces
Non Invasive
Summary






We are at the Cusp of the NUI computing revolution
There are still many “standards” yet to be adopted
Multi-Touch computing will become a mainstream aspect of
NUI and consequently of software development
 Paradigm Shift from GUI to NUI (Natural User Interface)
Touch capable computing changes the way you design
systems and differentiates your applications
Shifting from desktop applications to multi-touch based
applications will be a cultural change like the mouse was
There are other components of NUI other than just touch…
 Neural & Gesture based interfaces
Your Feedback is Important
Please fill out a session evaluation form
drop it off at the conference registration
desk.
Thank you!
Tim Huckaby
•
More info on InterKnowlogy:
●
●
•
More info on Actus Software:
●
•
www.InterKnowlogy.com
Silverlight.InterKnowlogy.com
Coming soon….
About Tim Huckaby…
●
●
●
●
●
●
●
●
●
Founder/Chairman, InterKnowlogy
Founder/CEO, Actus Software
Microsoft® Regional Director – Southern California
Microsoft® .NET Partner Advisory Council Founder /
Member
Microsoft® MVP - .NET
INETA Speaker – International .NET Users Group
Association
Board Member: Association for Competitive
Technology
Board Member: Dev Connections
Author / Speaker
Contact me: Tim Huckaby
E-mail: [email protected]
Phone:
Office: 760-444-8640
Mobile: 619 990 9200
Blog:
http://team.interknowlogy.com/blogs/timhuckaby
FaceBook:
http://www.facebook.com/Tim.Huckaby.Carlsbad
LinkedIn:
http://www.linkedin.com/in/timhuckaby