Integrating Microsoft® Kinect* with AutoCAD®
Download
Report
Transcript Integrating Microsoft® Kinect* with AutoCAD®
Integrating Microsoft® Kinect™ with AutoCAD®
Kean Walmsley
Senior Manager, Worldwide Developer Technical Services, Autodesk
© 2011 Autodesk
1
About the Presenter
Kean joined DevTech in August 1995, and has since worked
in a number of different positions and countries (UK, USA,
India and Switzerland). These days he spends most of his
time on management-related activities, but still manages to
keep involved at a technical level with our products.
Since July 2006 Kean has maintained a development-related blog at:
blogs.autodesk.com/through-the-interface
Kean holds a Licence d’Informatique from l’Université de Paris-Sud
and a Masters in Computer Science from the University of Kent at
Canterbury.
© 2011 Autodesk
2
Class Summary
This class will focus on learning how to integrate Kinect™, Microsoft's groundbreaking controller for the Xbox 360®, with AutoCAD using .NET.
We will look at the various drivers, SDKs, and middleware available to .NET
developers wanting to work with Kinect, as well as the kind of capabilities a device
such as Kinect brings to 3D products such as AutoCAD.
We will cover two main uses of Kinect: as a 3D reality capture device, feeding point
clouds into AutoCAD, and as a user-input device, extending the next generation of
gesture-detection capabilities to AutoCAD .NET developers.
© 2011 Autodesk
3
Learning Objectives
At the end of this class, you will be able to:
Implement gestures to allow users to control drawing, modeling, and navigation
operations via Kinect™
Describe industry trends in human-computer interaction and explain how devices
such as Kinect are changing the world
Integrate code that imports point cloud data from Kinect into AutoCAD
List the available drivers, SDKs, and middleware available for working with Kinect
© 2011 Autodesk
4
Agenda
Introducing Kinect™
The Kinect Device
The Race to Hack Kinect
Drivers, SDKs & Middleware
Considerations
Doing a Jig
CL NUI Platform & OpenKinect
OpenNI™ / NITE™
Microsoft® Kinect SDK
Getting Started Coding with Kinect for
AutoCAD®
Designing a Natural User Interface
Point Clouds
Interpreting Gestures
Skeleton Tracking
Implementing Gestures
Speech Recognition
Implementing Voice Commands
Model Navigation
Implementing Model Navigation
Kinect Data Useful to 3D Applications
NUI Design Considerations
© 2011 Autodesk
5
Introducing Kinect™
© 2011 Autodesk
6
© 2011 Autodesk
7
© 2011 Autodesk
8
© 2011 Autodesk
9
Image courtesy of Anya Quinn
© 2011 Autodesk
10
Image courtesy of Andre Manoel
© 2011 Autodesk
11
Image courtesy of Microsoft
© 2011 Autodesk
12
The Race to Hack Kinect™
Nov
th
4 ,
2010
Kinect for Xbox 360® launched in North America
Adafruit announced competition to reverse-engineer a Kinect driver
Nov 6th
First hacked by AlexP, who chose not to publish his code
Went on to create Code Laboratories™ NUI Platform
Nov 10th
Héctor Martín announced by Adafruit as winner of the $3K bounty
Dec
Posted code to github, creating libfreenect/OpenKinect
th
8
OpenNI™ launched
Created by PrimeSense™, providers of Kinect technology to Microsoft ®
© 2011 Autodesk
13
© 2011 Autodesk
14
© 2011 Autodesk
15
© 2011 Autodesk
16
Drivers, SDKs & Middleware
© 2011 Autodesk
17
Reverse-Engineered Drivers
Code Laboratories™ NUI Platform
Spawned from work by AlexP
Free, closed source
Windows only
C/C++ and .NET
OpenKinect / libfreenect
Spawned from work by Héctor Martín
Free, open source
Cross platform (Linux, Mac, Windows)
C/C++, .NET, Python, ActionScript
Both tools are primarily drivers
(i.e. no skeleton tracking, gesture recognition)
© 2011 Autodesk
18
OpenNI / NITE
For several months, the closest thing to “official” drivers
Multiple components required
OpenNI SDK (open source)
NITE middleware for skeleton tracking and gesture handling (closed source, for fee?)
SensorKinect patched driver (open source by avin2)
For .NET, nKinect, developed by Boris Scheiman (open source but 32-bit only)
Complex to get working
Matching builds of each component required
Plus a significant amount of manual configuration/tweaking
© 2011 Autodesk
19
Microsoft Kinect SDK
First Beta released June
th
16 ,
2011
Projected commercial release early 2012
Overall significant benefits vs. OpenNI
Simple component install (closed source)
32- and 64-bit versions available
Better skeleton tracking & real world mapping
Good sample code, documentation and support
Some disadvantages
More work needed for point cloud support
Previously relied on nKinect for heavy lifting
Currently doesn’t work inside a Virtual Machine
Commercial licensing costs?
© 2011 Autodesk
20
Getting Started Coding with Kinect for AutoCAD
Make sure you have a USB/power adaptor
1.
Comes with separate Kinect box
For compatibility with older Xbox 360 system
Does not come with a newer bundle
Must then be purchased separately
Install the latest Microsoft Kinect SDK
2.
Or OpenNI + NITE + SensorKinect + nKinect
3.
Test with the “Sample Skeletal Viewer”
4.
Create Class Library project
Add project references to AcMgd.dll, AcDbMgd.dll & Microsoft.Research.Kinect.dll
© 2011 Autodesk
21
Demo | Our first AutoCAD + Kinect app
© 2011 Autodesk
22
Designing a Natural User Interface (NUI)
© 2011 Autodesk
23
Kinect Data Useful to 3D Applications
Point cloud data
RGB + depth data
Good for “reality capture”
Skeleton data
Movement and gestures
Audio
Speech recognition
Settings/commands
Image courtesy of Kyle McDonald
© 2011 Autodesk
24
NUI Design Considerations
Do you want to display point cloud data in real-time?
Will the user be creating geometry in-place?
Do you need to interpret gestures?
How about spoken commands?
Do you need to manipulate the current view?
© 2011 Autodesk
25
Point Clouds
© 2011 Autodesk
26
Considerations
Performance a concern if displaying full point cloud interactively
640 x 480 = 307K points
Sample appropriately from the device
Each displayed pixel needs mapping to “skeleton space”
Colorize only when needed
SkeletonEngine.DepthImageToSkeleton()
NuiCamera.GetColorPixelCoordinatesFromDepthPixel()
If high point density required, consider clipping
Easy to filter point set based on clipping volume
© 2011 Autodesk
27
Doing a Jig
Jigs designed to get user input via keyboard/mouse
Graphics updated when messages received
Enabling graphics update for external devices…
For interactive display in a jig, no need to colorize
WorldGeometry.Polypoint() displays points using a single color
If defining paths, smooth off the jitters
… changing the cursor position by one pixel forces an update
Check distance against previous point
Avoid expensive operations to generate display graphics
e.g. break complex sweeps/extrusions in segments
© 2011 Autodesk
28
Fully Colorization
Outside of a jig, full colorization is straightforward
e.g. on completion of the jig
One approach is TXT → LAS → PCG
Minor issue of color mis-matched with depth
Strange shadowing occurs
Theoretically possible to combine data
Multiple Kinects or viewpoints
© 2011 Autodesk
29
Demo | Jigging point clouds, polylines and solids
© 2011 Autodesk
30
Interpreting Gestures
© 2011 Autodesk
31
Skeleton Tracking
20 joints available
2 “players” can be actively tracked
6 additional players, passively tracked
Skeleton space has Y upwards
To transform to “real space”, need to
Swap Y with Z
Negate X (depending on capture settings)
No need if gestures aren’t “in place”
i.e. if not overlaying with point cloud data
© 2011 Autodesk
32
Implementing Gestures
From the OnSkeletonFrameReady() callback
Iterate through the e.SkeletonFrame.Skeletons, look for tracked Skeletons
Moree work needed if supporting multiple participants
Access its Joints, select the entry you want, extract its Position
Transform them to the space you want
Relative to the model, if “in-place”
Can be left as is, if just navigating
Perform comparisons of relative points
e.g. the distance between the hands is < 5cm
Extract vectors from elbow/shoulder to hand, if needing direction information
© 2011 Autodesk
33
Speech Recognition
© 2011 Autodesk
34
Implementing Voice Commands
Kinect has an array of 4 microphones
Provides accurate positional data for voice commands
The generated audio stream can also be processed
Can associate speech with a particular skeleton
Microsoft Speech SDK provides speech recognition engine
Straightforward to integrate
Asynchronous callbacks for recognized words
Not on the GUI thread: take care with updating data
Useful for changing settings while gesturing
© 2011 Autodesk
35
Demo | Jigging segmented solids
© 2011 Autodesk
36
Model Navigation
© 2011 Autodesk
37
Implementing Model Navigation
Collaborative walk- and fly-throughs
The Killer App for Kinect + AutoCAD?
Need to adjust the current ViewTableRecord
No need for a jig, as we’re not displaying in-place
Glyphs can be displayed via Transient Graphics
Consider constraining axes of movement, to avoid strangeness
Also worth adding a “home” gesture
Image courtesy of Rafee Jewell
© 2011 Autodesk
38
Demo | Navigating an AutoCAD model
© 2011 Autodesk
39
Agenda
Introducing Kinect™
The Kinect Device
The Race to Hack Kinect
Drivers, SDKs & Middleware
Considerations
Doing a Jig
CL NUI Platform & OpenKinect
OpenNI™ / NITE™
Microsoft® Kinect SDK
Getting Started Coding with Kinect for
AutoCAD®
Designing a Natural User Interface
Point Clouds
Interpreting Gestures
Skeleton Tracking
Implementing Gestures
Speech Recognition
Implementing Voice Commands
Model Navigation
Implementing Model Navigation
Kinect Data Useful to 3D Applications
NUI Design Considerations
© 2011 Autodesk
40
Autodesk and AutoCAD are registered trademarks or trademarks of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. Microsoft, Kinect and Xbox 360 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Code Laboratories and the CL Logo are trademarks of Code Laboratories Incorporated. PrimeSense, the PrimeSense
© 2011
Autodesk
logo, NITE and OpenNI are trademarks of PrimeSense or its affiliates. All other brand names, product names, or trademarks belong to their respective holders. Autodesk reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document. © 2011 Autodesk,
Inc.
All rights reserved.
42