Transcript Kinect_Calibration_2012

Facoltà di Ingegneria
Kinect calibration
Ilya Afanasyev
Trento, 2012
25/01/2012
1/20
Content
1.
2.
3.
4.
5.
6.
7.
8.
25/01/2012
The description.
About the existing software.
Tests of Nicolas Burrus software binaries.
Open Kinect drivers installation.
Compilation of Nicolas Burrus source code.
Results.
Conclusions
References.
2/20
The description.
The software for Kinect Calibration should be used
for:
1.
2.
3.
16/01/2012
Saving the RGB images and IR-Depth data from Kinect sensor
under OS Windows.
Calibration the Kinect sensor with obtaining [1]:
- the intrinsic parameters (i.e. matrixes of intrinsic parameters
for IR camera (K_ir) & RGB camera (K_rgb)).
- the extrinsic parameters (i.e. extrinsic mapping between IR
(depth) and RGB Kinect camera (R,T)).
- distortion coefficients (kc_ir, kc_rgb).
- the intrinsic and extrinsic parameters (i.e. matrixes of intrinsic
parameters for IR camera (K_ir) & RGB camera (K_rgb) and
extrinsic mapping between IR (depth) and RGB Kinect camera
(R,T)).
Developing the interface of different algorithm with Kinect for
different real-time object recognition applications.
3/20
About the existing software.
The existing analogues of the software:
1.
2.
25/01/2012
- The author doesn’t know analogues under Windows7/64 bit . It
can be created, basing on Microsoft Kinect SDK for developers
[6] under the platform .NET Framework ver. 4.0.
- There is the software developed by Nicolas Burrus [2] under
OS Linux / Ubuntu 10.10 / 32 bit, which is claimed to be
compiled also with Windows-32 bit. Can it be used under
Windows7/64 bit ?? -> It seems that NO!!
4/20
Microsoft Kinect SDK for developers [6]
1.
16/01/2012
SDK (Software Development Kids) includes drivers,
APIs for raw sensor streams and human motion
tracking, installation documents, and resource
materials. It provides capabilities offered by the
Microsoft Kinect device connected to computers
under Windows 7 / Windows 8 to developers who
build applications with C++, C#, or Visual Basic by
using Microsoft Visual Studio 2010.
5/20
Nicolas Burrus software [2]
Nicolas Burrus software [2]:
There are different versions of the Burrus software:
1. Test programs running from binaries -> Windows binaries ->
Kinect RGBDemo v0.6.1. -> supports OpenNI/Nite backends [3]
and has the experimental infrared support with OpenNI (still
buggy) -> I tried to check this work for OpenNI backend under
Windows 64bit -> there is no the extrinsic mapping between IR
and RGB camera at Kinect calibration.
2. Compiling from source ->
2.1 Compilation on Windows ->
with OpenNI/Nite backends doesn’t give the extrinsic mapping
between IR and RGB camera at Kinect calibration.
with Libfreenect backend -> I wasn’t succeed to compile
under Windows 7/64 bit?!
2.2 Compilation on Linux (Ubuntu 10.10 - 32 bit) -> supports
Libfreenect backend -> the only workable variant with getting all
calibration parameters (intrinsic, extrinsic and distortion) was
realized on Alberto Fornaser’s PC (UniTN)
16/01/2012
6/20
1st attempt
The using of Nicolas Burrus software [2]
with OpenNI backend under Windows
1. OpenNI backend:
1.
2.
3.
4.
5.
6.
16/01/2012
Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports
OpenNI/Nite backends [2] and has the experimental infrared
support with OpenNI (still buggy) -> I tried to check this work for
OpenNI backend under Windows 64bit.
Install Sensor Kinect drivers under Windows 32 bit [4] and
OpenNI/Nite modules (Win32) [3] according to the instruction [5]
and verified that they work properly.
Use RGB-D Capture [1] to grab Kinect RGB and IR images with
intensity.raw and depth.raw files. -> the software doesn’t grab IR
images!! -> only RGB image, intensity.raw, depth.raw and
calibration file (calibration.yml without distortion parameters)!!
“calibrate-openni-intrinsics --pattern-size 0.0405 grab1
calibration.yml” -> gives “openni_calibration.yml” with Intrinsic
matrix and distortion coefficients for Kinect RGB camera !!
“calibrate-openni-depth.exe --pattern-size 0.0405 grab1” gives
partly processed figures with a message about errors.
The software doesn’t produce a file with xyz- coordinates and
RGB-colors info of every point!!!
7/20
The comparison of Kinect calibration results for
- Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit)
- OpenNI backend (under OS Windows 32bit)
Libfreenect backend
16/01/2012
The same figures of Intrinsic
Matrixes for Kinect RGB and
Depth Cameras
OpenNI backend
The Depth Camera Calibration
for OpenNI backend is
NOT available!!!
-
The comparison of Kinect calibration results for
- Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit)
- OpenNI backend (under OS Windows 32bit)
Libfreenect backend
23/11/2011
There is NO extrinsic mapping
between Kinect Depth and RGB
cameras for OpenNI backend
OpenNI backend
The reprojection of 3D points
from Kinect Depth Camera to
RGB image for OpenNI is NOT
possible!!
Install OpenKinect for Windows 7 [7]
It is necessary to have:
- Visual Studio 2010 (as a compiler on a 32 bit
Windows 7 installation).
- Kinect Sensor drivers for Windows 7
(OpenKinect-libfreenect ofTheo/ofxKinect) [9].
- freenect windows dependencies [8]:
 libusb-win32 1.2.4.0 (Currently there is a
breaking change in libusb 1.2.5.0, which I used!!);
 pthreads-win32
 Glut
17/01/2012
10/20
Kinect Sensor drivers installation [7]
- Download & extract the latest GitHub repository [9].
- Make sure your Kinect sensor is plugged into a
USB port on your computer.
- Open Device Manager (Start  Control Panel 
System and Security  System  Device
Manager)
- Find “Human Interface Device” (Xbox NUI Motor) 
Update Driver Software  uninstall the previous
Kinect drivers (if there are)  set path like
C:\Special\OpenKinect_libfreenect_v0_1_1_56\pl
atform\windows\inf\xbox nui camera\  the
drivers will correspond to the processor type.
18/01/2012
11/20
Kinect Sensor drivers installation [7]
19/01/2012
Check the files: libusb0.dll (x64) in C:\Windows\System32\
and libusb0.sys in C:\Windows\System32\drivers\
and libusb0.dll (x86) in C:\Windows\SysWOW64\
12/20
Cmake configuration & Generation of a Visual
Studio 10 solution [7]
1.
2.
Open Cmake GUI and set the Browse Source
path to /libfreenect/openkinect.
Set the browse build path to
/libfreenect/openkinect/build
The software should be configured when Kinect turned off!!
19/01/2012
13/20
Cmake configuration & Generation of a Visual
Studio 10 solution [7]
You will now need to point certain build paths to the
freenect dependencies (pthread).
The software should be configured when Kinect turned off!!
19/01/2012
14/20
Cmake configuration & Generation of a Visual
Studio 10 solution [7]
You will now need to point certain build paths to the
freenect dependencies (libusb).
19/01/2012
If take "msvc" instead of "msvc_x64" -> 1 fatal error of compilation less !!!
It doesn't matter if choose THREADS\LIB\ pthreadVC2.lib or
pthreadVSE2.lib
15/20
Cmake configuration & Generation of a Visual
Studio 10 solution [7]
Run Configure again. It should successfully complete the process
this time but we still need to add the Glut references.
19/01/2012
Run "Configure" and it should successfully complete. Click Generate
afterwards and then open the libfreenect.sln file from the build
folder the output was placed in - /openkinect/build/
16/20
Visual Studio 10 solution [7]
Disable the Antivirus if it disturbs the creation of the VS10 solution!
Choose Visual C++ 2010 and then select Build Solution from the
Debug Menu or press F7.
19/01/2012
RESULT: Rebuild All: 13 succeeded, 2 failed, 2 skipped.
You will receive a few errors when building the solution but it should compile
correctly regardless. The binaries will be placed in build/bin/Debug.
17/20
The result of building the c for Windows 7 [7]
The final step is to place a few .dll files in the
appropriate folder and then we can run some of the
samples provided:
 Copy freenect.dll and freenect_sync.dll from
\build\lib\Debug
 Copy from dependencies folders: pthreadVC2.dll,
glut32.dll, libusb0.dll (choose the correct version x86/x64)
 Copy tiltdemo.exe, regtest.exe, glview.exe,
glpclview.exe, glpclview.exe, regview.exe, hiview.exe
from \build\bin\Debug
 It can be helpful to copy freenect_sync.lib, freenect.lib,
freenect.exp from \build\lib\Debug to \lib\
Successful start of demo-samples  Kinect works!
19/01/2012
18/20
Demo-samples of OpenKinect for Windows 7
 tiltdemo.exe drives the Kinect motor.
 regview.exe shows the combined
IR&RGB video (IR up to 1 m)
 regtest.exe grabs bad-quality pictures:
-- registration_test_depth_registered.pgm,
-- registration_test_rgb.ppm,
-- registration_test_depth_raw.pgm,
-- registration_test_depth_mm.pgm
 glview.exe gives IR&RGB
video in separate windows
(IR up to 3.5 m)
19/01/2012
19/20
Demo-samples of OpenKinect for Windows 7
 hiview.exe gives IR&RGB
video in separate windows
(IR up to 3.5 m, RGB has hi-quality
& hi resolution)
 glpclview.exe gives color
3D video with big shadows
behind the objects.
 cppview.exe produces a
continuous list with messages
under the phrase:
“thread execution started”
19/01/2012
20/20
2nd attempt
19/01/2012
Compiling RGBDemo-0.6.1 under Min GW from
source with OpenKinect on Windows 7 [2]
The step-by-step procedure for Min GW, in case using
libfreenect:
1. Install QT opensource for Windows. This will also install Min
GW.
2. Add C:\Special\QtSDK\mingw\bin\ to the Path environment
variable: Start  Computer (properties)  Advanced System
Settings  Environmental Variables (button)  Variable: PATH
 Edit  Add.
3. Install and run cmake on rgbdemo  Disable the
NESTK_USE_OPENNI cmake variable  “Could not find
module FindOpenCV.cmake”  “Configuring incomplete,
errors occurred!”
4. Ungrouped Entries 
 OpenCV_DIR (NOT FOUND)  C:/Special/opencv.
 QT_QMAKE_EXECUTABLE (NOT FOUND) 
C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\qmake.exe 
it is better to use qmake.exe from QtSDK\mingw ??
21/20
Compiling RGBDemo-0.6.1 under Min GW from
source with OpenKinect on Windows 7 [2]
The step-by-step procedure for Min GW, in case using
libfreenect:
 Open the CMakeLists.txt in Qt Creator (from
C:\Special\RGBDemo-0.6.1-Source\RGBDemo-0.6.1Source\CMakeLists.txt open with
C:\Special\QtSDK\QtCreator\bin\qtcreator.exe).  It creates the
folder: qtcreator-build
 Add QT-qmake & OpenCV path to the Path
environment variable:
C:\Special\QtSDK\Desktop\Qt\4.7.4\mingw\bin\
C:\Special\opencv\
to the Path environment variable: Start  Computer
(properties)  Advanced System Settings 
Environmental Variables (button)  Variable: PATH 
Edit  Add.
20/01/2012
22/20
Compiling RGBDemo-0.6.1 under Min GW / MSVS 2010
from source with OpenKinect on Windows 7 [2]
The step-by-step procedure for Min GW / MSVS2010, in case
using libfreenect:
 Cmake 2.8.6  Ungrouped Entries 
PKG_CONFIG_EXECUTABLE 
C:\Special\opencv\OpenCVFindPkgConfig.cmake
Results: Cmake can’t compile the QT & OpenCV_DIR 
OpenCVConfig.cmake  We should use Qt!!
 Qt  Run Cmake  “CMAKE_MODULE_PATH
C:\Special\qtcreator-build\”  Run Cmake 
“CUDA_TOOLKIT_ROOT_DIR not found or specified” [10] 
if set path to C:\Special\PCL_1_4_0\ which contains CUDA
libraries, then we have the messages
“CMake Error: The source directory
"C:/Special/PCL_1_4_0/3rdParty/flann/bin" does not appear to
contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake
GUI”.
20/01/2012
Burrus software “RGBDemo-0.6.1” is not installed on Window7-64bit!!23/20
Conclusions
1. Nicolas Burrus software with OpenNI binaries [2] works
under Windows 7-64 bit, but the software doesn’t give the
extrinsic mapping between Kinect Depth and RGB cameras
and doesn’t produce a file with xyz- coordinates and RGBcolors info of every point!!!
2. Although the drivers of OpenKinect has been successfully
installed under Windows 7-64 bit, the Nicolas Burrus source
code RGBDemo-0.6.1 (with LGPL License) for Kinect
Calibration [2] is not compiled properly under Windows 7-64
bit.  It should be used another software or another Kinect
drivers!!
3. Both Microsoft Kinect SDK for developers [6] and
OpenKinect drivers [9] can be applicable to save RGB images
and IR-Depth data to calibrate Kinect sensor and use it under
Windows 7-64 bit, but no suitable software has not been
produced yet.
25/01/2012
24/2
0
5.
1.
2.
3.
4.
5.
6.
16/01/2012
References
Camera Calibration and 3d Reconstruction. // OpenCV (Open
Source Computer Vision) v2.1 documentation:
http://opencv.willowgarage.com/documentation/cpp/camera_cali
bration_and_3d_reconstruction.html
Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize
Kinect output. // http://nicolas.burrus.name/, 2011.
OpenNI Modules. // www.openni.org,
SensorKinect drivers. // github social coding.
https://github.com/avin2/SensorKinect.
How-to: Successfully install Kinect on Windows (OpenNI and
NITE). // Vangos Pterneas blog:
http://studentguru.gr/b/vangos/archive/2011/01/20/how-tosuccessfully-install-kinect-windows-openni-nite.aspx
Microsoft Kinect SDK for developers.
http://www.microsoft.com/en-us/kinectforwindows/
25/20
5.
7.
8.
9.
10.
16/01/2012
References
Install OpenKinect for Windows 7 and XP. //
http://kinect.dashhacks.com/kinect-guides/2011/01/09/installopenkinect-windows-7-and-xp
Getting Started using OpenKinect. //
http://openkinect.org/wiki/Getting_Started#Windows.
https://github.com/ofTheo/ofxKinect
“OpenCV: CUDA_TOOLKIT_ROOT_DIR not found or specified“.
http://xinyustudio.wordpress.com/2011/11/30/opencvcuda_toolkit_root_dir-not-found-or-specified/
26/20