OpenCV with Kinect - Windows SDK
What is OpenCV?
OpenCV is a library for Computer Vision - it is a standard library that simplifies many real-time computer vision problems.
OpenCV with Kinect
This entry provides a solution to use the Kinect (using the Windows SDK) as an input to OpenCV to process the depth and RGB streams received from the Kinect.
Windows Kinect SDK
This project uses the Windows Kinect SDK, available at http://www.microsoft.com/en-us/kinectforwindows/develop/ The solution can also be integrated with previous versions of the SDK, since the functions exposed are the same. However this has not been tested as yet.
1) To use the Windows SDK in Unity, it is required that you restart Unity every time you hit the play button and complete your playthrough. Hitting play another time without restarting Unity will cause it to crash. Running executables multiple times works fine. One work around for this is to comment out the function call to NUIShutdown in KinectSensor.cs the first time play is hit. Following this, before hitting play again, comment out the call to NUIInitialize in the same script. This will allow you to work with the Kinect multiple times. Remember, that if you use this workaround, NUIInitialize needs to be called again if you close Unity and open it again.
2) The example for gesture tracking provided on this page uses the near mode, which is only available on the Windows Kinect. The example uses hand tracking, and is configured for near mode. The algorithms, methods and general working structure are applicable to both regular mode and near mode.
Downloading and installing
- Download the Windows Kinect SDK from http://www.microsoft.com/en-us/kinectforwindows/develop/
- Download OpenCV Sharp files at OpenCV DLLs - 64 Bit
- Copy and paste these DLLs into the C:/Windows/SysWOW64 on your computer for a 64 bit OS.
- Download the OpenCV and Kinect Example Project - the Plugins folder has the required OpenCVSharp dlls that are required to run OpenCV in Unity
Integrating with Unity
Almost all public functionality of the Unity Wrapper is the same as the old SDK. For a detailed description of the scripts and behaviors of the Kinect Wrapper for Unity review Microsoft Kinect - Microsoft SDK
One additional feature on KinectSensor.cs is a checkbox for near mode. This will not be useful on the XBOX360 Kinect, but opens up the near mode functionality on the Windows Kinect hardware. Keep in mind, there is no skeletal data available when in near mode - making the use of OpenCV necessary for any type of tracking.
This implements hand gesture tracking, with an enumeration to return the various gestures that are tracked, as well as palm position for two hands. This is intended as an example of how to use OpenCV for a specific purpose - you can use the code as is, or track something different using this file as a template, since it shows how to get data from the Kinect.
- Automation of workaround - start and stop Kinect buttons for Windows SDK
- OpenCV + Kinect version for old Kinect SDK