Sergii Baidachnyi

During Global Developer Conference there were many announcements like Unity 5 availability for free. Of course, Unity has been offering a free version of the editor for many years but it was an editor without many important features. For example it was not possible to use external libraries in free version of the editor. It was the main stopper for many educational projects like projects related to Kinect. From one side we have a very cool solution like Kinect, which researches can use in many interesting projects but from other side, it’s very hard to develop something using DirectX directly. Unity might save the situation but Kinect SDK worked with Pro version only.

But Unity decided to change business model and announced that since Unity 5 there is Personal Edition, which contains all features.

Of course, I decided to return to my projects with Kinect and tried to rebuild some of them using the Unity 5 Personal. And there is great news – everything works fine! So the last stopper was broken and today you can try to use Kinect and Unity for free.

If you are interested in the topic I would recommend to use my previous article about Kinect as an instruction how to start.

In one of my earlier posts I already told about my first experience in Kinect and Unity integration. But at that time I was able to download only the beta version of the package, which didn’t support Face, Face HD and Fusion APIs there. Today, we have access to a release version of Kinect for Windows SDK 2.0 as well as access to Unity Pro packages and if you want to download them right now, you can find all needed links here http://www.microsoft.com/en-us/kinectforwindows/develop/downloads-docs.aspx.

Of course, you still need a Unity Pro for Kinect functionality but if you want to test some features right now, it’s easy to activate 30-days trial version of Unity Pro. It should be enough in order to understand some features of the Kinect as well as to decide if you want to start a business right now.

So, if you already downloaded the Unity Pro package, you could notice that it contains three packages inside (.unitypackage files). The first file Kinect.2.0.1410.19000.unitypackage contains base functionality of Kinect SDK for Unity. It will allow to track bodies, leans, colors and so on. But if you want to use functionality, which relates to face (emotions, face HD tracking etc.) you will need the second package – Kinect.Face.2.0.1410.19000.unitypackage. The last package contains API which will help to use data from Visual Gesture Builder in order to simplify a way to understand predefined gestures.

Before I start to create some code, I want to drag your attention to the hardware part. In order to start working with Kinect 2 SDK you need a Kinect 2 sensor. I know that the last sentence looks stupid but it’s realityJ I just found in Microsoft Store that Kinect 2 for Windows costs around 200 dollars – it’s very expensive, if you just want to test something. But Microsoft announced a solution of this problem as well. If you have Kinect for Xbox One, you will able to buy a special Kinect adapter for Windows, which will allow you to connect existing sensor to PC. The adapter costs around 50 dollars, which is much cheaper than a new Kinect sensor. Because I already have Xbox One, I decided to buy adapter only.

The adapter has a pretty big box, because it’s not just connector to USB. In general, Kinect requires more power than USB can provide, so the adapter allows to connect Kinect to power as well.

Because we already discussed some hardware questions I want to point to some additional requirements there. In order to build something with Kinect you will need to have Windows 8 (x64) operation system, USB 3.0 host controller and DirectX 11 capable graphics adapter.

Finally, we finished with hardware, so let’s create some code.

Let’s start with simple example where we will make some manipulation with cube using the base API. In order to do it I created a new Unity Pro project and imported Kinect.2.0.1410.19000.unitypackage there. Let’s put a cube in front of camera and create a script, which should be associated with the cube. We are going to do all work inside that script.

First of all we need to create some data fields in our class:

private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;

We are going to use KinectSensor in order to access to Kinect. KinectSensor class provides some properties, which allows us to get sources’ references. Kinect supports several sources:

· BodyFrameSource – provides basic information about tracked people (up to 6) like skeleton information, leans, hand states etc.;

· AudioSource – allows to track a sound source from a specific direction;

· BodyIndexFrameSource – shows if a particular pixel relates to body or to background;

· ColorFrameSource – gets a copy of video image, which Kinect got from the camera;

· DepthFrameSource – each pixel of this frame represents a distance between Kinect and tracked objects (up to 4.5 meters);

· InfraredFrameSource – supports black and white frame, which looks good with any sources of lights;

· LongExposureInfraredFrameSource – looks like infrared frame but it supports better quality with less noise. It requires a longer period of time in order to get data;

All of these properties support OpenReader method, which returns the appropriate reader. KinectSensor also supports OpenMultiSourceFrameReader, which allows to get data from several readers just using a single line of code.

Since we need to know basic body movements only, we will use just BodyFrameReader. Additionally, we need an array of Body class in order to store current information about the body.

Right now we are ready to write some initialization methods. Unity supports Start method for MonoBehaviour classes in order to provide a place for initialization. We will use this method to prepare out Kinect and reader.

void Start () 
{
    _Sensor = KinectSensor.GetDefault();

    if (_Sensor != null)
    {
        _Reader = _Sensor.BodyFrameSource.OpenReader();

        if (!_Sensor.IsOpen)
        {
            _Sensor.Open();
        }
    } 
}

In the previous post about Kinect I created bad code and forgot to dispose my objects. It worked fine there but if you make the same mistake here you will be able to get data from Kinect just for the first launch of your game inside Unity. After it you will not able to get any data as well as exceptions or something like it. So, in order to avoid having to restart Unity every time when you launch your application inside, we should include OnApplicationQuit method as well. We will call Dispose method for our objects there.

void OnApplicationQuit()
{
    if (_Reader != null)
    {
        _Reader.Dispose();
        _Reader = null;
    }

    if (_Sensor != null)
    {
        if (_Sensor.IsOpen)
        {
            _Sensor.Close();
        }
        _Sensor = null;
    }
}

Right now we are ready to implement Update method, which will be called on each frame update. This method will contain less Kinect related code. We need to get last frame using AcquireLatestFrame method and, in case if frame exists, we need to initialize Body array. Because Kinect supports up to 6 bodies, we need to create array based on this number but BodyFrameSource supports BodyCount property as well.

Please, don’t forget to dispose frame right after initialization of Body array.

Additionally, we should understand if at least one body was tracked. In order to do it we may use IsTracked property in Body class. If at least one body is tracked we will use its index in order to start moving our cube. Here is my version of the Update method:

void Update()
{
    if (_Reader != null)
    {
        var frame = _Reader.AcquireLatestFrame();

        if (frame != null)
        {
            if (_Data == null)
            {
                _Data = new Body[_Sensor.BodyFrameSource.BodyCount];
            }

            frame.GetAndRefreshBodyData(_Data);

            frame.Dispose();
            frame = null;

            int idx = -1;
            for (int i = 0; i < _Sensor.BodyFrameSource.BodyCount; i++)
            {
                if (_Data[i].IsTracked)
                {
                    idx = i;
                }
            }
            if (idx>-1)
            {
                if (_Data[idx].HandRightState != HandState.Closed)
                {
                    horizontal = 
                        (float)(_Data[idx].Joints[JointType.HandRight].Position.X 
                        * 0.1);
                    vertical = 
                        (float)(_Data[idx].Joints[JointType.HandRight].Position.Y 
                        * 0.1);

                    if (firstdeep == -1)
                    {
                        firstdeep = 
                            (float)(_Data[idx].Joints[JointType.HandRight].Position.Z 
                            * 0.1);
                    }
                    deep = 
                        (float)(_Data[idx].Joints[JointType.HandRight].Position.Z 
                        * 0.1) - firstdeep;

                    this.gameObject.transform.position = new Vector3(
                        this.gameObject.transform.position.x + horizontal, 
                        this.gameObject.transform.position.y + vertical, 
                        this.transform.position.z + deep);
                }
                if (_Data[idx].HandLeftState != HandState.Closed)
                {
                    angley = 
                        (float)(_Data[idx].Joints[JointType.HandLeft].Position.X );
                    anglex = 
                        (float)(_Data[idx].Joints[JointType.HandLeft].Position.Y);
                    anglez = 
                        (float)(_Data[idx].Joints[JointType.HandLeft].Position.Z);

                    this.gameObject.transform.rotation =
                        Quaternion.Euler(
                        this.gameObject.transform.rotation.x+anglex * 100,
                        this.gameObject.transform.rotation.y+angley * 100,
                        this.gameObject.transform.rotation.z+anglez * 100);
                }
            }
        }
    }
}

As you can see, I used HandLeftState and HandRightState in order to change properties of cube. User will be able to “close” his hand in order to avoid cube movement. In order to use z axis I initialize deep variable in current position of user’s hand by z because the axis shows distance between Kinect and user’s hand. But thanks to that initialization I am able to move cube backward or forward by z.

Next time we will create more advanced examples based on DepthFrameSource and AudioSource.