Archive for October 2014
Kinect, Unity 3D and Debugging
In this article, I was planning to make additional investigation of Kinect 2.0 SDK but I found a better theme – Debugging of Unity 3D applications using Visual Studio. Because I am not a professional developer in Unity 3D yet, I never thought about debugging of Unity 3D applications. But in the last article, I wrote about my first experience in Kinect and Unity integration and, frankly speaking, I had a headache during preparation of my demo. It happened due to the lack of ability to debug my demo scripts. Of course, Kinect 2 SDK was new for me, so I used many Debug.Log methods in order to understand what was happening with my variables.
I know that Mono Develop allows to debug Unity scripts but I don’t like this tool. So, I did not debug my scripts for some time and returned to this idea only two days ago. I was very surprised that there is an easy way to enable Debugging tools in Visual Studio. In order to do it you need to download and install Visual Studio Tools for Unity.
These tools were created by SyntaxTree team, which was bought by Microsoft in June-July this year. At once Microsoft announced availability of the tools for all developers for free and the first Microsoft version of them was presented in the end of July. The tools are integrated with Visual Studio 2010, 2012 and 2013 versions, so you can use them with older version of Visual Studio.
I have Visual Studio 2013, so I simply installed the package and enabled debugging in my project.
In order to enable debugging in an existing project you need to import package to it. If you install the tools successfully, you will find a new menu item there.
If you create a new project you will be able to add the package right in Create New Project dialog.
The package will add some scripts to your project, which will extend your standard menu. Additionally, the package contains some components that will support communication channel between Visual Studio and Unity.
In my case, I imported the package successfully, so I opened project in Visual Studio in order to start debugging. It was pretty simple, I just put some breakpoints there and clicked the F5 hot key in order to start debugging. Visual Studio switched to the Debug mode and I returned to Unity in order to start my application there. After that, I just clicked Start button in Unity and, finally, I was moved to Visual Studio in order to see a fired breakpoint.
You will be able to click and unclick Play (Pause) button as many time as you want. It will restart your game or just pause it but it will not affect Visual Studio which will stay in the Debug mode. If you want to exit the Debug mode in Visual Studio, you should just click Stop Debugging there.
To summarize, I spent a minute but I got a very powerful tool for my future Unity projects.
Kinect 2 and Unity 3D: My first experience
Several days ago, I got my first Kinect 2 for Windows devices in order to prepare my presentation for Developer Day event in Ukraine together with Dmytro Mindra. Right now, I already have some experience about Kinect and I am going to share it in my blog.
Before we create some code, we will need to download Kinect for Windows SDK 2.0. This SDK is in Public Preview now but it allows to create Windows Store applications (previous version didn’t allow it), so I was happy to install this version of SDK. I found it here: http://www.microsoft.com/en-us/download/details.aspx?id=43661.
Once SDK was installed I was able to review some examples and tools. I would recommend to use SDK Browser, that can be found in the following directory C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Tools\SDKBrowser. This is a very useful tool, which will show you the entire SDK structure like tools, examples and link to docs.
I would recommend to use the Body Basic-XAML example in order to understand that your Kinect works fine or you may use Kinect Studio in order to do some experiments with the sensorJ Because my Kinect was working fine I could start to think about some demos.
Kinect is a very powerful device and you can use it in many ways as a sensor for your home robot or as a part of your security system but the most popular way of Kinect usage is for entertainment. It allows to create many good games and many of such games are already available for Xbox. At same time I don’t have any desire to create a game in C++/DirectX and I was trying to find a solution for Unity 3d because it is one of the best game engines and I like Unity 3d.
Some time ago, I heard information about an add-in for Unity 3d but I had to spend much time in order to find it. I used several search systems, then I called Unity 3d guys and finally I found it in a very unusual place – on the Kinect for Windows SDK 2.0 download page. This link is collapsed by default and you need to expand System Requirements section in order to find it.
Pay attention that this add-in will work with Unity PRO version only, because the free version of Unity doesn’t support add-ins. Probably, there is a way to fix it for Windows Store applications because Non-Pro version allows to use add-ins for this type of applications but I will check it later and for now I use a Pro version.
If you download the add-in successfully, you will be able to find the unitypackage file as well as two examples inside of it. We already have a small game, which allows us to kill “friendly green men” with our battle stellar ship and we decided to implement some Kinect ready activities to make killing process funnier.
Finally, we can implement three steps, which will allow us to use our body during the game in killing process.
On the first step, we found our MonoBehaviour that was responsible for movement of our ship and we added some new code to it:
private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;
This code should be clear but I will describe it. In order to get access to sensor we needed KinectSensor class, which allows to initialize our Kinect and provides needed sources. In our case, we needed access to BodyFrame source, so we created BodyFrameReader variable that we used in order to read data from the source. Finally, we created Body array in order to store our data from the source.
On the next step, we needed to initialize Kinect and open our reader. That’s why we implemented the following code:
void Start()
{
_Sensor = KinectSensor.GetDefault();
if (_Sensor != null)
{
_Reader = _Sensor.BodyFrameSource.OpenReader();
if (!_Sensor.IsOpen)
{
_Sensor.Open();
}
}
}
We initialized the Kinect inside the very familiar method for Unity 3D developers and opened our reader there.
Finally, we realized a method, which was called from Update method of our Behaviour class. This method collected data from the sensor and calculated new position for the ship as well as fire ability:
horizontal = 0;
vertical = 0;
fireButton = false;
if (_Reader != null) {
var frame = _Reader.AcquireLatestFrame ();
if (frame != null) {
if (_Data == null) {
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
frame.GetAndRefreshBodyData (_Data);
int idx = -1;
for (var i = 0; i < 6; i++) {
if (_Data [i].LeanTrackingState !=
TrackingState.NotTracked) {
idx = i;
}
}
if (idx > -1) {
fireButton =
_Data [idx].Joints [JointType.HandLeft].Position.Y > 0;
horizontal = _Data [idx].Lean.X;
vertical = _Data [idx].Lean.Y;
}
}
frame.Dispose();
}
This code is more complex and Dima said that I should not show it because it’s not optimal but, probably, you will create the same code for your first Kinect application for Unity 3DJ
So, in this code, we used the Reader to grab current (latest) body frame that was needed in order to found bodies data inside. Kinect 2 supports up to 6 bodies, so _Sensor.BodyFrameSource.BodyCount should return 6. Using the latest frame we filled Body array in order to aggregate data for each body in our frame. At this point we finished work with Kinect sensor and tried to analyze bodies data. But, if you have just one ship (in our case we had two) and you don’t have any specific interface for Kinect initialization, you will not know about location of real body in the array. That’s why we were trying to find a specific index of the body of a pilot. Because we used leans as a main control mechanism for our ship we checked every body and we were trying to find LeanTrackingState status. If we find a tracked body we fixed index of this body and recalculated position of the ship as well as the fire ability.
I think that we spent about 30 minutes for this example and our presentation was very funny. If you know Russian, you can find our video there: http://devday.in.ua. And I just cut a short video from our presentation there are two guys (me and Dima) were doing a strange things on the stageJ
In the next article I am planning to integrate Kinect with simple Windows 8 application and will discuss different sources of data.
Sergii Baidachnyi 