Navigating through a Natural User Interface using your palm is quite common – after all, it’s the primary navigation mechanism XBOX uses. Many Windows Kinect applications implement hand tracking for similar purposes. Today, I would like to share a Kinect hand cursor control I developed and you can use for your own apps. This hand cursor control will save you tons of time and you’ll be able to integrate it right into your existing WPF code!
Here is the final result of this handful user control:
Using the control in your project is fairly easy. Read on!
OK, let’s type some quick code now.
Step 1: Download the project from GitHub
Download the source code and build it using Visual Studio. Locate the assembly named KinectControls.dll.
Step 2: Import the assembly to your project
Create a new WPF project and add a reference to the assembly you built previously.
Step 3: Import the assembly to your XAML code
Type the following line of code in your Window, view or user control definition:
You can now drag a Canvas and place the hand cursor inside it:
<Canvas Name="canvas" Width="640" Height="480"> <Controls:KinectCursor x:Name="cursor" Width="100" Height="100" /> </Canvas>
Step 4: Move the cursor using C#
The KinectCursor class contains a method named Update. You need to call that method whenever you want to move the cursor. The Update method gets the X and Y coordinates of the hand in the 2D color or depth space. Alternatively, you can provide a ColorSpacePoint or DepthSpacePoint that contains the coordinates. I placed this code inside SkeletonFrameReady event handler. You can specify the “active” hand by comparing the Z values of the left and right hands.
// Select the hand that is closer to the sensor. var activeHand = handRight.Position.Z <= handLeft.Position.Z ? handRight : handLeft; var position = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint( activeHand.Position, ColorImageFormat.RgbResolution640x480Fps30); cursor.Flip(activeHand); cursor.Update(position);
Also, notice the Flip method? The Flip method mirrors the cursor visual, so to properly match the active hand. That’s it! You can check the complete source code on GitHub.
The hand image is a scalable vector shape I designed. You can change its color, dimensions or shadow effects easily:
<Controls:KinectCursor x:Name="cursor" Width="400" Height="400" Fill="Blue" />
You are free to use the user control as you wish for your personal and commercial projects, just by making a simple attribution in your project.
If you enjoyed this article, then you’ll love Vitruvius. Vitruvius is a set of powerful Kinect extensions that will help you build stunning Kinect apps in minutes. Vitruvius includes avateering, HD Face, background removal, angle calculations, and more. Check it now.
Want to hire my team for your next cutting-edge fitness app? Drop me an email.
Your “Copyrights” isn’t a copyright message, but rather a simple, non-standard license that actually contradicts the GPL license for your project on GitHub.
Hey Chris. Thank you very much for noting that issue! I changed the license to MIT License. You can see the update on GitHub. I just want everyone to use this control for free in personal or commercial projects.
Have I mentioned that I hate lawyers? 😉
I didn’t try your code yet
but I think it only works in your application right?
will it work outside your application?
like controlling the windows cursor.
Hi tcboy. The code works on any application. You simply provide the required coordinates. It will not control the mouse, as this is not a good practice for most applications.
If you need a more solid solution, try Microsoft Kinect Toolkit and KinectRegion.
I didn’t tried you code till now.
Does your code work also in a Windows store app?
And is your code compatible with kinect v2 sensor?
Hi Helumt. Yes, this cursor control is compatible with Kinect v2 🙂
Hello, it’s July now. When can we see the new book 🙁
Hi Chris. Please be patient, as the book is under technical and lingual review. Thank you 🙂
Hey, I just built the source against the latest Kinect v2.00 pre-release SDK (1406) and it fails to build due to DepthImagePoint & DepthImagePoint structures?
Hi Ken. Since you are using Kinect v2, simply replace ColorImagePoint with ColorSpacePoint and DepthImagePoint with DepthSpacePoint.
Qué SDK usas? 1.7 ?
We are stuck at using kinect to bind with the mouse which is done by your application.
FIrst of all thanks a lot for the things you post on your blog regading to the kinect and I am waiting for your book.
I have built your code in VS and included it in my project.
I am working on a kinect V2 app so it’s worth to mention that the xaml code to include the reference for v2 would look like this
including the code and the cursor control went just fine and compiled with no errors.
Now, could you please help me on how to update the cursor in the Kinect V2 code?
var position = ??
and should I put this bit in the event where I read a body frame or what?
Hi George. Here is the updated code for v2. I’m glad that you find my tutorials helpful.
Hey! Thank you for yours great work!
I have similar question, I have Kinect v2 and SDK 2.0.
I am not sure how this line should looks:
var position = _sensor.CoordinateMapper.MapSkeletonPointToColorPoint(activeHand.Position,ColorImageFormat.RgbResolution640x480Fps30);
Could you help me?
Hi Michal. You can use the v2 version from the same GitHub repository. The line should be something like this:
var position = _sensor.CoordinateMapper.MapCameraPointToColorSpace(activeHand.Position);
this is actually a very nice example. i just want to know whether it is possible for Kinect to detect hand gesture when the Hand position is in horizontal mode, like we actually doing with mouse.
i am trying to detect button click using hand push like in example but in my scenario the palm will not be facing the kinect and hand will be in horizontal position and move downward instead of forward.
Hi Ravi. Are you trying to perform a “push” gesture? That is, moving your hand back and forth very quickly. You could check 15-20 frames and compare the relative distance between the hand and the sensor. 15-20 frames is the window size you’d need to use (aka the “time” between a back-and-forth gesture).
Yeah i have tried that and it works but in my case palm is not facing the sensor actually, my hand is in horizontal position i.e. the fingertip is facing the sensor and hand is moving from upwards to downwards.
It should be the same algorithm. Simply consider the Y value instead of the Z value.
I have a question not exactly related to this article. I have followed this post ( https://social.msdn.microsoft.com/Forums/en-US/970aa6d7-da26-4dc1-9445-90d09d6a15eb/doing-drag-drop-with-handpointers?forum=kinectv2sdk ) to implement drag and drop functionality in my application. I have three user controls (Left,Middle,Right) that are being hosted in one main user control. In my Left UC i have three images so if a user drags and drops any of these three images in Right UC with the help of Hand Cursor, i want the dropped image to be copied in Right UC (or a new instance of the image is created which is visible in Right UC or a snapshot is taken and shown in Right UC which can be viewed later). The original image in Left UC should be at the same place before the drag and drop operation.
Can you please guide me how can i implement this with Kinect V2 Hand Cursor as there is no support for copy and paste operations as it is for mouse cursor and also no events are fired when the hand cursor is over any control or it enters or leaves a specific control. Thank you.
Hi. To implement the hover states, you need to check whether two visual elements are overlapping each other.
Check this StackOverflow thread for more.
I was just wonderin, where is the KinectControls.dll file? It’s not in the directory you provided.
Pls send the file to my email at firstname.lastname@example.org.
Hi Anand. The dll files are generated when you build the project for the desired platform. Download the code, open with Visual Studio and build.
I‘m a beginner and i found your blog fortunately.Your work is wonderful . Currently i use Kinect v2 with SDKv2.0_1409,but i find the source code doesn’t work. VS2013 shows me about 22 errors.What should i do?Thank you!
Hello Jack. What errors are you getting?
i am trying to control mouse courser by kinect 360 for some pursuit rotatory games.
but i dont know any computer language. what can i do. is it possible guide me by pic or send me their codes.
Hi Hasan. This is a custom project, so you’ll need to work with a developer.
Hi! Can I use it for WinForm application?
Yes, but you’ll need to find the alternatives of the Canvas. The core logic would be the same, but WinForms is using different visual components.
Hi! First, I want to thank you for the awesome work you do!
Second, I would like to know if its possible to change the cursor?
Thank you very much 🙂
All you need to do is simply replace the PNG image of the hand. The image can be found inside the Images folder.
That didnt seems to work, I changed the PNG image and build the project again to get the dll.
Did I missed something?
Also, the original PNG image has more besides the “hand”.
Mea culpa. You need to go to the KinectCursor.xaml file and replace the Path element with an Image element. You’ll give the PNG as a parameter to your Image element. Hope this helps.
That’s just what I wanted!!
Thank you very much! 😀
hey, can i use this on unity 3D?