Kinect for Windows version 2: Hand tracking

NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change.

Kinect is awesome for body tracking, but Kinect version 2 is more than finding the human joints positions. Today we’ll see how we can track hands and thumbs and specify the hands’ state. In this article, we’ll extend the functionality we built in the previous blog posts, so you might want to revise them before going on.

• The color, depth and infrared blog post will guide you through the way of initializing the Kinect sensor and displaying the streams
• The body tracking blog post will show you how to display the human body joints

Download the source code of this example

Requirements

• Kinect for Windows v2
• Windows 8/8.1
• Visual Studio 2013
• USB 3.0 port

Video

Here is a brief video I created demonstrating the functionality we are going to examine (cudos to my young sister).
[youtube code=”zM6Xcuja04I”] [Tweet “Hand states and hand tracking using Kinect”]

Hand tracking

Finding the position of a hand or thumb is just like finding the position of every other joint. Remember the snippet we used before? Here’s how you can get the position of each hand:

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    var reference = e.FrameReference.AcquireFrame();
    // Color
    // Display the color stream...
    // Body
    using (var frame = reference.BodyFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            canvas.Children.Clear();
            _bodies = new Body[frame.BodyFrameSource.BodyCount];
            frame.GetAndRefreshBodyData(_bodies);
            foreach (var body in _bodies)
            {
                if (body != null)
                {
                    if (body.IsTracked)
                    {
                        // Find the joints
                        Joint handRight = body.Joints[JointType.HandRight];
                        Joint thumbRight = body.Joints[JointType.ThumbRight];
                        Joint handLeft = body.Joints[JointType.HandLeft];
                        Joint thumbLeft = body.Joints[JointType.ThumbLeft];
                    }
                }
            }
        }
    }
}

Displaying these joints is easy. Simply call the DrawPoint method we created before. In today’s example, I have slightly modified the source code to draw different colors:

// Find the joints
Joint handRight = body.Joints[JointType.HandRight];
Joint thumbRight = body.Joints[JointType.ThumbRight];
Joint handLeft = body.Joints[JointType.HandLeft];
Joint thumbLeft = body.Joints[JointType.ThumbLeft];
// Draw hands and thumbs
canvas.DrawPoint(handRight);
canvas.DrawPoint(handLeft);
canvas.DrawPoint(thumbRight);
canvas.DrawPoint(thumbLeft);

Hand states

Currently, Kinect supports the following hand states. They are parts of the HandState enum:

• Open
• Closed
• Lasso
• Unknown
• NotTracked

Note that the hand states are members of the body object, regardless of the corresponding hand joints.. Here’s the way to access the states:

if (body.IsTracked)
{
    // Find the right hand state
    switch (body.HandRightState)
    {
        case HandState.Open:
            break;
        case HandState.Closed:
            break;
        case HandState.Lasso:
            break;
        case HandState.Unknown:
            break;
        case HandState.NotTracked:
            break;
        default:
            break;
    }
    // Find the left hand state
    switch (body.HandLeftState)
    {
        case HandState.Open:
            break;
        case HandState.Closed:
            break;
        case HandState.Lasso:
            break;
        case HandState.Unknown:
            break;
        case HandState.NotTracked:
            break;
        default:
            break;
    }
}

Can’t be any easier!

After that, I simply created some <TextBlock> elements (named tblRightHandState and tblLeftHandState respectively) and updated each hand state. Here’s the complete source code:

using (var frame = reference.BodyFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        canvas.Children.Clear();
        _bodies = new Body[frame.BodyFrameSource.BodyCount];
        frame.GetAndRefreshBodyData(_bodies);
        foreach (var body in _bodies)
        {
            if (body != null)
            {
                if (body.IsTracked)
                {
                    // Find the joints
                    Joint handRight = body.Joints[JointType.HandRight];
                    Joint thumbRight = body.Joints[JointType.ThumbRight];
                    Joint handLeft = body.Joints[JointType.HandLeft];
                    Joint thumbLeft = body.Joints[JointType.ThumbLeft];
                    // Draw hands and thumbs
                    canvas.DrawPoint(handRight);
                    canvas.DrawPoint(handLeft);
                    canvas.DrawPoint(thumbRight);
                    canvas.DrawPoint(thumbLeft);
                    // Find the hand states
                    string rightHandState = "-";
                    string leftHandState = "-";
                    switch (body.HandRightState)
                    {
                        case HandState.Open:
                            rightHandState = "Open";
                            break;
                        case HandState.Closed:
                            rightHandState = "Closed";
                            break;
                        case HandState.Lasso:
                            rightHandState = "Lasso";
                            break;
                        case HandState.Unknown:
                            rightHandState = "Unknown...";
                            break;
                        case HandState.NotTracked:
                            rightHandState = "Not tracked";
                            break;
                        default:
                            break;
                    }
                    switch (body.HandLeftState)
                    {
                        case HandState.Open:
                            leftHandState = "Open";
                            break;
                        case HandState.Closed:
                            leftHandState = "Closed";
                            break;
                        case HandState.Lasso:
                            leftHandState = "Lasso";
                            break;
                        case HandState.Unknown:
                            leftHandState = "Unknown...";
                            break;
                        case HandState.NotTracked:
                            leftHandState = "Not tracked";
                            break;
                        default:
                            break;
                    }
                    tblRightHandState.Text = rightHandState;
                    tblLeftHandState.Text = leftHandState;
                }
            }
        }
    }
}

What’s next?

We are expecting a new SDK update from Microsoft in the next few days, so stay tuned for more great staff like facial expressions.

• Download the source code of this example

PS 2: New Kinect book – 20% off

Well, I am publishing a new ebook about Kinect development in a couple months. It is an in-depth guide about Kinect, using simple language and step-by-step examples. You’ll learn usability tips, performance tricks and best practices for implementing robust Kinect apps. Please meet Kinect Essentials, the essence of my 3 years of teaching, writing and developing for the Kinect platform. Oh, did I mention that you’ll get a 20% discount if you simply subscribe now? Hurry up 😉