Finger Tracking using Kinect v2

By January 24, 2016C#, CodeProject, Kinect, Videos
Kinect Finger Tracking

During the past few days, I have been working on a really cool personal project: finger tracking using Kinect version 2! As always, I’m providing the source code to you.

If you want to understand how things work, keep reading!

Prerequisites

Finger Tracking Algorithm

While developing the project, I had the following facts in mind: the algorithm should be able to track both hands and also know who those hands belong to. The Kinect SDK provides us with information about the human bodies. The finger tracking algorithm should be able to extend this functionality and assign fingers to specific Body objects.

Existing finger tracking algorithms simply process the depth frame and search for fingers withing a huge array (512×424) of data. Most of the existing solutions provide no information about the body; they only focus on the fingers. I did something different: instead of searching the whole 512×424 array of data, I am only searching the portion of the array around the hands! This way, the search area is limited tremendously! Moreover, I know who the hands belong to.

So, how does the finger tracking algorithm work?

Step 1 – Detect the Hand joints

This is important. The algorithm starts by detecting the Hand joints and the HandStates of a given Body. This way, the algorithm knows whether a joint is properly tracked and whether it makes sense to search for fingers. For example, if the HandState is Closed, there are no fingers visible.

If you’ve been following my blog for a while, you already know how to detect the human body joints:

Joint handLeft = body.Joints[JointType.HandLeft];
Joint handRight = body.Joints[JointType.HandRight];

If you need more information about how Kinect programming works, read my previous Kinect articles.

Step 2 – Specify the search area

Since we have detected the Hand joints, we can now limit the search area. The algorithm only searches within a reasonable distance from the hand. What exactly is a “reasonable” distance? Well, I have chosen to search within the 3D area that is limited between the Hand and the Tip joints (10-15 cm, approximately).

Kinect Finger Tracking Search Area

Step 3 – Find the contour

This is the most interesting step. Since we have strictly defined the search area in the 3D space, we can now exclude any depth values that do not fall between the desired range! As a result, every depth value that does not belong to a hand will be rejected. We have an, almost perfect, shape of a hand. The outline of this shape is the contour of the hand!

Kinect Finger Tracking Contour

Step 4 – Find the Convex Hull

The contour of a hand is a big set of points. However, only five (or less) of these points correspond to valid fingertips. The fingertips are the edges of a polyline that contains all of the contour points. In the Eucledian space, this is called “convex hull”.

Kinect Finger Tracking Convex Hull

Consequently, the edges of the convex hull above the wrist define, well, the fingers.

Kinect Finger Tracking Complete

So, you now understand how the algorithm works. Let’s see how to use it.

How-to-use

While building this project, my goal was simple: I did not want you to mess with the all of the complexity. So, I encapsulated all of the above algorithm into a handy class: HandsController.

Finger Tracking is under the LightBuzz.Vitruvius.FingerTracking namespace. This namespace should be imported whenever you need to use the finger tracking capabilities.

using LightBuzz.Vitruvius.FingerTracking;

Everything is encapsulated into the HandsController class. To use the HandsController class, first create a new instance:

private HandsController _handsController = new HandsController();

You can specify whether the controller will detect the left hand (DetectLeftHand property), the right hand (DetectRightHand property), or both hands. By default, the controller tracks both hands.

Then, you’ll need to subscribe to the HandsDetected event. This event is raised when a new set of hands is detected.

_handsController.HandsDetected += HandsController_HandsDetected;

Then, you have to update the HandsController with Depth and Body data. You’ll need a DepthReader and a BodyReader (check the sample project for more details).

private void DepthReader_FrameArrived(object sender, DepthFrameArrivedEventArgs e)
{
  using (DepthFrame frame = e.FrameReference.AcquireFrame())
  {
    if (frame != null)
    {
      using (KinectBuffer buffer = frame.LockImageBuffer())
      {
        _handsController.Update(buffer.UnderlyingBuffer, _body);
      }
    }
  }
}

Finally, you can access the finger data by handling the HandsDetected event:

private void HandsController_HandsDetected(object sender, HandCollection e)
{
  if (e.HandLeft != null)
  {
    // Contour in the 2D depth space.
    var depthPoints = e.HandLeft.ContourDepth;

    // Contour in the 2D color space.
    var colorPoints = e.HandLeft.ContourColor;

    // Contour in the 3D camera space.
    var cameraPoints = e.HandLeft.ContourCamera;

    foreach (var finger in e.HandLeft.Fingers)
    {
      // Finger tip in the 2D depth space.
      var depthPoint = finger.DepthPoint;

      // Finger tip in the 2D color space.
      var colorPoint = finger.ColorPoint;

      // Finger tip in the 3D camera space.
      var cameraPoint = finger.CameraPoint;
    }
  }

  if (e.HandRight != null)
  {
    // Do something with the data...
  }
}

And… That’s it! Everything is encapsulated into a single component!

Need the position of a finger in the 3D space? Just use the CameraPoint property. Need to display a finger point in the Depth or Infrared space? Use the DepthPoint. Want to display a finger point on the Color space? Use the ColorPoint.

Download the source code

Til next time… Keep Kinecting!

PS: Vitruvius

If you enjoyed this post, consider checking Vitruvius. Vitruvius is a set of powerful Kinect extensions that will help you build stunning Kinect apps in minutes. Vitruvius includes avateering, HD Face, background removal, angle calculations, and more. Check it now.

Author Vangos Pterneas

Vangos Pterneas is an award-winning Microsoft Most Valuable Professional. He is helping companies from all over the world grow their revenue by creating profitable software products. He loves Motion Technology and Mixed Reality. Vangos is the CEO of LightBuzz Inc and author of The Dark Art Of Freelancing.

More posts by Vangos Pterneas

Join the discussion 29 Comments

  • […] Vangos Pterneas Kinect Finger Tracking […]

  • George says:

    The code is brilliant. I have ran the solution and it works wonderfully. Just a question I have, currently I am working on a project that tracks hands, detects fingertips and then traces one of the finger tip location to write in air and then run character recognition on the trace.

    I was wondering if it would be possible to record the trace of just one fingertip, lets say the right hand, index finger?

    • Hi George. Thank you for your comment. To identify each finger type, you need to e.g. sort them based on their distance from the thumb or distance from the palm. This way, you’ll be able to track a specific tip per second and draw its trace.

  • Mughees Ismail says:

    1) Good algorithm, but I’ve noticed one thing that it’s not very fast.
    Why didn’t you try tracking fingers through machine learning like the rest of body joints?
    2) Is there a way to get the finger joints too and not just the tips?

    • Hi Mughees.

      1) The algorithm is fast (and it’s actually one of the fastest algorithms out there). If you are getting a delay, that’s because you are drawing a lot of ellipses on your Canvas. To reduce this lagging, simply turn on the “Optimize Code” checkbox in your Project Properties.

      2) Right now, you can only detect the tips.

  • Luis says:

    Congratulations for your work, it’s really awesome what you`ve done, unfortunately I have problems to run the application, when I try to run the program the next error message appear: “visual studio cannot start debugging because the debug target is missing ‘C:/Users/ etc etc..’ Please build the project and retry, or set the OutputPath and AssemblyName properties appropriately to point at the correct location for the target assembly”.
    May you help me to resolve this issue?
    Thank you very much.

    • Hi Luis. Right-click the project named “KinectFingerTracking” and select “Set as startup project”. Then, hit Debug to run the project again. I guess that would solve your problem.

  • Muhammad Usman says:

    have you achieved this on unity3d project too ?

  • Hannan says:

    Hello Mr.Pterneas

    can I ask you .. after drawing the fingertips , is there any way can I compare the distance between them to know if the fingers hold something or not?

    thank you very much

  • hanan says:

    Hello,
    can you explain to me the following and why use it ?

    LockImageBuffer()

    buffer

    UnderlyingBuffer

    Thanks

    • Hello Hanan. The Underlying Buffer is the native representation of the depth information. It’s equivalent to using the ushort[] byte array. It’s just a little faster than using CopyFrameDataToArray. All it contains is a pointer to an array of depth values. Locking the buffer is necessary before accessing it to avoid leaks.

  • hanan says:

    so , why we need to convert the frame to bitmap image

    • Because you need to convert a byte array to a Bitmap object. Otherwise, you won’t be able to display it. XAML is using BitmapImage and WriteableBitmap. Unity is using Texture2D. Each class is a “wrapper” of byte arrays.

  • Neo Cheng says:

    Hi, Mr.Pterneas:
    It’s really a brilliant Method, but it seems can only used in some special situation with convex hull. I’m a student who also researching Kinect based finger tracking in China, counld you show me any idea of tracking other finger joints?

    • Hello Neo. Thanks for your comment. To perform accurate finger joint tracking for the other joints, you’d better use the Leap Motion sensor. It’s much more accurate than Kinect.

  • Pau says:

    Hello!
    Can you do something similar with kinect 360 (v1) , with your code?

  • Abdulhafiz Jeylan says:

    How to determine whether the index finger is pointing to the camera or not ..by calculating the distance

  • Abdulhafiz Jeylan says:

    thanks for your attention sir , am thinking to develop something like for example counting fingers….just like this demo on youtube
    https://www.youtube.com/watch?v=tRiLaPwN2BU

  • madonna sabry says:

    mr.pterneas
    i am a student researching on kinect please help me i want to display only the hand within contour not to show body and every thing else then i want to capture this image
    thanks a lot for your useful tutorials

    • Hello. You can simply toggle the depth display off and do not feed the Image element with depth bitmaps.

      • madonna sabry says:

        i am new in kinect and i am not completely understand you
        may you please express it in code

        • It is quite a lot of code to write it all down. You need to process the depth frame (an array of ushort values) and select the values that correspond to the hands. Copy those values to a new array of the same size, while leaving the remaining values equal to zero.

          This way, you’ll only display the points that belong to the hands and remove everything else.

Leave a Reply