Kinect for Windows version 2: Color, depth and infrared streams

A month ago, I was happy to receive a brand-new Kinect for Windows version 2 Developer Preview sensor. You can read my previous blog post about the capabilities of the new device. Kinect v2 now includes 5 main types of input streams:

• Color
• Depth
• Infrared
• Body
• Audio

Today I will show you how you can acquire and display each bitmap input in a Windows application. In the next blog post, we’ll talk about body tracking. Here is a quick video I made which demonstrates the different streams provided by the new sensor.

You can download the source code here. Read on if you want to implement this functionality yourself.

Requirements

• Kinect for Windows v2
• Windows 8/8.1
• Visual Studio 2013
• USB 3.0 port

Creating a new project

Come on, launch Visual Studio and create a new WPF application. Make it target .NET framework 4.5 and give it a proper name. I named it KinectStreams. In the XAML file, add an <Image> element and three <Button> controls. The image’s source will be updated every time we have a new Kinect frame. The buttons simply switch between color, depth and infrared frame types.

<Grid>
    <Image Name="camera" />
    <Grid>
        <StackPanel>
            <Button Content="Color" Click="Color_Click" />
            <Button Content="Depth" Click="Depth_Click" />
            <Button Content="Infrared" Click="Infrared_Click" />
        </StackPanel>
    </Grid>
</Grid>

Initializing the sensor

Let’s now dive into the C# code! The primary namespace which provides us with the Kinect-related functionality remains the same:

using Microsoft.Kinect;

Do not forget to add a reference to the corresponding dll file (you’ll normally find it under C:\Program Files\Microsoft SDKs\Kinect\v2.0-DevPreview1311\Assemblies\Microsoft.Kinect.dll).

You now need two classes for connecting to the sensor and reading its camera streams. The first one is KinectSensor class. The second, and this is new to Kinect v2, is MultiSourceFrameReader class:

KinectSensor _sensor;
MultiSourceFrameReader _reader;

Comparing to the first SDK, Kinect SDK v2 Developer Preview provides an easier way for accessing the active sensor. Here’s how you connect to the sensor and start it up:

_sensor = KinectSensor.GetDefault();
if (_sensor != null)
{
    _sensor.Open();
}

Reading the streams

That’s it! We have now connected to the sensor. Let’s specify which streams we need:

_reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
                                             FrameSourceTypes.Depth |
                                             FrameSourceTypes.Infrared);
_reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;

The MultiSourceFrameArrived event fires when a frame of the specified types is available. Here’s how we can open the color, depth and infrared frames in a single method.

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    // Get a reference to the multi-frame
    var reference = e.FrameReference.AcquireFrame();
    // Open color frame
    using (var frame = reference.ColorFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }
    // Open depth frame
    using (var frame = reference.DepthFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }
    // Open infrared frame
    using (var frame = reference.InfraredFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }
}

We’ll now see how to convert each frame reference into a WPF bitmap and display it to our <Image> element.

Color stream

The raw color images have been increased to 1920×1080 resolution. Kinect only gives us an array of pixel RGB values. We need to convert this array into a proper WPF representation. I will not get into more details about WPF bitmap processing – you can simply copy and paste the following method:

private ImageSource ToBitmap(ColorFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;
    byte[] pixels = new byte[width * height * ((PixelFormats.Bgr32.BitsPerPixel + 7) / 8)];
    if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
    {
        frame.CopyRawFrameDataToArray(pixels);
    }
    else
    {
        frame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Bgra);
    }
    int stride = width * format.BitsPerPixel / 8;
    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

Depth stream

The depth stream provides us with the depth value of every point in the visible area. The depth value is the distance of the sensor. We usually represent the furthest distances using black color and the nearest distances using white color. However, it’s up to you to pick different colors when visualizing depth data. In this conversion, we need two arrays: one array for storing the depth values and another one for storing the corresponding colored values.

private ImageSource ToBitmap(DepthFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;
    ushort minDepth = frame.DepthMinReliableDistance;
    ushort maxDepth = frame.DepthMaxReliableDistance;
    ushort[] depthData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];
    frame.CopyFrameDataToArray(depthData);
    int colorIndex = 0;
    for (int depthIndex = 0; depthIndex < depthData.Length; ++depthIndex)
    {
        ushort depth = depthData[depthIndex];
        byte intensity = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);
        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green
        pixelData[colorIndex++] = intensity; // Red
        ++colorIndex;
    }
    int stride = width * format.BitsPerPixel / 8;
    return BitmapSource.Create(width, height, 96, 96, format, null, pixelData, stride);
}

Infrared stream

The infrared sensor is the ability to view clearly into the dark! Accessing the infrared stream is a new feature, added to Kinect v2. Similarly to displaying depth data, we again need two arrays for holding the infrared and the depth values. Here you go:

private ImageSource ToBitmap(InfraredFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;
    ushort[] infraredData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];
    frame.CopyFrameDataToArray(infraredData);
    int colorIndex = 0;
    for (int infraredIndex = 0; infraredIndex < infraredData.Length; ++infraredIndex)
    {
        ushort ir = infraredData[infraredIndex];
        byte intensity = (byte)(ir >> 8);
        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green   
        pixelData[colorIndex++] = intensity; // Red
        ++colorIndex;
    }
    int stride = width * format.BitsPerPixel / 8;
    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

That’s it! You can now display every bitmap stream! The only thing left to do is call the corresponding method and display the frame. This is how to display the color frame, for example:

// Open color frame
using (var frame = reference.ColorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = ToBitmap(frame);
    }
}

Download the source code and test for yourself. Note that the demo contains additional code for body tracking. We’ll cover body tracking throughout the next blog post.

PS 1: Vitruvius

If you want to automate the above bitmap-conversion processes, consider downloading Vitruvius. Vitruvius is a free & open-source library I built, which provides many utilities for your Kinect applications, including gesture detection, voice recognition and drawing extensions. Give it a try, enjoy and even contribute yourself!

• Download Vitruvius from GitHub

Here is how you can display a color frame using Vitruvius in a single line of C# code:

using (var frame = colorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = frame.ToBitmap();
    }
}

Similarly, you can display a depth or infrared frame with no pain!

PS: Vitruvius

This project is part of the Premium version of Vitruvius (original cost: $299). Vitruvius is a set of powerful Kinect extensions that will help you build stunning Kinect apps in minutes. Vitruvius includes avateering, HD Face, background removal, angle calculations, and more. Check it now.

Want to hire my team for your next cutting-edge app? Drop me an email.