Avateering Using Kinect v2

OK, you have requested this tutorial over a hundred times. You have been asking for this via email, social media, and private messages. So, today, I’ll show you how to animate a 3D body model using Kinect. To make things easy, we’ll be using Vitruvius throughout this tutorial. Let’s get started.

What is Avateering?

A 3D model can mimic and follow the movements of a person standing in front of a Kinect sensor. A player can control one or multiple 3D models (avatars). Here’s the difference between controlling one or many avatars:

Prerequisites

• Kinect for XBOX v2 sensor with an adapter (or Kinect for Windows v2 sensor)
• Kinect for Windows v2 SDK
• Windows 8.1 or higher
• Visual Studio 2013 or higher
• A dedicated USB 3 port

Source Code

• The complete Unity source code demo is included in Vitruvius.

Step 1: Create a new Unity project

Launch Unity and create a new project, or open an existing Kinect project. We are using Unity because it has awesome 3D support.

Vitruvius includes a handy .unitypackage file. You can double-click or import that package in your project. The package includes Windows.Kinect.dll and LightBuzz.Vitruvius.dll. Those are the required assemblies to use the complete power of Vitruvius.

Step 2: Import the 3D models

Open your scene file in the Editor and import the 3D models you want to animate. Place them whenever you think it’s necessary for your game. I have imported the male and female 3D body models. They are named “male” and “female”, respectively.

This is how my editor looks like:

Step 3: Action!

It’s now time to write your Kinect script. You’ll need to include the following namespaces:

using Windows.Kinect;
using LightBuzz.Vitruvius;

After importing the assemblies, we’ll create the required members to open the sensor, acquire the Body data and update the avatars accordingly.

The separatedPlayers variable indicates whether a single player will control one avatar or multiple avatars.

Members

// Kinect
BodyFrameReader bodyReader;
Body[] users;
// 3D models
Model[] models;
public FBX female;
public FBX male;
// Determines whether
// one player will control all avatars, or
// whether each player will control one avatar
public bool seperatedPlayers;

Start

The Start method initializes Kinect and Vitruvius. It’s important to open the sensor and enable avateering functionality to your 3D models. The code is really simple and self-explanatory:

void Start()
{
    // 1. Initialize Kinect
    sensor = KinectSensor.GetDefault();
    if (sensor != null)
    {
        bodyReader = sensor.BodyFrameSource.OpenReader();
        sensor.Open();
    }
    // 2. Enable Avateering
    Avateering.Enable();
    // 3. Specify the 3D models to animate.
    models = new Model[]
    {
        female,
        male
    };
    // 4. Initialize each 3D model.
    for (int i = 0; i < models.Length; i++)
    {
        models[i].Initialize();
    }
}

Update

Now, it’s time for the cool part! To see your avatars animated, you simply have to loop through the tracked bodies and update the corresponding avatars. Vitruvius does this with just one line of code.

void Update()
{
    if (bodyReader != null)
    {
        using (var frame = bodyReader.AcquireLatestFrame())
        {
            if (frame != null)
            {
                users = frame.Bodies().
                        Where(b => b.IsTracked).ToArray();
                if (seperatedPlayers)
                {
                    // Each user controls a different avatar.
                    for (int index = 0; index < users.Length; index++)
                    {
                       Model model = models[index];
                       Body user = users[index];
                       
                       // Yes, this line does ALL of the hard work for you.
                       Avateering.Update(model, user);
                   }
                } else if (users.Length > 0)
                {
                    // A single user controls all of the avatars.
                    for (int index = 0; index < models.Length; index++)
                    {
                        Model model = models[index];
                        Body user = users[0];
                        // Yes, this line does ALL of the hard work for you.
                        Avateering.Update(model, user);
                    }
                }
            }
        }
    }
}

Dispose

Finally, do not forget to dispose Kinect and clean any resources.

public void Dispose()
{
    // 1. Dispose Kinect
    if (bodyReader != null)
    {
        bodyReader.Dispose();
        bodyReader = null;
    }
    if (sensor != null)
    {
        if (sensor.IsOpen)
        {
            sensor.Close();
        }
        sensor = null;
    }
    // 2. Dispose 3D models
    for (int i = 0; i < models.Length; i++)
    {
        models[i].Dispose();
    }
    // 3. Disable avateering
    Avateering.Disable();
}

This is it! You have just developed an avateering app using Kinect v2!

BONUS – New 3D models

Vitruvius is bundled with nine 3D models you can use in your commercial projects. We are adding new 3D models all the time. Our models are perfectly rigged and work with Kinect just fine. Each model is coming in multiple textures. to modify a texture, simply open the Samples/Models/FBX folder and pick the 3D models and textures of your choice.

The following models are included:

• Female Casual – a beautiful female model with various hair colors and clothes.
• Male Casual – a male model with various clothes.
• Female Athletic – a female model with sports clothing.
• Male Athletic – a male model with sports clothing.
• Kid – a 3D model of a kid.
• Skeleton – a fantastic model of a human skeleton with bones and skull!
• Longsleeve – a shirt with long sleeves and various colors.
• Tshirt – a simple t-shirt model with various colors.
• Pants – a pair of trousers with various styles (jeans, black, formal).

Bodies

Clothes

So, what will you do with Vitruvius? Let me know in the comments below!

Vitruvius

If you enjoyed this post, consider checking Vitruvius. Vitruvius is a set of powerful Kinect extensions that will help you build stunning Kinect apps in minutes. Vitruvius includes avateering, HD Face, background removal, angle calculations, and more. Check it now.