Kinect and HTML5 using WebSockets and Canvas

UPDATE (Dec 10, 2013): Want to learn more about WebSockets? Consider my new book, Getting started with HTML5 WebSocket Programming.

Kinect defined Natural User Interaction. HTML5 redefined the Web. Currently, there are various tutorials describing how to interact with a Kinect sensor using Windows Forms or WPF for the user interface. But what about using a web interface for handling Kinect data? Trying to combine those two hot, cutting-edge technologies, I came up with a pretty and open-source solution, which I am going to describe in this blog post.

I am going to use the official Kinect SDK for my demo, but the same principles apply to OpenNI SDK, too.

• Grab the source code from Github

Prerequisites

• Kinect for Windows or Kinect for XBOX sensor
• Microsoft Kinect SDK (or OpenNI alternative)

Results

The project consists of two sub-projects: A server-side application which uses Kinect SDK and a client-side web page displaying the skeleton joints on an HTML5 canvas.

Here is the end result (current shot was taken from Google Chrome browser):

Tutorial

Here is a way to achieve the above functionality, step by step:

Step 1: Server application

The server application’s job is straightforward: Detect the users’ joints, pack the data and send them to the clients using web sockets.

In order to detect the joints’ coordinates, we need to add a reference to our preferred Kinect SDK and handle the skeleton events. I recommend Microsoft SDK over OpenNI because it’s far less complicated:

static void Nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    List users = new List();
    foreach (var user in e.SkeletonFrame.Skeletons)
    {
        if (user.TrackingState == SkeletonTrackingState.Tracked)
        {
            users.Add(user);
        }
    }
    if (users.Count > 0)
    {
        string json = users.Serialize();
        foreach (var socket in _sockets)
        {
            socket.Send(json);
        }
    }
}

In order to send the data to the subscribed clients, we need to “pack” the joints’ coordinates in a way that the clients will be able to understand and process. I decided to encode the coordinates in JSON format, as JSON is a lightweight and easy-to-understand way of transmitting data through the web. You can find the JSON-encoder class in the source code files.

Considering the data transmission, I highly recommend the use of Fleck. Fleck, based on Nugget, is a C# web socket library that does what it says with minimum configuration effort: It broadcasts the desired data to the subscribed clients using lightweight web sockets. Here is how you can initialize a web socket server:

var server = new WebSocketServer("ws://localhost:8181");
server.Start(socket =>
{
    socket.OnOpen = () =>
    {
        Console.WriteLine("Connected to " + socket.ConnectionInfo.ClientIpAddress);
        _clients.Add(socket);
    };
    socket.OnClose = () =>
    {
        Console.WriteLine("Disconnected from " + socket.ConnectionInfo.ClientIpAddress);
        _clients.Remove(socket);
    };
    socket.OnMessage = message =>
    {
        Console.WriteLine(message);
    };
});

After collecting, packing and transmitting the data, clients can now consume and process them accordingly.

Step 2: Client application

Time for HTML5 bits! We have the server set up, so let’s add the web page clients. HTML5 spec recommends web socket support, currently implemented by Internet Explorer 10, Google Chrome and Mozilla Firefox. Web sockets are great for direct message communication between server and clients.

All we need is some JavaScript for receiving the server data and a canvas for drawing the points. Here is the JavaScript event handler for getting the data and drawing the joints to a Canvas element:

socket.onmessage = function (event) {
    if (typeof event.data === "string") {
        status.innerHTML = "Kinect skeleton received.";
        // 1. Get the data in JSON format.
        var jsonObject = JSON.parse(event.data);
        context.clearRect(0, 0, canvas.width, canvas.height);
        // 2. Display the skeleton joints.
        for (var i = 0; i < jsonObject.skeletons.length; i++) {
            for (var j = 0; j < jsonObject.skeletons[i].joints.length; j++) {
                var joint = jsonObject.skeletons[i].joints[j];
                // Draw!!!
                context.fillStyle = "#FF0000";
                context.beginPath();
                context.arc(joint.x, joint.y, 10, 0, Math.PI * 2, true);
                context.closePath();
                context.fill();
            }
        }
        // Inform the server about the update.
        socket.send("Skeleton updated on: " + (new Date()).toTimeString());
    }
};

Step 3: Mixin it up!

Now run the server application and then open the HTML file in Internet Explorer 10 or Google Chrome. Stand in front of a Kinect sensor and may the force be with you.

FAQ

• Do I really need a server application? Yes. You can’t run the Kinect SDK directly from your browser. JavaScript cannot access your hardware. The server app is nothing but a Visual Studio Console project!
• Do I need to pack my data to JSON? No. You can use any format you’d like. JSON is a common format for transmitting data, especially when file size matters. You could use XML or any other custom format, but JSON is lightweight and needs no custom parser. After all, JSON is a subset of JavaScript, so JavaScript can handle it directly.
• Why not using WCF instead of web sockets? Because it’s tougher. Using web sockets, messages can be sent after a server event occurs. And it’s a lot easier!

Downloads

You will definitely need to download source code and binaries in order to develop your own HTML5 Kinect applications. Enjoy!