A few days ago, I received an interesting comment from one of the fans of this blog. The fellow-developer was asking me about a good way to draw on top of a virtual canvas using his hands. So, today, I’ll show you how you can easily use a Kinect sensor to draw in the air! This video shows what we are about to build:
Prerequisites
- Kinect for XBOX v2 sensor with an adapter (or Kinect for Windows v2 sensor)
- Kinect for Windows v2 SDK
- Windows 8.1 or higher
- Visual Studio 2013 or higher
- A dedicated USB 3 port
Video & Source Code
As usual, I’m providing you with the complete source code, as well as a demo video.
The XAML User Interface
First things first. Launch Visual Studio and create a new WPF or Windows Store project. Select .NET Framework 4.5 or higher.
If using WPF, add a reference to Microsoft.Kinect.dll.
If using Windows Store (WinRT), add a reference to WindowsPreview.Kinect.dll.
Navigate to the XAML code of your main page. Nothing fancy – we just want to display the feed of the Color camera and the drawing brush. We’ll use an Image component for the RGB stream and a Canvas component for the drawing brush. For proper scaling reasons, I’ve placed both the Image and a Canvas into a Viewbox element. The Viewbox element will automatically scale its contents according to the size of your screen.
<Viewbox>
<Grid Width="1920" Height="1080">
<Image Name="camera" />
<Canvas Name="canvas">
<Polyline Name="trail" Stroke="Red" StrokeThickness="15">
<Polyline.Effect>
<BlurEffect Radius="20" />
</Polyline.Effect>
</Polyline>
</Canvas>
</Grid>
</Viewbox>Inside the Canvas, I placed a Polyline control. A Polyline is a set of points in the 2D space, connected with a line. The Polyline will be updated with new points on each frame.
Hint: some developers are using multiple Ellipse or Line controls instead. Even though this approach would definitely work, the Polyline control is much more efficient when dealing with big volumes of data. We’ll be drawing 15 to 30 ellipses per second, so a single XAML control would work better than multiple XAML controls in the same Canvas.
The C# code
Our user interface is ready to accept the drawing functionality. Let’s get started. Navigate to your .xaml.cs file and add the C# code below.
Step 1 – Declare the required Kinect members
First things first! We need to specify the objects we’ll work with. If you are following this blog, you already know what you need:
// Kinect Sensor reference
private KinectSensor _sensor = null;
// Reads Color frame data
private ColorFrameReader _colorReader = null;
// Reads Body data
private BodyFrameReader _bodyReader = null;
// List of the detected bodies
private IList<Body> _bodies = null;
// Frame width (1920)
private int _width = 0;
// Frame height (1080)
private int _height = 0;
// Color pixel values (bytes)
private byte[] _pixels = null;
// Display bitmap
private WriteableBitmap _bitmap = null;Step 2 – Initialize Kinect
After you have declared the required members, you need to initialize the Kinect sensor and create the event handlers. You can do it in the constructor of your page/window.
if (_sensor != null)
{
_sensor.Open();
_width = _sensor.ColorFrameSource.FrameDescription.Width;
_height = _sensor.ColorFrameSource.FrameDescription.Height;
_colorReader = _sensor.ColorFrameSource.OpenReader();
_colorReader.FrameArrived += ColorReader_FrameArrived;
_bodyReader = _sensor.BodyFrameSource.OpenReader();
_bodyReader.FrameArrived += BodyReader_FrameArrived;
_pixels = new byte[_width * _height * 4];
_bitmap = new WriteableBitmap(_width, _height, 96.0, 96.0, PixelFormats.Bgra32, null);
_bodies = new Body[_sensor.BodyFrameSource.BodyCount];
camera.Source = _bitmap;
}Step 3 – Display the RGB Color stream
The event named “ColorReader_FrameArrived” will be called whenever Kinect has a new RGB frame. All we need to do is grab the frame and transform it into a WriteableBitmap. The following source code acquires the raw RGB values, copies the values into our byte array, and, finally, creates the displayable bitmap:
private void ColorReader_FrameArrived(object sender, ColorFrameArrivedEventArgs e)
{
using (var frame = e.FrameReference.AcquireFrame())
{
if (frame != null)
{
frame.CopyConvertedFrameDataToArray(_pixels, ColorImageFormat.Bgra);
_bitmap.Lock();
Marshal.Copy(_pixels, 0, _bitmap.BackBuffer, _pixels.Length);
_bitmap.AddDirtyRect(new Int32Rect(0, 0, _width, _height));
_bitmap.Unlock();
}
}
}Step 4 – Detect the hand
Now, let’s move to our BodyReader_FrameArrived event handler. This where we’ll detect the active body and its joints. We only need one joint: the right (or left) hand. We detect the desired hand by using the JointType enumeration.
Hint: for educational purposes, I am only showing you how to get the hand joint. In a real-world project, you could check a number of additional parameters, such as the distance between the hand and the body, or the distance between the body and the sensor. You could even have custom gestures to start and stop the painting process.
private void BodyReader_FrameArrived(object sender, BodyFrameArrivedEventArgs e)
{
using (var frame = e.FrameReference.AcquireFrame())
{
if (frame != null)
{
frame.GetAndRefreshBodyData(_bodies);
Body body = _bodies.Where(b => b.IsTracked).FirstOrDefault();
if (body != null)
{
Joint handRight = body.Joints[JointType.HandRight];
}
}
}
}Step 5 – Coordinate Mapping
As you know, Kinect provides us with the position of the hand in the 3D space (X, Y, and Z values). To detect where the hand is pointing in the 2D space, we’ll need to convert the 3D coordinates to 2D coordinates. The 3D coordinates are measured in meters. The 2D coordinates are measured in, well, pixels. So, how could we convert meters to pixels? The SDK includes a powerful utility, called CoordinateMapper. Using CoordinateMapper, we can find the position of the hand in the 2D space!
You can read more about coordinate mapping in this blog post. Alternatively, you can use Vitruvius and have automatic coordinate mapping.
The CoordinateMapper will convert the position of the hand to a ColorSpacePoint with X and Y values. The X and Y values of the ColorSpacePoint are limited to our 1920×1080 canvas.
CameraSpacePoint handRightPosition = handRight.Position;
ColorSpacePoint handRightPoint = _sensor.CoordinateMapper.MapCameraPointToColorSpace(handRightPosition);
float x = handRightPoint.X;
float y = handRightPoint.Y;Step 6 – Draw!
Finally, we’ll update the point collection of our Polyline component by adding the new color point.
if (!float.IsInfinity(x) && ! float.IsInfinity(y))
{
// DRAW!
trail.Points.Add(new Point { X = x, Y = y });
}Compile and run the code. You’ll notice that I have also applied a blur effect to make the trail smoother.
Conclusion
So, this is it, folks! Hope you enjoyed this tutorial. Feel free to extend the source code, add your own effects, constraints, and functionality, and use it in your own projects.
Download the source code on GitHub.
‘Till the next time, keep Kinecting!
PS: Vitruvius
If you enjoyed this article, consider checking Vitruvius. Vitruvius is a set of powerful Kinect extensions that will help you build stunning Kinect apps in minutes. Vitruvius includes avateering, HD Face, background removal, angle calculations, and more. Check it now.
Hi Vangos!
I just tried your app and i must say its amazing!
I tried a way of implementing a pen up function as so
if (!float.IsInfinity(x) && ! float.IsInfinity(y))
{
if (handRight.Position.Z > 0.60f && handRight.Position.Z < 1.11f)
{
// DRAW!
trail.Points.Add(new Point { X = x, Y = y });
Canvas.SetLeft(brush, x – brush.Width / 2.0);
Canvas.SetTop(brush, y – brush.Height);
}
}
so when i would be outside the canvas, it would stop writing, but when i re enter the canvas it traces back from the last known point and joins my letters and making it a continuous trail.
is there a way to start from a new point every time i re enter the canvas so i can write words on it?
cheers mate!
Hi Abdullah,
That’s a great addition and a great question, too. There a couple ways to “change line”:
Solution #1 – Use a different Canvas. Using a different canvas, you can place it wherever you like. You could create any new canvases dynamically with C#.
Solution #2 – Use different Polyline elements. When the hand is re-entering into drawing distance, you can create a new Polyline element. Alternatively, you can set the 0,0 point to where the hand is.
Finally, you could specify an “active threshold” to separate letters. That would be tricky, since it would require you to measure the frames of inactivity that would mean that someone finished writing one letter and moved to another.
Thanks a lot for you reply!
Solution 2 is more plausible for my project, I am mainly trying to focus on the latter part =”setting the 0,0 point to where the hand is ”
So each time the hand is re enters the drawing distance the line begins from the position of the hand.
Unfortunately, I have tried several solutions but have failed, I have set reset the x and y values to 0 on leaving the canvas but everytime it traces for the last known position before leaving canvas 🙁
This is what i tried
if (_hasBeenOutside == true)
{
//reset x and y to current hand position
x = handRightPoint.X;
y = handRightPoint.Y;
trail.Points.Add(new Point { X = x, Y = y });
Canvas.SetLeft(brush, x – brush.Width / 2.0);
Canvas.SetTop(brush, y – brush.Height);
_hasBeenOutside = !_hasBeenOutside;
}
Hi Abdullah. Why don’t create a new Polyline every time the hand re-enters the scene?
Polyline polyline = new Polyline();
polyline.Points.Add(yourPoint);
canvas.Children.Add(polyline);
Apologies for returning to you yet again, this is my current updated code as per ur suggestion (which makes absolute sense to me)
if (handRight.Position.Z > 0.60f && handRight.Position.Z < 1.11f)
{
// DRAW!
//trail.Points.Add(new Point { X = x, Y = y });
Polyline polyline = new Polyline();
//define attributes
polyline.Stroke = new SolidColorBrush(Colors.Black);
polyline.StrokeThickness = 20;
polyline.Points.Add(new Point { X = x, Y = y });
canvas.Children.Add(polyline);
//Brush
Canvas.SetLeft(brush, x – brush.Width / 2.0);
Canvas.SetTop(brush, y – brush.Height);
}
I have left the xaml unchanged as i am defining polyline attributes everytime its created, the brush moves with the position but i get no trace. I apologise again for disturbing you with these questions. I have achieved this functionality with plotting ellipse objects, i realize this polylines are a better and smoother way but unfortunately I am unable to implement it.
Apologies for the inconvenience.
Hi Vangos,
based on Abdullah’s question, I want to draw Polylines with both hands. Furthermore, I need to draw lines only when HandState = closed. Is it possible using a Polyline? I’ve followed the instructions you gave, but I’ve not reached what I expected.
Thanks in advance!
Hi Claire. You can add a second polyline into your canvas and update that polyline accordingly, e.g. polyline2.Points.Add(somePoint).
Regarding your second question, you can simply check the hand state and update the polyline only when the state is Closed:
if (body.HandRightState == HandState.Closed)
{
polyline.Points.Add(point);
}
Thanks for the answer.
As you’ve suggested, instead of creating a second polyline, I’ve created a List of polylines and I’ve subsequently modified your code. Probably it isn’t the best solution, but for me it works. I write the solution, maybe it helps also other users:
// Right hand
Joint handRight = body.Joints[JointType.HandRight];
if (handRight.TrackingState != TrackingState.NotTracked)
{
CameraSpacePoint handRightPosition = handRight.Position;
ColorSpacePoint handRightPoint = _sensor.CoordinateMapper.MapCameraPointToColorSpace(handRightPosition);
float x = handRightPoint.X;
float y = handRightPoint.Y;
if (body.HandRightState == HandState.Closed)
{
isClose = true;
if (!float.IsInfinity(x) && !float.IsInfinity(y))
{
// DRAW!
trailR[i].Points.Add(new Point { X = x, Y = y });
Canvas.SetLeft(brush, x – brush.Width / 2.0);
Canvas.SetTop(brush, y – brush.Height);
}
}
else
{
if(isClose)
{
i++;
trailR.Add(new Polyline());
trailR[i].Stroke = Brushes.Red;
trailR[i].StrokeThickness = 15;
trailR[i].Effect = plBlur;
canvas.Children.Add(trailR[i]);
isClose = false;
}
}
}
Any advice is welcome!
Thanks for sharing with the community!
Just one thought: you could use multiple Ellipses instead of multiple Polylines. I think it would be a little more efficient.
Thanks for the suggestion, I agree with you, but when I’ve implemented a List of ellipses instead of polilynes, I’ve seen a progressive reduction of Kinect’s frame rate. Probably because of ellipses’s index that increases more rapidly than that of polilynes. At this point I’ve decided to mantain polilynes.
Great! Thanks for sharing!
Dear Claire,
what did you change in MainWindow.xaml?
How did you declare list of polylines?
Hi Abdullah,
why this doesn’t draw on my canvas? I’ve paste your code, didn’t change anything in xaml, but when I run it, everything seems works, but I don’t see the drawing… ((
Hi Vangos,
I like this Drawing tool and would like to implement in my project. But my project does not need to track the body. It is just tracking the Multiple hands. So is it possible to integrate it in that kind of project.
Hi Sanket. You need to track a body to track its hands using Kinect. You do not need to display the body, but you definitely need to track it.
ohh ok..Actually i am not tracking the body instead i just tracked the multiple hands using EMGUCV also my kinect is put downwards to the TABLE surface so it cant see or detect the body. Now i would like to make a drawing tool. 🙂
Just took a quick look at the discussion thread here, an idea is to make a bigger canvas that extends outside its container and is centered in it. You could host it in a Viewbox to achieve that for example, setting it to not stretch the content (just center it). I think it also does clipping on WPF
That’s true. Thanks for your feedback!
Additionally, you could check the hand states (Open, Closed, Lasso) to determine when someone will start/stop drawing.
Hello,
I’ve tried to implement a variation of your code where whatever is drawn will be erased if the user lifts his left hand above the right hand and will start drawing afresh once the left hand is lowered again. The following code lets me erase whatever is drawn but once I lower my left hand My right hand no longer draws anything. Does anyone have any idea where I’m going wrong?
CS Code:
void paintBrush(Joint j1, Joint j2)
{
DepthImagePoint j1P = this.myKinect.CoordinateMapper.MapSkeletonPointToDepthPoint(j1.Position, DepthImageFormat.Resolution640x480Fps30);
DepthImagePoint j2P = this.myKinect.CoordinateMapper.MapSkeletonPointToDepthPoint(j2.Position, DepthImageFormat.Resolution640x480Fps30);
float x = j2P.X;
float y = j2P.Y;
if (!float.IsInfinity(x) && !float.IsInfinity(y))
{
// DRAW!
trail.Points.Add(new System.Windows.Point { X = x, Y = y });
}
if (j1P.Y < j2P.Y) skeletonCursorCanvas.Children.Clear();
}
XAML:
Try the following:
if (j1.Position.Y < j2.Position.Y) skeletonCursorCanvas.Children.Clear();
Thank you for your reply. Unfortunately that didn’t work either. Perhaps it has something to do with my XAML code?
Window x:Class=”KinectTwoPlayer.MainWindow”
xmlns=”http://schemas.microsoft.com/winfx/2006/xaml/presentation”
xmlns:x=”http://schemas.microsoft.com/winfx/2006/xaml”
Title=”Two Player” SizeToContent=”WidthAndHeight” Loaded=”Window_Loaded” Closing=”Window_Closing”
StackPanel
TextBlock Name=”KinectStatusTextBlock” Text=”Status” FontSize=”20″ TextAlignment=”Center”/
Grid
Image Name=”kinectDepth” Height=”480″ Width=”640″ /
Canvas Height=” 480″ Width=”640″ HorizontalAlignment=”Center”
Canvas Name=”skeletonCanvas” Height=”480″ Width=”640″ HorizontalAlignment=”Center” Background=”Transparent” /
Canvas Name=”skeletonCursorCanvas” Height=”480″ Width=”640″ HorizontalAlignment=”Center”
Polyline Name=”trail” Stroke=”Red” StrokeThickness=”15″
Polyline.Effect
BlurEffect Radius=”20″ /
/Polyline.Effect
/Polyline
/Canvas
/Canvas
/Grid
TextBlock Name=”FingerTipBlock” Text=”InOrOut” FontSize=”20″ TextAlignment=”Center”/
/StackPanel
/Window
Sir kindly give me how to show coloured image in window store app and body tracking using color image in window store app