Apple has officially entered the AI-powered body-tracking industry! With its new pose estimation capabilities, ARKit 3.5 is a Kinect alternative for iOS mobile devices. People occlusion and human pose estimation are now core parts of the latest ARKit 3.5 framework. More importantly, the new iPad Pro 2020 (as well as the upcoming iPhone PRo) is now equipped with a LiDAR depth camera.
Why is that important? It’s crucial because depth-sensing allows the camera to better understand its surroundings and estimate their distances. In terms of body-tracking, LiDAR camera allows developers to capture human motion in 3D.
So, without further ado, I am going to show you how to develop body-tracking apps for iPad (and future iPhone) devices in Unity3D!
More specifically, we’ll detect and visualize the following human body joints:
Prerequisites
Since we are developing for the Apple ecosystem, we need the proper Mac computer to develop our applications and the proper iOS device to run them.
Hardware
In terms of hardware, you need a MacOS computer that is compatible with MacOS Catalina. Also, body tracking applications need the powerful Apple A12 Bionic processors to run properly. The following Mac computers and iOS devices would be eligible:
Computers | Mobile devices |
12-inch MacBook | iPad Pro 2020 with LiDAR |
MacBook Air, 2012 and later | iPhone 2020 with LiDAR (expected later this year) |
MacBook Pro, 2012 and later | |
Mac mini, 2012 and later | |
iMac, 2012 and later | |
iMac Pro | |
Mac Pro, 2013 and later |
For this guide, I am using a MacBook Pro computer with an 11” iPad Pro 2020.
Software
To run the demos, you need to install the following software on your Mac computer:
- Unity3D 2020.2.07a with iOS build target
- MacOS Catalina 10.15
- XCode 11.4
Your iOS device should be updated to iOS 13.4 or iPadOS 13.4.
If you are in a hurry, download the complete source code on GitHub. Keep reading to understand how to create your own body-tracking apps!
Did you know?
For the past 10 years, I have been helping Fortune-500 companies and innovative startups create amazing body-tracking applications and games. If you are looking for a reliable contractor to develop your next Motion Tracking project, get in touch with me.
Body Tracking step-by-step
Enough said… Let’s dive right into the ARKit magic. On your computer, launch Unity3D 2020.2 and create a new project.
Step 1 – Set up the main scene
Unity3D will start with an empty scene. Before adding any visual objects or writing any code, we first need to import the proper dependencies. The skeleton-tracking functionality is part of the ARKit toolkit. As a result, we need to import the ARKit XR 4.0.0 and ARFoundation 4.0.0 dependency packages.
Troubleshooting
At the time of this writing, you should use the preview versions 4.0.0-preview.1. If you can’t see those packages in Unity 2020, do the following:
- Open the Package Manager view (Window Package → Manager).
- Click Advanced.
- Check “Show preview packages”.
The AR Session Origin
Now, create a new scene and add an AR Session and an AR Session Origin object. These objects are controlling the iOS camera while providing a ton of ARKit goodies.
Also, add an empty game object, name it e.g. Human Body Tracking, and attach a new C# script (HumanBodyTracking.cs).
The structure of the scene should look like this:
Step 2 – Set up the Skeleton
Since the visual elements are in place, we can now start adding some interactivity. Open the HumanBodyTracking.cs script and add a reference to the ARHumanBodyManager class. The ARHumanBodyManager is the primary script that analyzes the camera data to detect human bodies.
[SerializeField] private ARHumanBodyManager humanBodyManager;
To display the joints, we’ll use some simple Unity3D spheres. Each sphere will correspond to a specific joint type. Add a C# Dictionary class to update the joint data, frame-by-frame.
private Dictionary<JointIndices3D, Transform> bodyJoints;
Finally, add references to the user interface elements of the skeleton. We’ll need a sphere object for the joints and a line object for the bones.
[SerializeField] private GameObject jointPrefab; [SerializeField] private GameObject lineRendererPrefab; private LineRenderer[] lineRenderers; private Transform[][] lineRendererTransforms;
You can find the complete C# code in the HumanBodyTracking.cs class on GitHub.
Step 3 – Detect the Tracked Bodies
This is the most important part of the tutorial! ARKit has made body-tracking incredibly easy and accessible. All you need to do is use the ARHumanBodyManager object and subscribe to the humanBodiesChanged event.
private void OnEnable() { humanBodyManager.humanBodiesChanged += OnHumanBodiesChanged; } private void OnDisable() { humanBodyManager.humanBodiesChanged -= OnHumanBodiesChanged; }
The event handler is where the magic happens. The information about the tracked bodies is part of the event arguments. This is how to acquire the bodies:
private void OnHumanBodiesChanged(ARHumanBodiesChangedEventArgs eventArgs) { foreach (ARHumanBody humanBody in eventArgs.added) { UpdateBody(humanBody); } foreach (ARHumanBody humanBody in eventArgs.updated) { UpdateBody(humanBody); } }
Piece of cake, right? So, let’s bring everything together and display the skeleton in the Unity user interface we created in the previous steps.
Note: as of the time of this writing, the ARKit only supports one tracked body.
Step 4 – Display the Skeleton
The following lines of code update the positions of the joints in the camera space. The spheres and lines are overlayed on top of the iOS camera feed.
private void UpdateBody(ARHumanBody arBody) { if (jointPrefab == null) return; if (arBody == null) return; if (arBody.transform == null) return; InitializeObjects(arBody.transform); NativeArray<XRHumanBodyJoint> joints = arBody.joints; foreach (KeyValuePair<JointIndices3D, Transform> item in bodyJoints) { UpdateJointTransform(item.Value, joints[(int)item.Key]); } for (int i = 0; i < lineRenderers.Length; i++) { lineRenderers[i].SetPositions(lineRendererTransforms[i]); } }
Apple supports 92 joint types (indices). However, not all of these joint types are actually tracked! Most of them are inferred, based on the positions of their neighboring joints. For your convenience, I have selected 14 joint types, so I can have a fair comparison with the Kinect camera.
This is how to connect the proper joints and form the human bones:
private void InitializeObjects(Transform arBodyT) { if (bodyJoints == null) { bodyJoints = new Dictionary<JointIndices3D, Transform> { { JointIndices3D.head_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.neck_1_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_arm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_arm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_forearm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_forearm_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_hand_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_hand_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_upLeg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_upLeg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_leg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_leg_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.left_foot_joint, Instantiate(jointPrefab, arBodyT).transform }, { JointIndices3D.right_foot_joint, Instantiate(jointPrefab, arBodyT).transform } }; lineRenderers = new LineRenderer[] { Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // head neck Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // upper Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // lower Instantiate(lineRendererPrefab).GetComponent<LineRenderer>(), // right Instantiate(lineRendererPrefab).GetComponent<LineRenderer>() // left }; lineRendererTransforms = new Transform[][] { new Transform[] { bodyJoints[JointIndices3D.head_joint], bodyJoints[JointIndices3D.neck_1_joint] }, new Transform[] { bodyJoints[JointIndices3D.right_hand_joint], bodyJoints[JointIndices3D.right_forearm_joint], bodyJoints[JointIndices3D.right_arm_joint], bodyJoints[JointIndices3D.left_arm_joint], bodyJoints[JointIndices3D.left_forearm_joint], bodyJoints[JointIndices3D.left_hand_joint]}, new Transform[] { bodyJoints[JointIndices3D.right_foot_joint], bodyJoints[JointIndices3D.right_leg_joint], bodyJoints[JointIndices3D.right_upLeg_joint], bodyJoints[JointIndices3D.left_upLeg_joint], bodyJoints[JointIndices3D.left_leg_joint], bodyJoints[JointIndices3D.left_foot_joint] }, new Transform[] { bodyJoints[JointIndices3D.right_arm_joint], bodyJoints[JointIndices3D.right_upLeg_joint] }, new Transform[] { bodyJoints[JointIndices3D.left_arm_joint], bodyJoints[JointIndices3D.left_upLeg_joint] } }; for (int i = 0; i < lineRenderers.Length; i++) { lineRenderers[i].positionCount = lineRendererTransforms[i].Length; } } }
ARKit is giving us the position and rotation of the joints in the 3D space! This is how to update the scale, position, and rotation of the sphere in the 2D screen space:
private void UpdateJointTransform(Transform jointT, XRHumanBodyJoint bodyJoint) { jointT.localScale = bodyJoint.anchorScale; jointT.localRotation = bodyJoint.anchorPose.rotation; jointT.localPosition = bodyJoint.anchorPose.position; }
This is it! Let’s build and run our project on an actual iOS device!
Step 5 – Build and Deploy
Finally, we need to build and run the project on an actual device. Given that ARKit is part of iOS and iPadOS, we cannot test our code on MacOS (I would love to see a simulator, though).
In Unity, select File → Build Settings. Click the iOS build target and hit the Build button. You’ll need to specify a location to store the generated project. Wait patiently until Unity finishes with the build process.
Unity will create an XCode project (.xcodeproj). Open the project with XCode 11.4. If you use a previous version of XCode, you’ll get an error and your project will not run properly.
When the project is launched, provide your iOS Development credentials, connect your iOS 13.4 device, and click the Run button. This way, the project will be deployed to the device.
When finished, you should point the camera to a person and you’ll start seeing the 3D overlay on top of the tracked body!
By the way…
For the past 10 years, I have been helping Fortune-500 companies and innovative startups create amazing body-tracking applications and games. If you are looking for a reliable contractor to develop your next Motion Tracking project, get in touch with me.
If you liked this article, remember to share it on social media, so you can help other developers, too! Also, let me know your thoughts in the comments below. ‘Til the next time… keep coding!
Thank you! Is there a way to extract the data you gather from this app to a file for data processing purposes?
Hi Dylan. You’d need to serialize the Vector coordinates to CSV and store that file on your iOS device or a remote server.
Is there any way to find the depth of each body part from the camera… I need to calculate the exact thickness of each body part..
Sure, just use the Z coordinate of the pelvic joint. That would be the distance between the person and the camera.