Kinect Skeleton Tracking

Question

I am currently using Microsoft Kinect to measure angles between joints. Most measurements work correctly. Whenever a person sits sideways (on a chair), Kinect will not accurately track the skeleton. To illustrate my problem, I added 3 Kinect depth overview images.

As you can see, 2 out of 3 dimensions work "correctly." Whenever I lift my leg, Kinect correctly tracks skeleton tracking. Does anyone have a solution to this problem, or is it just a Kinect limitation?

Thank.

Update 1: JointTrackingState-Enumeration on these tracked links shown in screenshot 2 is marked as Inferred , however the depth view keeps track of my full body.

Update 2: In screenshot 2, I'm trying to track the front foot highlighted as green. I know that the other leg is not being tracked, but it does not matter, I think.

Update 3: The following code selects the skeleton:

 private Skeleton StickySkeleton(Skeleton[] skeletons) { if (skeletons.Count<Skeleton>(skeleton => skeleton.TrackingId == _trackedSkeletonId) <= 0) { _trackedSkeletonId = -1; _skeleton = null; } if (_trackedSkeletonId == -1) { Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton => skeleton.TrackingState == SkeletonTrackingState.Tracked); if (foundSkeleton != null) { _trackedSkeletonId = foundSkeleton.TrackingId; return foundSkeleton; } } return _skeleton; } 

Whenever the skeleton is tracked, the data will be used to draw the joints and calculate the angle between the joints.

Update 4: I tested sitting on a “block,” much more simplified than a chair. Unfortunately, Kinect still acts the same.

Below are 2 screenshots:

Answer 1

As Renault Dumont said, I would do something with JointTrackingState . Since you use knees, I used the variables leftknee and rightknee , which are done by Joints . Here is the code, you can use the types JointType.FootRight and JointType.FootLeft and Hip , but I will leave it to you.

 static Skeleton first = new Skeleton(); Joint leftknee = first.Joints[JointType.KneeLeft]; Joint rightknee = first.Joints[JointType.KneeRight]; if ((leftknee.TrackingState == JointTrackingState.Inferred || leftknee.TrackingState == JointTrackingState.Tracked) && (rightknee.TrackingState == JointTrackingState.Tracked || rightknee.TrackingState == JointTrackingState.Inferred)) { } 

Or alternately, if you just want one knee to be tracked at a time, or both, you could do this:

 if ((leftknee.TrackingState == JointTrackingState.Inferred || leftknee.TrackingState == JointTrackingState.Tracked) && (rightknee.TrackingState == JointTrackingState.Tracked || rightknee.TrackingState == JointTrackingState.Inferred)) { } else if (leftknee.TrackingState == JointTrackingState.Inferred || leftknee.TrackingState == JointTrackingState.Tracked) { } else if (rightknee.TrackingState == JointTrackingState.Inferred || rightknee.TrackingState == JointTrackingState.Tracked) { } 

FYI Skeleton reason is first is static , because then it can be used to create joints

  static Skeleton first; 

Against

  Skeleton first; 

Change 1

I came to the conclusion that this is a complex process extremely , I think the above method will work, but I just wanted to include what I'm working on, in case you can find some way to make it work. Anyways Here is the code I was working on, which is another class , which is another SkeletalTrackingState , I tried to make Inferred enum in him. But unfortunately enum impossible until inherit . If you find something that works, I will respect you as an excellent programmer for me forever;). Without further ado: .dll I tried to do:

 using System; using System.Collections.Generic; using System.Linq; using System.Text; using Microsoft.Kinect; namespace IsInferred { public abstract class SkeletonInferred : Skeleton { public bool inferred; static Skeleton first1 = new Skeleton(); Joint handright; Joint handleft; Joint footright; Joint footleft; Joint ankleleft; Joint ankleright; Joint elbowleft; Joint elbowright; Joint head; Joint hipcenter; Joint hipleft; Joint hipright; Joint shouldercenter; Joint shoulderleft; Joint shoulderright; Joint kneeleft; Joint kneeright; Joint spine; Joint wristleft; Joint wristright; public SkeletonInferred(bool inferred) { } public enum Inferred { NotTracked = SkeletonTrackingState.NotTracked, PositionOnly = SkeletonTrackingState.PositionOnly, Tracked = SkeletonTrackingState.Tracked, Inferred = 3, } private void IsInferred(object sender, AllFramesReadyEventArgs e) { handright = first1.Joints[JointType.HandRight]; handleft = first1.Joints[JointType.HandLeft]; footright = first1.Joints[JointType.FootRight]; footleft = first1.Joints[JointType.FootLeft]; ankleleft = first1.Joints[JointType.AnkleLeft]; ankleright = first1.Joints[JointType.AnkleRight]; elbowleft = first1.Joints[JointType.ElbowLeft]; elbowright = first1.Joints[JointType.ElbowRight]; head = first1.Joints[JointType.Head]; hipcenter = first1.Joints[JointType.HipCenter]; hipleft = first1.Joints[JointType.HipLeft]; hipright = first1.Joints[JointType.HipRight]; shouldercenter = first1.Joints[JointType.ShoulderCenter]; shoulderleft = first1.Joints[JointType.ShoulderLeft]; shoulderright = first1.Joints[JointType.ShoulderRight]; kneeleft = first1.Joints[JointType.KneeLeft]; kneeright = first1.Joints[JointType.KneeRight]; spine = first1.Joints[JointType.Spine]; wristleft = first1.Joints[JointType.WristLeft]; wristright = first1.Joints[JointType.WristRight]; if (handleft.TrackingState == JointTrackingState.Inferred & handright.TrackingState == JointTrackingState.Inferred & head.TrackingState == JointTrackingState.Inferred & footleft.TrackingState == JointTrackingState.Inferred & footright.TrackingState == JointTrackingState.Inferred & ankleleft.TrackingState == JointTrackingState.Inferred & ankleright.TrackingState == JointTrackingState.Inferred & elbowleft.TrackingState == JointTrackingState.Inferred & elbowright.TrackingState == JointTrackingState.Inferred & hipcenter.TrackingState == JointTrackingState.Inferred & hipleft.TrackingState == JointTrackingState.Inferred & hipright.TrackingState == JointTrackingState.Inferred & shouldercenter.TrackingState == JointTrackingState.Inferred & shoulderleft.TrackingState == JointTrackingState.Inferred & shoulderright.TrackingState == JointTrackingState.Inferred & kneeleft.TrackingState == JointTrackingState.Inferred & kneeright.TrackingState == JointTrackingState.Inferred & spine.TrackingState == JointTrackingState.Inferred & wristleft.TrackingState == JointTrackingState.Inferred & wristright.TrackingState == JointTrackingState.Inferred) { inferred = true; } } } } 

Code in your project (compiler error)

  using IsInferred; static bool Inferred = false; SkeletonInferred inferred = new SkeletonInferred(Inferred); static Skeleton first1 = new Skeleton(); Skeleton foundSkeleton = skeletons.FirstOrDefault<Skeleton>(skeleton => skeleton.TrackingState == SkeletonTrackingState.Inferred); 

Good luck, hope this helps you in the right direction or helps you!

My code

Here is my code you requested. Yes, this is from the Basics of Skeleton Tracking , but this code was here, and I didn't want to start a new project with most of the same thing. Enjoy it!

The code

  // (c) Copyright Microsoft Corporation. // This source is subject to the Microsoft Public License (Ms-PL). // Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details. // All other rights reserved. using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Windows; using System.Windows.Controls; using System.Windows.Data; using System.Windows.Documents; using System.Windows.Input; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Navigation; using System.Windows.Shapes; using Microsoft.Kinect; using Coding4Fun.Kinect.Wpf; namespace SkeletalTracking { /// <summary> /// Interaction logic for MainWindow.xaml /// </summary> public partial class MainWindow : Window { public MainWindow() { InitializeComponent(); } bool closing = false; const int skeletonCount = 6; Skeleton[] allSkeletons = new Skeleton[skeletonCount]; private void Window_Loaded(object sender, RoutedEventArgs e) { kinectSensorChooser1.KinectSensorChanged += new DependencyPropertyChangedEventHandler(kinectSensorChooser1_KinectSensorChanged); } void kinectSensorChooser1_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e) { KinectSensor old = (KinectSensor)e.OldValue; StopKinect(old); KinectSensor sensor = (KinectSensor)e.NewValue; if (sensor == null) { return; } var parameters = new TransformSmoothParameters { Smoothing = 0.3f, Correction = 0.0f, Prediction = 0.0f, JitterRadius = 1.0f, MaxDeviationRadius = 0.5f }; sensor.SkeletonStream.Enable(parameters); //sensor.SkeletonStream.Enable(); sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady); sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30); try { sensor.Start(); } catch (System.IO.IOException) { kinectSensorChooser1.AppConflictOccurred(); } } void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e) { if (closing) { return; } //Get a skeleton Skeleton first = GetFirstSkeleton(e); if (first == null) { return; } //set scaled position //ScalePosition(headImage, first.Joints[JointType.Head]); ScalePosition(leftEllipse, first.Joints[JointType.HandLeft]); ScalePosition(rightEllipse, first.Joints[JointType.HandRight]); ScalePosition(leftknee, first.Joints[JointType.KneeLeft]); ScalePosition(rightknee, first.Joints[JointType.KneeRight]); GetCameraPoint(first, e); } void GetCameraPoint(Skeleton first, AllFramesReadyEventArgs e) { using (DepthImageFrame depth = e.OpenDepthImageFrame()) { if (depth == null || kinectSensorChooser1.Kinect == null) { return; } //Map a joint location to a point on the depth map //head DepthImagePoint headDepthPoint = depth.MapFromSkeletonPoint(first.Joints[JointType.Head].Position); //left hand DepthImagePoint leftDepthPoint = depth.MapFromSkeletonPoint(first.Joints[JointType.HandLeft].Position); //right hand DepthImagePoint rightDepthPoint = depth.MapFromSkeletonPoint(first.Joints[JointType.HandRight].Position); DepthImagePoint rightKnee = depth.MapFromSkeletonPoint(first.Joints[JointType.KneeRight].Position); DepthImagePoint leftKnee = depth.MapFromSkeletonPoint(first.Joints[JointType.KneeLeft].Position); //Map a depth point to a point on the color image //head ColorImagePoint headColorPoint = depth.MapToColorImagePoint(headDepthPoint.X, headDepthPoint.Y, ColorImageFormat.RgbResolution640x480Fps30); //left hand ColorImagePoint leftColorPoint = depth.MapToColorImagePoint(leftDepthPoint.X, leftDepthPoint.Y, ColorImageFormat.RgbResolution640x480Fps30); //right hand ColorImagePoint rightColorPoint = depth.MapToColorImagePoint(rightDepthPoint.X, rightDepthPoint.Y, ColorImageFormat.RgbResolution640x480Fps30); ColorImagePoint leftKneeColorPoint = depth.MapToColorImagePoint(leftKnee.X, leftKnee.Y, ColorImageFormat.RgbResolution640x480Fps30); ColorImagePoint rightKneeColorPoint = depth.MapToColorImagePoint(rightKnee.X, rightKnee.Y, ColorImageFormat.RgbResolution640x480Fps30); //Set location CameraPosition(headImage, headColorPoint); CameraPosition(leftEllipse, leftColorPoint); CameraPosition(rightEllipse, rightColorPoint); Joint LEFTKNEE = first.Joints[JointType.KneeLeft]; Joint RIGHTKNEE = first.Joints[JointType.KneeRight]; if ((LEFTKNEE.TrackingState == JointTrackingState.Inferred || LEFTKNEE.TrackingState == JointTrackingState.Tracked) && (RIGHTKNEE.TrackingState == JointTrackingState.Tracked || RIGHTKNEE.TrackingState == JointTrackingState.Inferred)) { CameraPosition(rightknee, rightKneeColorPoint); CameraPosition(leftknee, leftKneeColorPoint); } else if (LEFTKNEE.TrackingState == JointTrackingState.Inferred || LEFTKNEE.TrackingState == JointTrackingState.Tracked) { CameraPosition(leftknee, leftKneeColorPoint); } else if (RIGHTKNEE.TrackingState == JointTrackingState.Inferred || RIGHTKNEE.TrackingState == JointTrackingState.Tracked) { CameraPosition(rightknee, rightKneeColorPoint); } } } Skeleton GetFirstSkeleton(AllFramesReadyEventArgs e) { using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame()) { if (skeletonFrameData == null) { return null; } skeletonFrameData.CopySkeletonDataTo(allSkeletons); //get the first tracked skeleton Skeleton first = (from s in allSkeletons where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault(); return first; } } private void StopKinect(KinectSensor sensor) { if (sensor != null) { if (sensor.IsRunning) { //stop sensor sensor.Stop(); //stop audio if not null if (sensor.AudioSource != null) { sensor.AudioSource.Stop(); } } } } private void CameraPosition(FrameworkElement element, ColorImagePoint point) { //Divide by 2 for width and height so point is right in the middle // instead of in top/left corner Canvas.SetLeft(element, point.X - element.Width / 2); Canvas.SetTop(element, point.Y - element.Height / 2); } private void ScalePosition(FrameworkElement element, Joint joint) { //convert the value to X/Y //Joint scaledJoint = joint.ScaleTo(1280, 720); //convert & scale (.3 = means 1/3 of joint distance) Joint scaledJoint = joint.ScaleTo(1280, 720, .3f, .3f); Canvas.SetLeft(element, scaledJoint.Position.X); Canvas.SetTop(element, scaledJoint.Position.Y); } private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e) { closing = true; StopKinect(kinectSensorChooser1.Kinect); } private void kinectDepthViewer1_Loaded(object sender, RoutedEventArgs e) { } } } 

Xaml

 <Window x:Class="SkeletalTracking.MainWindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" Title="MainWindow" Height="600" Width="800" Loaded="Window_Loaded" xmlns:my="clr-namespace:Microsoft.Samples.Kinect.WpfViewers;assembly=Microsoft.Samples.Kinect.WpfViewers" Closing="Window_Closing" WindowState="Maximized"> <Canvas Name="MainCanvas"> <my:KinectColorViewer Canvas.Left="0" Canvas.Top="0" Width="640" Height="480" Name="kinectColorViewer1" Kinect="{Binding ElementName=kinectSensorChooser1, Path=Kinect}" /> <Ellipse Canvas.Left="0" Canvas.Top="0" Height="50" Name="leftEllipse" Width="50" Fill="#FF4D298D" Opacity="1" Stroke="White" /> <Ellipse Canvas.Left="100" Canvas.Top="0" Fill="#FF2CACE3" Height="50" Name="rightEllipse" Width="50" Opacity="1" Stroke="White" /> <my:KinectSensorChooser Canvas.Left="250" Canvas.Top="380" Name="kinectSensorChooser1" Width="328" /> <Image Canvas.Left="66" Canvas.Top="90" Height="87" Name="headImage" Stretch="Fill" Width="84" Source="/SkeletalTracking;component/c4f-color.png" /> <Ellipse Canvas.Left="283" Canvas.Top="233" Height="23" Name="leftknee" Stroke="Black" Width="29" /> <Ellipse Canvas.Left="232" Canvas.Top="233" Height="23" Name="rightknee" Stroke="Black" Width="30" /> </Canvas> 

Here's a picture just to show how Kinect can sometimes be:

Hint: notice how only my hand and part of the background are detected

Answer 2

Kinect does not track the skeleton accurately due to limitations in its own SDK. Essentiall every time the device tries to pick up the skeleton, it always assumes that it is in front. If you keep the joints, take them from top to bottom and draw the x and z coordinates, it's pretty easy to see.

Answer 3

I also work with Kinect, you can get the value for the joint angles for the joint angles of your knee, but the problem you seem to run into is related to Kinect itself, as the estimated values ​​are not shown. You better use the entire skeleton body to track your knee, and I hope you get all the values ​​for your movement of each joint.