Tracking and comparing Skeletons with pre-defined Skeleton

Question

I'm currently working on a project for school that requires me to create a certain piece of software with the Kinect hardware. The software is closely related to an aerobics/exercise "game".

However, being that I have never worked for the Kinect, I'm somewhat clueless as to where to start. I have downloaded the SDK and the Toolkit browser, as well as some (unofficial?) Toolkit.

I have some experience in Java programming, but no knowledge whatsoever how to work with C/C++/C# nor Visual Studio.

I'm basically looking for tutorials that could help me get more knowledgeable with the Kinect and how programming for it works. I tried looking for a few but they were either outdated or really confusing that I couldn't keep up with them.

My main goal in the project is to figure out how I can see when a live skeleton holds both his arms above his head and when they're next to his body (A jumping jack exercise).

Can anyone help me along the right direction with a few links or examples?

Answer 1

C# is not too conceptually different from Java. Microsoft even has online help on the subject: The C# Programming Language for Java Developers. Though I'd suggest finding a good book that doesn't wrap C# in the context of another language -- learn C#, not "C# as it relates to Java."

Learning Kinect development should be done through the Toolkit examples. There is really no better way. BEWARE online tutorials! Many were written for older versions of the SDK that are no longer compatible. Anything written for SDK < 1.5 will just not work, without effort to port it.

Detecting a jumping-jack could be done through gesture recognition. There are a few libraries out there that provide this for the official Microsoft Kinect SDK -- the two I generally point to are Kinect Toolbox and the Fizbin Gesture Library. Both provide gesture recognition, just using different methods.

In the case of the Fizbin Gesture Library you declare a series of classes that define how a gesture is constructed. For example, here is how the library define swiping the left hand towards the right of the body:

namespace Fizbin.Kinect.Gestures.Segments
{
    /// <summary>
    /// The first part of the swipe right gesture
    /// </summary>
    public class SwipeRightSegment1 : IRelativeGestureSegment
    {
        /// <summary>
        /// Checks the gesture.
        /// </summary>
        /// <param name="skeleton">The skeleton.</param>
        /// <returns>GesturePartResult based on if the gesture part has been completed</returns>
        public GesturePartResult CheckGesture(Skeleton skeleton)
        {
            // left hand in front of left Shoulder
            if (skeleton.Joints[JointType.HandLeft].Position.Z < skeleton.Joints[JointType.ElbowLeft].Position.Z && skeleton.Joints[JointType.HandRight].Position.Y < skeleton.Joints[JointType.HipCenter].Position.Y)
            {
                // left hand below shoulder height but above hip height
                if (skeleton.Joints[JointType.HandLeft].Position.Y < skeleton.Joints[JointType.Head].Position.Y && skeleton.Joints[JointType.HandLeft].Position.Y > skeleton.Joints[JointType.HipCenter].Position.Y)
                {
                    // left hand left of left Shoulder
                    if (skeleton.Joints[JointType.HandLeft].Position.X < skeleton.Joints[JointType.ShoulderLeft].Position.X)
                    {
                        return GesturePartResult.Succeed;
                    }

                    return GesturePartResult.Pausing;
                }

                return GesturePartResult.Fail;
            }

            return GesturePartResult.Fail;
        }
    }

    /// <summary>
    /// The second part of the swipe right gesture
    /// </summary>
    public class SwipeRightSegment2 : IRelativeGestureSegment
    {
        /// <summary>
        /// Checks the gesture.
        /// </summary>
        /// <param name="skeleton">The skeleton.</param>
        /// <returns>GesturePartResult based on if the gesture part has been completed</returns>
        public GesturePartResult CheckGesture(Skeleton skeleton)
        {
            // left hand in front of left Shoulder
            if (skeleton.Joints[JointType.HandLeft].Position.Z < skeleton.Joints[JointType.ElbowLeft].Position.Z && skeleton.Joints[JointType.HandRight].Position.Y < skeleton.Joints[JointType.HipCenter].Position.Y)
            {
                // left hand below shoulder height but above hip height
                if (skeleton.Joints[JointType.HandLeft].Position.Y < skeleton.Joints[JointType.Head].Position.Y && skeleton.Joints[JointType.HandLeft].Position.Y > skeleton.Joints[JointType.HipCenter].Position.Y)
                {
                    // left hand left of left Shoulder
                    if (skeleton.Joints[JointType.HandLeft].Position.X < skeleton.Joints[JointType.ShoulderRight].Position.X && skeleton.Joints[JointType.HandLeft].Position.X > skeleton.Joints[JointType.ShoulderLeft].Position.X)
                    {
                        return GesturePartResult.Succeed;
                    }

                    return GesturePartResult.Pausing;
                }

                return GesturePartResult.Fail;
            }

            return GesturePartResult.Fail;
        }
    }

    /// <summary>
    /// The third part of the swipe right gesture
    /// </summary>
    public class SwipeRightSegment3 : IRelativeGestureSegment
    {
        /// <summary>
        /// Checks the gesture.
        /// </summary>
        /// <param name="skeleton">The skeleton.</param>
        /// <returns>GesturePartResult based on if the gesture part has been completed</returns>
        public GesturePartResult CheckGesture(Skeleton skeleton)
        {
            // //left hand in front of left Shoulder
            if (skeleton.Joints[JointType.HandLeft].Position.Z < skeleton.Joints[JointType.ElbowLeft].Position.Z && skeleton.Joints[JointType.HandRight].Position.Y < skeleton.Joints[JointType.HipCenter].Position.Y)
            {
                // left hand below shoulder height but above hip height
                if (skeleton.Joints[JointType.HandLeft].Position.Y < skeleton.Joints[JointType.Head].Position.Y && skeleton.Joints[JointType.HandLeft].Position.Y > skeleton.Joints[JointType.HipCenter].Position.Y)
                {
                    // left hand left of left Shoulder
                    if (skeleton.Joints[JointType.HandLeft].Position.X > skeleton.Joints[JointType.ShoulderRight].Position.X)
                    {
                        return GesturePartResult.Succeed;
                    }

                    return GesturePartResult.Pausing;
                }

                return GesturePartResult.Fail;
            }

            return GesturePartResult.Fail;
        }
    }
}

You can follow the comments in the code to see how the hand is moving across the body.

In the case of your jumping-jack, you can define the hands relative to a series of joints on the way up and then on the way down. If I were to quickly spit out a series of checks they would be:

1. Left/Right Hand below Hip Center
2. Left/Right Hand above Hip Center & Left/Right Hand outside Left/Right Elbow
3. Left/Right Hand above Spine & Left/Right Hand outside Left/Right Elbow
4. Left/Right Hand above Left/Right Shoulder & Left/Right Hand outside Left/Right Elbow
5. Left/Right Hand above Head

... if all those match, you have half a jumping-jack. Do all the checks in reverse under the same Gesture and you have a full upper-body jumping-jack. You can add in leg positions to get a full-body jumping-jack.

The Kinect Toolbox could also be used to define the gesture. I've just not used it personally, so I can't speak to the steps involved.