Background removal using Kinect: noise suppression around body shape

Question

The objective is to display the person on a different background (aka background removal).

I'm using the Kinect with Microsoft's Beta Kinect SDK to do so. With help of the depth, the background is filtered and we get only the image of the person.

This is pretty simple to do, and we can find the code that does that everywhere on the Internet. However, the depth signal is noisy, and we get pixels which do not belong to the person that are displayed.

I applied an edge detector to see if it was useful, and I currently get this:

Here's another without edge detection:

My question is: Which way can I get rid of these noisy white pixels around the person?

I tried morphological operations, but some parts of the body are erased and still leave white pixels behind.

The algorithm doesn't need to be real-time, I can just apply it when I press a 'Save image' button.

Edit 1:

I just tried to do background substraction with the closest frames on the shape border. The single pixels you see are flickering, which means it is noise and I can get easily get rid of them.

Edit 2:

The project is now over, and here's what we did: manual calibration of the Kinect by using the OpenNI driver, which provides directly the infrared image. The result is really good, but each calibration is specific to each Kinect.

Then, we applied a little transparency on the borders, and the result looks really nice! I can't provide pictures, however.

Answer 1

Your problem isn't just the noisy white pixels. You're missing significant parts of the person as well, e.g. part of his right hand. I'd recommend being more conservative with your thresholding of the depth data (allow more false positives). This would give you more noisy pixels, but at least you'd have the person in their entirety.

To get rid of the noisy pixels, I can think of a couple of things:

• Feather the outer pixels (reduce them in intensity/increase their transparency if you're using an alpha channel)
• Smooth the image, perform the edge detection on the smoothed image, then use these edges with your original sharp image.
• Do some skin region detection to mark parts that definitely belong to a person. See skin detection in the YUV color space? and Skin Color Detection
• For clothes, work with the hue and saturation image. If you know the color of the t-shirt (or that at least that it's not a neutral color), then this will stand out easily. If you don't know this information, then it may be worth building up a model of the person using the other frames (if there's a big gray blob that's moving around in your video, chances are that your subject is wearing a gray shirt)

The approaches aren't mutually exclusive so it may be worth trying to do them in combination. If I think of anything else, I'll post back here.

Answer 2

If there is no other way of resolving the jitter on the edges you could always try anti-alias as post-process.