How to display SkeletonStream on a DepthStream or ColorStream Kinect Wpf

Question

I would like to know how to overlay two Image objects in Wpf. I've made two streams which output a video, one for bones and another just a normal video. Now I would like to add them together, so the skeleton would be displayed on the color video. A part of my XAML code is

<Grid Name="layoutGrid">
    <Grid Name="VideoGrid" ClipToBounds="True" Background="AliceBlue">
        <Image Name="ColorImage" Width="640" Height="480"/>
        <Image Name="SkeletalImage" Width="640" Height="480"/>

        <Canvas Background="Transparent"/>
    </Grid>
    <StatusBar VerticalAlignment="Bottom" HorizontalAlignment="Center">
        <Button Name="Button1" Content="Skeleton Only" Width="120"/>
    </StatusBar>
</Grid>

When I do this, only the Skeletal image is displayed and if I switch the both Images then the color image is shown. It seems like Canvas background doesn't do the trick (to make the black part of skeletal image transparent)

Answer 1

I thought the problem was in my XAML code but it was actually at drawing the transparent background to set the render size. The background for skeleton image was black

dc.DrawRectangle(Brushes.Black, null, 
                  new Rect(0.0, 0.0, RenderWidth, RenderHeight));

instead of transparent

dc.DrawRectangle(Brushes.Transparent, null, 
                  new Rect(0.0, 0.0, RenderWidth, RenderHeight));

that fixed my problem. Thanks though

Answer 2

I've tried overlaying Skeletal onto color or depth on a canvas, but it never worked out. Instead, when I looked at the Kinect Explorer sample application in the SDK 1.6.0, they overlay using a grid. Unfortunately, I have not learned that much yet, but they used a grid for the color and depth, and a canvas for the skeletal tracking. I can post a snippet if it helps. Good luck!

Answer 3

You are asking basically two different questions: 1) How to display SkeletonStream on a DepthStream, which is very easy, 2) How to display SkeletonStream on a ColorStream, which is still easy but a little more complicated.

Answer to Question 1) The XYZ Vectors of the joints in the skeletal streams are in the same coordinate system as the Depth Stream. So if you plot the Deapth Stream as a quadratic mesh using the focal length of the depth camera from the Kinect SDK, you can plot the skeleton as a line plot in the same coordinate system. The source code that implements exactly what you want to do in OpenGL is available here that will give you an idea how to implement this in wpf.

Answer to Question 2) The video camera has a different field of view from the depth camera. To display the skeleton stream to the ColorStream you need first to map the XYZ vectors of the joints in the skeletal streams to the coordinate system of the depth stream and then draw first the video frame in the background and the skeleton as a line plot on the front. You can easily do the mapping using the U-V texture coordinates given by the Kinect SDK. See how you can get the UV here.