C# image processing on Kinect video using AForge

Question

My goal :
use Kinect video to do shape recognition (large rectangle on the picture), draw rectangle on the picture to highlights the results and display.

The techno I use :

• C# code,
• AForge and more specifically its shape checker

http://www.aforgenet.com/articles/shape_checker/

How the magic should work :

1. Every time a frame is ready I get the frame data as bytes array and transform it to bitmap to allow me to analyze it
2. Apply the shape recognition algorithm
3. Render the result...

My problem :
The whole process works so far but when I try to render the result in a WPF Image it lags terribly... (1 frame every 10 sec)...

My code:

// AllFramesReady is called every time a frame is ready to use...
private void AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {
        using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
        {
            if (colorFrame == null)
            {
                return;
            }

            _Pixels = new byte[colorFrame.PixelDataLength];
            colorFrame.CopyPixelDataTo(_Pixels);

            // Analyze the image

            int stride = colorFrame.Width * 4;
            System.Drawing.Size size = new System.Drawing.Size(colorFrame.Width, colorFrame.Height);
            // get the bitmap from bytes
            Bitmap btmap = BytesToBmp(_Pixels, size);
            //analyze the data...
            btmap = _shapeReco.AnalyzeImage(btmap);

            // copy the new data back to pixels
            _Pixels = BmpToBytes(btmap);

            // rendering the analyzed image
            imageAnalyzed.Source =
                BitmapSource.Create(colorFrame.Width, colorFrame.Height,
                96, 96, PixelFormats.Bgr32, null, _Pixels, stride);
        }
    }


//
// HERE IS MY SHAPE RECOGNIZER THAT IMPLEMENTS THE SHAPE RECOGNITION ALGORITHM
//

using System;
using System.Collections.Generic;
using System.Drawing;
using System.Drawing.Imaging;

using AForge;
using AForge.Imaging;
using AForge.Math.Geometry;


namespace KinectSetupDev
{
    class MyShapeRecognizer
    {

        private static String TAG = "MyShapeRecognizer";

        /***************************************************************************
         *                                VARIABLES                                *
         ***************************************************************************/

        private SimpleShapeChecker _ShapeChecker;
        private Bitmap _Image; // the image to analyze
        private Blob[] _Blobs;
        private BlobCounter _BlobCounter;

        /***************************************************************************
         *                              CONSTRUCTOR                                *
         ***************************************************************************/

        public MyShapeRecognizer()
        {
            Debug.Log(TAG, "MyShapeRecognizer");

            _ShapeChecker = new SimpleShapeChecker();
            _Image = new Bitmap(300, 400);
            _Blobs = null;
            _BlobCounter = null;
        }

        /***************************************************************************
         *                                METHODS                                  *
         ***************************************************************************/

        public Bitmap AnalyzeImage(Bitmap image)
        {
            Debug.Log(TAG, "AnalyzeImage");

            this._Image = image;
            this.LocatingObjects();
            this.AnalyzeObjects();

            return _Image;
        }

        private void LocatingObjects()
        {
            Debug.Log(TAG, "LocatingObjects");

            // lock image
            BitmapData bitmapData = _Image.LockBits(
                new Rectangle(0, 0, _Image.Width, _Image.Height),
                ImageLockMode.ReadOnly, _Image.PixelFormat);

            //locating objects
            _BlobCounter = new BlobCounter();

            _BlobCounter.FilterBlobs = true;
            _BlobCounter.MinHeight = 5;
            _BlobCounter.MinWidth = 5;

            _BlobCounter.ProcessImage(bitmapData);
            _Blobs = _BlobCounter.GetObjectsInformation();

            // unlock image
            _Image.UnlockBits(bitmapData);
        }

        private void AnalyzeObjects()
        {
            Debug.Log(TAG, "AnalyzeObjects");

            Graphics g = Graphics.FromImage(_Image);

            [DRAW RECT OR CIRCLE ON GRAPHICS]

            g.Dispose();
        }

        // Conver list of AForge.NET's points to array of .NET points
                                        private System.Drawing.Point[] ToPointsArray(List<IntPoint> points)
    {
        System.Drawing.Point[] array = new System.Drawing.Point[points.Count];

        for (int i = 0, n = points.Count; i < n; i++)
        {
            array[i] = new System.Drawing.Point(points[i].X, points[i].Y);
        }

        return array;
    }

    }
}

I can provide full code (MV C# 2010 project...). I appreciate any help!

Thanks.

Answer 1

Hmmm, from your comment above it looks like the AForge code is making things run slowly. From what I can think of, you have the following options

1. (Re)write the blob processing to use OpenCL (on CPU or GPU) to speed things up
2. Use a faster image processing library like EmguCV? There may be many more
3. If real-time processing is not required, buffer the input frames and process them as fast as AForge.NET can in multiple background threads to try and absorb as much latency as you can.

And perhaps more I'm missing here - but these should get you on the right track.