Taking mean of images for background subtraction - incorrect results

Question

When I try to sum up N previous frames stored in a list and then dividing by num frames, the background model produced is not as expected. I can tell because I've tried the algo in Matlab earlier on the same video.

class TemporalMeanFilter {
private:
    int learningCount;
    list<Mat> history;
    int height, width;

    Mat buildModel(){
        if(history.size() == 0)
            return Mat();

        Mat image_avg(height, width, CV_8U, Scalar(0));
        double alpha = (1.0/history.size());

        list<Mat>::iterator it = history.begin();
        cout << "History size: " << history.size() << " Weight per cell: " << alpha << endl;

        while(it != history.end()){
            image_avg += (*it * alpha);
            it++;
        }

        return image_avg;
    }

public:
    TemporalMeanFilter(int height, int width, int learningCount){
        this->learningCount = learningCount;
        this->height = height;
        this->width = width;
    }

    void applyFrameDifference(Mat& frame_diff, Mat& bg_model, Mat& input_frame){
        if(history.size() == learningCount)
            history.pop_front();

        history.push_back(input_frame);

        bg_model = buildModel();
        frame_diff = bg_model - input_frame;
    }
};

//The main looks like this

// ... reading video from file

TemporalMeanFilter meanFilter(height, width, 50); //background subtraction algorithm
meanFilter.applyFrameDifference(diff_frame, bg_model, curr_frame);

//... displaying on screen ... prog ends

Image: http://imagegur.com/images/untitled.png

The left one is the bg_model, the middle is the curr_frame, and the right one is the output.

Maybe it's because of the rounding off done on CV_U8? I tried changing to CV_32FC1, but then the program just crashed because for some reason it couldn't add two CV_32FC1 Matrices.

Any insight would be greatly appreciated. Thanks!

More Info:

Inside the class, I now keep the average in a CV_16UC1 Mat to prevent clipping, how it results in an error after successive addition.

The add function / operator + both change the type of result from CV_16UC1 to CV8UC1. This error is caused by that. Any suggestion how to ask it preserve the original datatype? (PS: I asked politely... didn't work)

background_model += *it;

OpenCV Error: Bad argument (When the input arrays in add/subtract/multiply/divid
e functions have different types, the output array type must be explicitly speci
fied) in unknown function, file C:\buildslave64\win64_amdocl\2_4_PackSlave-win32
-vc11-shared\opencv\modules\core\src\arithm.cpp, line 1313

Answer 1

You're right that it's almost certainly the rounding errors you get by accumulating scaled greyscale values. There's no reason why it should crash using floating point pixels though, so you should try something like this:

Mat buildModel()
{
    if (history.size() == 0)
        return Mat();

    Mat image_avg = Mat::zeros(height, width, CV_32FC1);
    double alpha = (1.0 / history.size());

    list<Mat>::iterator it = history.begin();

    while (it != history.end())
    {
        Mat image_temp;
        (*it).convertTo(image_temp, CV_32FC1);
        image_avg += image_temp;
        it++;
    }
    image_avg *= alpha;

    return image_avg;
}

Depending on what you want to do with the result, you may need to normalize it or rescale it or convert back to greyscale before display etc.