OpenCV warping image based on calcOpticalFlowFarneback

Question

I'm trying to perform a complex warp of an image using Dense Optical Flow. I am trying to warp the second image into roughly the same shape as the first image.

cv::Mat flow;
cv::calcOpticalFlowFarneback( mGrayFrame1, mGrayFrame2, flow, 0.5, 3, 15, 3, 5, 1.2, 0 );

cv::Mat newFrame = cv::Mat::zeros( frame.rows, frame.cols, frame.type() );
cv:remap( frame, newFrame, flow, cv::Mat(), CV_INTER_LINEAR );

I calculate the flow from two grayscale frames. I am now trying to remap my original (i.e. non-grayscale) image using this flow information using the cv::remap function. However, I get a very badly distorted image from it. I simply end up with an orange and black image that bears a small resemblance to my original image.

How do I use cv::remap with the calculated flow?

Answer 1

The remap function cannot work with flow directly. One must use a separate map that is computed by taking the backwards flow (from frame2 to frame1) and then offsetting each flow vector by its (x, y) location on the pixel grid. See details below.

Recall the backwards optical flow formula:

frame1(x, y) = frame2(x + flowx(x, y), y + flowy(x, y))

The remap function transforms the source image using a specified map:

dst(x, y) = src(mapx(x, y), mapy(x, y))

Comparing the two equations above, we may determine the map that remap requires:

mapx(x, y) = x + flowx(x, y)
mapy(x, y) = y + flowy(x, y)

Example:

Mat flow; // backward flow
calcOpticalFlowFarneback(nextFrame, prevFrame, flow);

Mat map(flow.size(), CV_32FC2);
for (int y = 0; y < map.rows; ++y)
{
    for (int x = 0; x < map.cols; ++x)
    {
        Point2f f = flow.at<Point2f>(y, x);
        map.at<Point2f>(y, x) = Point2f(x + f.x, y + f.y);
    }
}

Mat newFrame;
remap(prevFrame, newFrame, map);