Stitching Two Webcam Feeds Together in OpenCV

Question

Using the code below:

#include <opencv2/opencv.hpp>
#include <opencv2/stitching/stitcher.hpp>
#include <iostream>
#include <vector>

using namespace std;
using namespace cv;

int main(int argc, char *argv[])
{
    Mat fr1, fr2, pano;
    bool try_use_gpu = false;
    vector<Mat> imgs;
    VideoCapture cap(0), cap2(1);

    while (true)
    {
        cap >> fr1;
        cap2 >> fr2;
        imgs.push_back(fr1.clone());
        imgs.push_back(fr2.clone());

        Stitcher test = Stitcher::createDefault(try_use_gpu);
        Stitcher::Status status = test.stitch(imgs, pano);

        if (status != Stitcher::OK)
        {
            cout << "Error stitching - Code: " <<int(status)<<endl;
            return -1;
        }

        imshow("Frame 1", fr1);
        imshow("Frame 2", fr2);
        imshow("Stitched Image", pano);

        if(waitKey(30) >= 0) 
            break;
    }
    return 0;
}

This code throws a status error of 1 out there. I don't know what that means, nor do I know why this thing is having a hard time with webcam feeds. What's the matter?

-Tony

Answer 1

The problem, for whatever reason, lies in the .clone() segment. Changing the code to:

int main(int argc, char *argv[])
{
    Mat fr1, fr2, copy1, copy2, pano;
    bool try_use_gpu = false;
    vector<Mat> imgs;
    VideoCapture cap(0), cap2(1);

    while (true)
    {
        cap >> fr1;
        cap2 >> fr2;
        fr1.copyTo(copy1);
        fr2.copyTo(copy2);        

        imgs.push_back(copy1);
        imgs.push_back(copy2);

        //ETC
     }
     return 0;
}

This worked just fine.

Answer 2

The error is somewhere in your capture process, not the stitching part. This code works fine (using these example images):

#include <opencv2/opencv.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/stitching/stitcher.hpp>
#include <iostream>
#include <vector>

using namespace std;
using namespace cv;

int main()
{
    Mat fr1 = imread("a.jpg");
    Mat fr2 = imread("b.jpg");
    Mat pano;
    vector<Mat> imgs;

    Stitcher stitcher = Stitcher::createDefault(); // The value you entered here is the default

    imgs.push_back(fr1);
    imgs.push_back(fr2);

    Stitcher::Status status = stitcher.stitch(imgs, pano);

    if (status != Stitcher::OK)
    {
        cout << "Error stitching - Code: " <<int(status)<<endl;
        return -1;
    }

    imshow("Frame 1", imgs[0]);
    imshow("Frame 2", imgs[1]);
    imshow("Stitched Image", pano);
    waitKey();

    return 0;
}

The error message Nik Bougalis dug up sounds like the stitcher can't connect the images. Are the images clear enough for the stitcher to find correspondences?

If you're sure they are, split your problem further to find the real error. Can you tweak the stitcher to work with still frames from your cameras? Are your cameras capturing correctly? Which type of image do they return?

On another note, stitching isn't very likely to work in real time, which makes your loop during capturing look a bit out of place. You might want to either capture your frames in advance and do it all in post-processing or expect a lot of manual optimization to get anywhere near a respectable frame rate.

Answer 3

Looking through the OpenCV website, we find this:

class CV_EXPORTS Stitcher
{
public:
    enum { ORIG_RESOL = -1 };
    enum Status { OK, ERR_NEED_MORE_IMGS };

    // ... other stuff

Since the returned code is of type Sticher::Status we can be fairly certain that 1 actually is Sticher::Status::ERR_NEED_MORE_IMGS. Which suggests that the sticher needs more images.

Not very informative I'm afraid, but it's a start for you. Have you looked at any of the stitching examples out there?