Image Stitching from a live Video Stream in OpenCv

Question

I am trying to Stitch an image from a live video camera (more like a panorama) using OpenCv. The stitching is working fine. My problem is, i want the stitching to be done in real time say around 30 mph but the processing of the stitching is slow.

I want to use Threads to improve the speed but in order to use them do i need to store my live video stream or is there any way to directly use threads for the live stream.

Here is a sample code:

  SapAcqDevice *pAcq=new SapAcqDevice("Genie_HM1400_1", false);
  SapBuffer *pBuffer = new SapBuffer(20,pAcq);
  SapView *pView=new SapView(pBuffer,(HWND)-1);
  SapAcqDeviceToBuf *pTransfer= new SapAcqDeviceToB(pAcq,pBuffer,XferCallback,pView);



  pAcq->Create();
  pBuffer->Create();
  pView->Create();

  pTransfer->Create();  
  pTransfer->Grab();

  printf("Press any key to stop grab\n");
  getch();

  pTransfer->Freeze();  
  pTransfer->Wait(5000);

  printf("Press any key to terminate\n");
  getch();

This above code is used to capture the live stream. The XferCallback function is used to do the processing of the frames. In this function i call my stitch engine. Since the processing of the engine is slow i want to use threads.

Here is a sample code of the callback function:

SapView *pView = (SapView *) pInfo->GetContext();
SapBuffer *pBuffer;
pBuffer = pView->GetBuffer();

void *pData=NULL;
pBuffer->GetAddress(&pData);

int width=pBuffer->GetWidth();
int height=pBuffer->GetHeight();
int depth=pBuffer->GetPixelDepth();

IplImage *fram;
fram = cvCreateImage(cvSize(width,height),depth,1);
cvSetImageData(fram,pData,width);

stitching(frame_num , fram);

cvWaitKey(1);

frame_num++;

I want many threads working on the stitch engine.

Answer 1

If you think you can get the stitching fast enough using threads, then go for it.

do i need to store my live video stream or is there any way to directly use threads for the live stream.

You might benefit from setting up a ring buffer with preallocated frames. You know the image size isn't going to change. So your Sapera acquisition callback simply pushes a frame into the buffer.

You then have another thread that sits there stitching as fast as it can and maintaining state information to help optimize the next stitch. You have not given much information about the stitching process, but presumably you can make it parallel with OpenMP. If that is fast enough to keep up with frame acquisition then you'll be fine. If not, then you will start dropping frames because your ring buffer is full.

As hinted above, you can probably predict where the stitching for the next frame ought to begin. This is on the basis that movement between one frame and the next should be reasonably small and/or smooth. This way you narrow your search and greatly improve the speed.