How to create video using avcodec from jpeg images of type OpenCV::Mat?

Question

I have colored jpeg images of OpenCV::Mat type and I create from them video using avcodec. The video that I get is upside-down, black & white and each row of each frame is shifted and I got diagonal line. What could be the reason for such output? Follow this link to watch the video I get using avcodec. I'm using acpicture_fill function to create avFrame from cv::Mat frame!

P.S. Each cv::Mat cvFrame has width=810, height=610, step=2432 I noticed that avFrame (that is filled by acpicture_fill) has linesize[0]=2430 I tried manually setting avFrame->linesizep0]=2432 and not 2430 but it still didn't helped.

======== CODE =========================================================

AVCodec *encoder = avcodec_find_encoder(AV_CODEC_ID_H264);
AVStream *outStream = avformat_new_stream(outContainer, encoder);
avcodec_get_context_defaults3(outStream->codec, encoder);

outStream->codec->pix_fmt = AV_PIX_FMT_YUV420P;
outStream->codec->width = 810;
outStream->codec->height = 610;
//...

SwsContext *swsCtx = sws_getContext(outStream->codec->width, outStream->codec->height, PIX_FMT_RGB24,
                                    outStream->codec->width, outStream->codec->height,  outStream->codec->pix_fmt, SWS_BICUBIC, NULL, NULL, NULL);

for (uint i=0; i < frameNums; i++)
{
    // get frame at location I using OpenCV
    cv::Mat cvFrame;
    myReader.getFrame(cvFrame, i); 
    cv::Size frameSize = cvFrame.size();    
    //Each cv::Mat cvFrame has  width=810, height=610, step=2432


1.  // create AVPicture from cv::Mat frame
2.  avpicture_fill((AVPicture*)avFrame, cvFrame.data, PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
3avFrame->width = frameSize.width;
4.  avFrame->height = frameSize.height;

    // rescale to outStream format
    sws_scale(swsCtx, avFrame->data, avFrame->linesize, 0, outStream->codec->height, avFrameRescaledFrame->data, avFrameRescaledFrame ->linesize);
encoderRescaledFrame->pts=i;
avFrameRescaledFrame->width = frameSize.width;
    avFrameRescaledFrame->height = frameSize.height;

av_init_packet(&avEncodedPacket);
    avEncodedPacket.data = NULL;
    avEncodedPacket.size = 0;

    // encode rescaled frame
    if(avcodec_encode_video2(outStream->codec, &avEncodedPacket, avFrameRescaledFrame, &got_frame) < 0) exit(1);
    if(got_frame)
    {
        if (avEncodedPacket.pts != AV_NOPTS_VALUE)
            avEncodedPacket.pts =  av_rescale_q(avEncodedPacket.pts, outStream->codec->time_base, outStream->time_base);
        if (avEncodedPacket.dts != AV_NOPTS_VALUE)
            avEncodedPacket.dts = av_rescale_q(avEncodedPacket.dts, outStream->codec->time_base, outStream->time_base);

        // outContainer is "mp4"
        av_write_frame(outContainer, & avEncodedPacket);

        av_free_packet(&encodedPacket);
    }
}

UPDATED

As @Alex suggested I changed the lines 1-4 with the code below

int width = frameSize.width, height = frameSize.height; 
avpicture_alloc((AVPicture*)avFrame, AV_PIX_FMT_RGB24, outStream->codec->width, outStream->codec->height);
for (int h = 0; h < height; h++)
{
     memcpy(&(avFrame->data[0][h*avFrame->linesize[0]]), &(cvFrame.data[h*cvFrame.step]), width*3);
}

The video (here) I get now is almost perfect. It's NOT upside-down, NOT black & white, BUT it seems that one of the RGB components is missing. Every brown/red colors became blue (in original images it should be vice-verse). What could be the problem? Could rescaling(sws_scale) to AV_PIX_FMT_YUV420P format causes this?

Answer 1

The problem in a nutshell: avpicture_fill() expects no padding between rows, ie the stride (step) to be equal to width*sizeof(pixel), ie 810*3 = 2430. The actual stride of the data in cv::Mat step as you say is 2432 which is different, so just passing the data directly won't work. There is no way to tell avpicture_fill() to use a different stride for the input data; it is not part of the API (you might say it should be :)

There are two possible solutions:

Create an array in which the input data is contiguous, no padding between rows. You'd have to memcopy each row from the cv::Mat into that array. Then pass it to avpicture_fill().

int width, height; // get from mat
uint8_t* buf = malloc(width * height * 3); // 3 bytes per pixel
for (int i = 0; i < height; i++)
{
    memcpy( &( buf[ i*width*3 ] ), &( mat->data[ i*mat->step ] ), width*3 );
}
avpicture_fill(..., buf, ...)

Btw, to flip the video vertically, you can do this to copy the last row to the first and so forth:

...
    memcpy( &( buf[ i*width*3 ] ), &( mat->data[ (height - i - 1)*mat->step ] ), width*3 );
...

Or, fill in the AVPicture yourself:

AVPicture* pic = malloc(sizeof(AVPicture));
avpicture_alloc(pic, PIX_FMT_BGR24, width, height);
for (int i = 0; i < height; i++)
{
    memcpy( &( pic->data[0][ i*pic->linesize[0] ] ),  &( mat->data[ i*mat->step ] ), width*3);
}

There is no need to allocate pic->data[0] or set pic->linesize[0], avpicture_alloc() should do that. There is also no need to fill in data[1] or data[2], those should be null.

EDIT: Removed old code which showed copying R, G, B to separate planes. PIX_FMT_BGR24 is not a planar format.

I'm not familiar enough with OpenCV C++ API to figure out how to get the width and height (it's not mat->width, obviously) but I think you know what I mean.

P.S. Btw, your video is not actually black and white. It's just that each successive row is offset by two bytes, so the colors are rotated: red becomes green, green becomes blue, and so forth. The result is grayscale-ish, but if you look closely the individual rows are colored.

Answer 2

Have you considered using OpenCV's features to create the video for you? It's much more easier since your data is already store in a cv::Mat.

If you would like to keep your approach, you could simply rotate the cv::Mat.

Answer 3

About the color problem in the UPDATE of the original post. Is that caused by,

OpenCV Mat is (BGR) -> FFmpeg AVFrame is (RGB) ?

If so, try,

cvtColor( cvFrame , cvFrame , CV_BGR2RGB ) ; 

before line 1.