x264 encoding with libav

Question

I try to encode raw image data to x264 with libav:

AVPacket vpkt = { 0 };
av_init_packet(&vpkt);

int got;
int ret = avcodec_encode_video2(vcodec, &vpkt, frameyuv.get(), &got);

if (!ret && got && vpkt.size) {
    if (vpkt.pts != AV_NOPTS_VALUE) {
        vpkt.pts = av_rescale_q(vpkt.pts, vcodec->time_base, videost->time_base);
    }
    if (vpkt.dts != AV_NOPTS_VALUE) {
        vpkt.dts = av_rescale_q(vpkt.dts, vcodec->time_base, videost->time_base);
    }

    vpkt.stream_index = videost->index;

    if(vcodec->coded_frame->key_frame) {
        vpkt.flags |= AV_PKT_FLAG_KEY;
    }
    /* -> will return -22 if max_b_frames > 0 */
    ret = av_interleaved_write_frame(oc, &vpkt);
}

Runs fine when vcodec->max_b_frames is set to 0, but on any other value av_interleaved_write_frame returns -22 (invalid argument).

/* will fail */
c->max_b_frames = 3;
/* -> ok*/
c->max_b_frames = 0;

Why? Am i missing something?

Codec options are

AVDictionary *opts = NULL;
av_dict_set(&opts, "vprofile", "baseline", 0);

/* ... */
c->codec_type = AVMEDIA_TYPE_VIDEO;
c->bit_rate = 500 * 1000;
c->width = VideoWidth;
c->height = VideoHeight;
c->time_base.den = fps;
c->time_base.num = 1;
c->pix_fmt = AV_PIX_FMT_YUV420P;

Container format is mp4.

Answer 1

You need to av_rescale_q into your output context's timebase, not the video stream one. What it appears that you are doing is nothing with av_rescale_q. Try:

av_rescale_q(vpkt.pts, vcodec->time_base, [output context]->time_base);

If you continue having problems, almost always problems with B frames are indicative of a bad DTS value. Consider setting DTS to AV_NOPTS_VALUE and let the demuxer solve it itself.

Remember to decode a B frame you must know the frames on either side of it, thus they must be demuxed first, consider 3 frames:

I B I

The first frame must be decoded first, followed by the third and finally the second. This information is derived from the DTS.

Finally, figure out why you're getting a -22 by following this stack overflow. I would be shocked if it isn't a "non-monotonically increasing PTS/DTS" error.