Segmentation fault while avcodec_encode_video2

Question 1

Finally i solved my problem.

The problem is (apart from the documentation of libav) avpacket is not a (real) copy of the picture in the packet. it just points to the data of the packet. You have to make a copy, or better you have to let it libav do.

So first i created a new avframe for the output and a buffer on which the output avframe is pointing to.

AVFrame *outpic = avcodec_alloc_frame();
nbytes = avpicture_get_size(codecCtxOut->pix_fmt, codecCtxOut->width, codecCtxOut->height);
uint8_t* outbuffer = (uint8_t*)av_malloc(nbytes);

This buffer is used for the conversion from input to output. Then in the loop i have to fillup the outpic (avframe) with the buffer. I have found in the code that this function is filling up the plane pointers with the buffer. see here

avpicture_fill((AVPicture*)outpic, outbuffer, AV_PIX_FMT_YUV420P, codecCtxOut->width, codecCtxOut->height);

Then i converted the inpic to outpic using sws_scale. But first you have to setup the swscontext.

SwsContext* swsCtx_ = sws_getContext(codecCtxIn->width, codecCtxIn->height,
                                     codecCtxIn->pix_fmt,
                                     codecCtxOut->width, codecCtxOut->height,
                                     codecCtxOut->pix_fmt,
                                     SWS_BICUBIC, NULL, NULL, NULL);

sws_scale(swsCtx_, inpic->data, inpic->linesize, 0, codecCtxIn->height, outpic->data, outpic->linesize);

Then you can encode the outpic into pktout (avpacket for output). But first do free the output packet, otherwise you will get an error and a leak... see here

av_free_packet(pktOut);

if(avcodec_encode_video2(streamOut->codec, pktOut, outpic, &fff) < 0) {
  std::cout << "shit frame" << std::endl;
  continue;
}
// and write it to the file
formatOut->write_packet(formatCtxOut, pktOut);

So now it works (nearly fine) for me. Still a small memory leak, but this i can spot later.

Question 2

I see at least two problems with that transcoding loop:

1) You're not checking whether the decoder produced a frame. Many decoders have a delay between input and output, so a decode call won't necessarily produce a frame even if no error occurs. You just have to keep passing packets to the decoder until it starts returning frames (and then flush it with NULL packets at the end, as described in the documentation).

The result is that you're passing an uninitialized frame to the encoder, which is probably the reason for the crash.

2) Another problem I see is that you're initing the ouput packet only once. As the documentation says

The user can supply an output buffer by setting avpkt->data and avpkt->size prior to calling the function, but if the size of the user-provided data is not large enough, encoding will fail. All other AVPacket fields will be reset by the encoder using av_init_packet(). If avpkt->data is NULL, the encoder will allocate it. The encoder will set avpkt->size to the size of the output packet. The returned data (if any) belongs to the caller, he is responsible for freeing it.

So if you only initialize it once before starting the transcode loop, on each iteration after the first it will contain the old data. The encoder will think you want to use that buffer for encoding and overwrite it. This will end in tears if you've already passed that packet to a muxer or something like that. So make sure to init packet data and size to NULL/0 before each encode call.

Segmentation fault while avcodec_encode_video2

Version:

Addendum: