Problems of using MediaCodec.getOutputFormat() for an encoder in Android 4.1/4.2 devices

Question

I'm trying to use MediaCodec to encode frames (either by camera or decoder) into a video. When processing the encoder output by dequeueOutputBuffer(), I expect to receive the return index = MediaCodec.INFO_OUTPUT_FORMAT_CHANGED, so I can call getOutputFormat() to get the encoder output format as the input of the currently used ffmpeg muxer.

I have tested some pad/phone devices with Android version 4.1~4.3. All of them have at least one hardware video AVC encoder and is used in the test. On the devices with version 4.3, the encoder gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED before writing the encoded data as expected, and the output format returned from getOutputFormat() can be used by the muxer correctly. On the devices with 4.2.2 or lower, the encoder never gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED while it can still output the encoded elementary stream, but the muxer cannot know the exact output format.

I want to ask the following questions:

1. Does the behavior of encoder (gives MediaCodec.INFO_OUTPUT_FORMAT_CHANGED or not before outputing encoded data) depend on the Android API Level or the chips on individual devices?
2. If the encoder writes data before MediaCodec.INFO_OUTPUT_FORMAT_CHANGED appears, is there any way to get the output format of the encoded data?
3. The encoder still output the codec config data (with flag MediaCodec.BUFFER_FLAG_CODEC_CONFIG) on the devices before the encoded data. It is mostly used to config a decoder, but can I derive the output format by the codec config data?

I have tried these solutions to get the output format but failed:

• Call getOutputFormat() frequently during the whole encode process. However, all of them throw IllegalStateException without the appearance of MediaCodec.INFO_OUTPUT_FORMAT_CHANGED.

• Use the initial MediaFormat use to config the encoder at the beginning, like the example:

  m_init_encode_format = MediaFormat.createVideoFormat(m_encode_video_mime, m_frame_width, m_frame_height);
  int encode_bit_rate = 3000000;
  int encode_frame_rate = 15;
  int encode_iframe_interval = 2;
  
  m_init_encode_format.setInteger(MediaFormat.KEY_COLOR_FORMAT, m_encode_color_format);
  m_init_encode_format.setInteger(MediaFormat.KEY_BIT_RATE, encode_bit_rate);
  m_init_encode_format.setInteger(MediaFormat.KEY_FRAME_RATE, encode_frame_rate);
  m_init_encode_format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, encode_iframe_interval);
  
  m_encoder = MediaCodec.createByCodecName(m_video_encoder_codec_info.getName());
  m_encoder.configure(m_init_encode_format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  
  // Assume m_init_encode_format is the output format of the encoder
  

  However it fails since the output format of the encoder is still "changed" from the initial one.

Please help me to realize the behavior of an encoder, and if there is any solution to query the output format if the required MediaCodec.INFO_OUTPUT_FORMAT_CHANGED is missing.

---

By comparing the output format and the codec config data, the missing fields are csd-0, csd-1, and a "what" field with value = 1869968451. (I do not understand the "what" field. It seems to be a constant and is not required. Can anyone tell me about its meaning?)

If I parse the codec config data as the csd-1 field (last 8 bytes) and csd-0 field (remaining bytes), it seems that the muxer can work correctly and output a video playable on all of the testing devices. (But I want to ask: is this 8-byte assumption correct, or there is more reliable way to parse the data?)

However, I got another problem that If I decode the video by Android MediaCodec again, the BufferInfo.presentationTimeUs value get by dequeueOutputBuffer() is 0 for most of the decoded frames. Only the last few frames has correct time. The sample time get by MediaExtractor.getSampleTime() is correct and exactly the value I set to the encoder/muxer, but the decoded frame time is not. This issue only happen on 4.2.2 or lower device.

It is strange that the frame time is incorrect but the video can be playback in correct speed on the device. (Most of the devices with 4.2.2 or lower I've tested has only 1 Video AVC decoder.) Do I need to set other fields that may affect the presentation time?

Answer 1

The behavior of MediaCodec encoders was changed in Android 4.3 to accommodate the introduction of the MediaMuxer class. In Android 4.3, you will always receive INFO_OUTPUT_FORMAT_CHANGED from the encoder. In previous releases, you will not. (I've updated the relevant FAQ entry.)

There is no way to query the encoder for the MediaFormat.

I haven't used an ffmpeg-based muxer, so I'm not sure what information it needs. If it's looking for the csd-0 / csd-1 keys, you can extract those from the CODEC_CONFIG packet (I think you have to parse the SPS / PPS values out and place them in the separate keys). Examining the contents of the MediaFormat on a 4.3 device will show you which fields you're lacking.

Answer 2

To init ffmpeg muxer for video correctly i use next:

int outputBufferIndex = videoCodec.dequeueOutputBuffer(bufferInfo, -1);
if (MediaCodec.BUFFER_FLAG_CODEC_CONFIG == bufferInfo.flags) {
    ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
    headerData = new byte[bufferInfo.size];
    outputBuffer.get(headerData);

    // jni call
    WriteVideoHeader(headerData, headerData.length);

    videoCodec.releaseOutputBuffer(outputBufferIndex, false);   
}

In jni I use something like this:

jint Java_com_an_FileWriterEx_WriteVideoHeader(JNIEnv * env, jobject this, jbyteArray data, jint datasize)
{
    jboolean isCopy;
    jbyte* rawjBytes = (*env)->GetByteArrayElements(env, data, &isCopy);    
    WriteVideoHeaderInternal(env, m_pFormatCtx, m_pVideoStream, rawjBytes, datasize);   
    (*env)->ReleaseByteArrayElements(env, data, rawjBytes, 0);
    return 0;
}

jint WriteVideoHeaderInternal(JNIEnv * env, AVFormatContext* pFormatCtx, AVStream*     pVideoStream, jbyte* data, jint datasize)
{
  jboolean bNoError = JNI_TRUE;

  jbyte* pExtDataBuffer = av_malloc(datasize);
  if(!pExtDataBuffer) 
  {
    LOGI("av alloc error\n");
    bNoError = JNI_FALSE;
  }

  if (bNoError)
  {
    memcpy(pExtDataBuffer, data, datasize * sizeof(jbyte));

    pVideoStream->codec->extradata = pExtDataBuffer;
    pVideoStream->codec->extradata_size = datasize;
  } 
}

Answer 3

For the parsing of codec config data, it is wrong that assuming the last 8 bytes are the PPS data. The data must be parsed according to the start code and nal_unit_type.