Extended client ownership of MediaCodec encoder output buffers for RTMP streaming

Question

Background:

I've connected Android's MediaCodec to FFmpeg for muxing a variety of formats not supported by MediaMuxer, including rtmp:// output via a .flv container. Such streaming muxers require longer, unpredictable ownership of MediaCodec's output buffers, as they may perform networking I/O on any packet processing step. For my video stream, I'm using MediaCodec configured for Surface input. To decouple muxing from encoding, I queue MediaCodec's ByteBuffer output buffers to my muxer via a Handler.

All works splendidly if I mux the .flv output to file, rather than rtmp endpoint.

Problem:

When muxing to rtmp://... endpoint I notice my streaming application begins to block on calls to eglSwapBuffers(mEGLDisplay, mEncodingEGLSurface) at dequeueOutputBuffer() once I'm retaining even a few MediaCodec output buffers in my muxing queue as MediaCodec seems to be locked to only 4 output buffers.

Any tricks to avoid copying all encoder output returned by MediaCodec#dequeueOutputBuffers and immediately calling releaseOutputBuffer(...)?

The full source of my project is available on Github. Specifically, see:

• AndroidEncoder.java: Abstract Encoder class with shared behavior between Audio and Video encoders: mainly drainEncoder(). Writes data to a Muxer instance.
• FFmpegMuxer.java: Implements Muxer
• CameraEncoder.java. Sends camera frames to an AndroidEncoder subclass configured for Video encoding.

Systrace

Here's some systrace output streaming 720p @ 2Mbps video to Zencoder.

Solved

Copying then releasing the MediaCodec encoder output ByteBuffers as soon as they're available solves the issue without significantly affecting performance. I recycle the ByteBuffer copies in an ArrayDeque<ByteBuffer> for each muxer track, which limits the number of allocations.

Answer 1

Unfortunately this kind of use case is not supported by most Android phones. MediaCodec is just an abstraction of the OMX IL API used by codec vendors on the devices. The vendor codecs require a certain number of input and output buffers for a given configuration.

While theoretically, having one output buffer queued with the encoder should be enough, a lot of the times this is not supported by vendor codecs because it results in lower encoding performance; hence the codec stalls. This is even more prevalent with input buffers.

There is no application control on the number of input/output buffers that are allocated for a MediaCodec instance - Android tries to allocate the minimum number of buffers required, to save memory. Therefore, your only option is to copy the output buffers. While this is not ideal, the encoded buffers tend to be reasonably small.