How to invoke my custom decoder in android source code

Question

I am trying to add my software HEVC decoder to android source code. I wish to generate the libstagefright.so file.

I have referred to other similar questions and were not of much help for me. I was not able to get a clear picture with the following questions or the reference guide.

1)Integrating a Codec to Android multimedia framework.

2)Integrate a custom decoder to play on android and display fps at the same time.

3)How to register a OMX core for adding a new decoder.

Can someone please help me with the steps for integration of a new custom decoder with Android.

Right now, I have just added the to media_codecs.xml and { "OMX.google.h265.decoder" , "hevcdec" , "video_decoder.hevc" } in SoftOMXPlugin.cpp file.

Which other files need to be edited for a new format to be added to android source code? Which functions invoke my decoder source code(Does the softomxcplugin source file invoke my decoder ?).

Answer 1

From your query and earlier discussions, please find the steps to integrate a HEVC component into Android framework. Since, your component is a SW component, you would have to integrate the same as a SoftVideoDecoderOMXComponent.

What is SoftVideoDecoderOMXComponent?

In the latest releases of the Android Stagefright framework, SW accelerated codecs are integrated into the framework using an OMX like interface. Earlier, this wasn't the case and had some challenges of it's own. So, Google have streamlined the integration strategy and mandated that all SW codecs will employ a SoftOMXComponent like interface.

Majority of the OMX IL calls are handled by the base class and hence, the implementation of a new codec component is relatively easy as described below.

I would take the example of AVC codec integration to provide an overview. It is highly recommended that the reader is familiar with OMX IL 1.1.2 specification which describes the structure, functioning and state machine of an OMX IL video decoder component.

Note: HEVC is not yet a part of OMX IL specification and hence, the recommendation is mainly to understand the structure and functioning of the component.

Creation of SoftHEVC component

Please refer to the header file of SoftAVC.h and the corresponding source SoftAVC.cpp.

You will have to implement a similar set of files. It is highly recommended to reuse the overall implementation from AVC due to some inherent similarities.

SoftHEVC.cpp implementation

1. You would have to define a SoftHEVC component in SoftHEVC.h which derives from SoftVideoDecoderOMXComponent. This would ensure that all OMX calls are handled suitably by the base class.

2. You would have to define a table of support profile-level combinations as found in CodecProfileLevel.

3. In the constructor, you can initialize most of the variables in a similar fashion. Since this is a video decoder component, you will have to initialize 2 ports viz.,input and output. ctor invokes a initDecoder to initialize the component. You would also have to implement a similar functionality for your codec.

4. The dtor is self explanatory and hence, I will skip explaining the same.

5. onQueueFilled is invoked when a buffer filled with a frame worth of bitstream data is provided for processing on input port or a free buffer is provided for output. This invokes the main decoding function H264SwDecDecode. Now, for the first frame, you could encounter a change in resolutions as compared to the originally initialized resolution. This is handled by 2 scenarios as described in the next point.

6. You would observe 2 functions handlePortSettingsChanged and handleCropRectEvent. These 2 events are important from an output buffer's perspective. handlePortSettingsChanged signifies a change in the output buffer dimensions as compared to the originally initialized size and hence, provides an event callback to the user to free up the current allocation and reallocate the same. handleCropRectEvent indicates that the __cropping window__which is communicated to the user. Usually, this doesn't require a buffer reallocation.

7. drainOneOutputBuffer will copy the decoded frame onto the output port's buffer and notify the caller about the availability of the decoded buffer.

8. In onQueueFilled, after a successful decode, the input buffer which has been consumed is also returned back to the caller.

9. The rest of the functions are very straightforward and I feel you could simply reuse majority of the implementation.

10. For registration of the component, you will have to implement createSoftOMXComponent which creates the SoftHEVC component as shown here.

Since, you have already handled the registered the component, I am skipping that part. Just for reference, I presume you have registered the component in kComponents array in SoftOMXPlugin. Also, since HEVC is not a known MIME type, you would have to register the same. There would be a change required in MediaDefs.cpp where you will have to introduce a new entry, MEDIA_MIMETYPE_VIDEO_HEVC similar to existing formats like AVC and support changes in OMXCodec.cpp and ACodec.cpp.

With these steps, I presume you should be able to integrate your SW decoder and be able to realize your playback.