Right choice to play audio and video content

Question

What I'm doing :

I need to play audio and video files that are not supported by Apple on iPhone/iPad for example mkv/mka files which my contain several audio channels.

I'm using libffmpeg to find audio and video streams in media file. Video is being decoded with avcodec_decode_video2 and audio with avcodec_decode_audio3 the return values are following for each function are following

• avcodec_decode_video2 - returns AVFrame structure which encapsulates information about the video video frame from the pakcage, specifically is has data field which is a pointer to the picture/channel planes.
• avcodec_decode_audio3 - returns samples of type int16_t * which I guess is the raw audio data

So basically I've done all this and successfully decoding the media content.

What I have to do : I've to play the audio and video accordingly using Apples services. The playback I need to perform should support mixing of audio channels while playing video, i.e. let say mkv file contains two audio channel and a video channel. So I would like to know which service will be the appropriate choice for me ? My research showed that AudioQueue service might be useful audio playback, and probably AVFoundation for video.

Please help to find the right technology for my case i.e. video playeback + audio playback with possible audio channel mixing.

Answer 1

You are on the right path. If you are only playing audio (not recording at all) then I would use AudioQueues. It will do the mixing for you. If you are recording then you should use AudioUnits. Take a look at the MixerHost example project from Apple. For video I recommend using OpenGL. Assuming the image buffer is in YUV420 then you can render this with a simple two pass shader setup. I do believe there is an Apple example project showing how to do this. In any case you could render any pixel format using OpenGL and a shader to convert the pixel format to RGBA. Hope this help.