Convert MPEG4 to MPEGTS on Android with FFmpeg

Question

Ok, so obviously I know very little to none about ffmpeg API when I made the original post... it is quite overwhelming when one starts learning about digital media and conversion details. After reading quite a bit more and going through ffmpeg source, I was able to get a working output from mp4 to mpegts. The concept is similar to executing:

ffmpeg -i in.mp4 -vcodec copy -acodec copy -vbsf h264_mp4toannexb out.ts

But as I mentioned before, I need to implement it with ffmpeg API in C.

So, although I am able to generate a playable .ts file, its video and audio streams are not synced. That is, playing them back on Android tablet plays the video very slowly while audio is playing at normal speed and then (once audio stream ends) video plays at normal speed to the end. Playing the same generated .ts file in VLC produces a very condensed audio (as though fast-forwarded) and plays video fine.

There are still many aspects of media conversion that I am not familiar with. I am sure that some of them prevent me from successful conversion.

Here is some information (via ffprobe) about the files: - in.mp4 - file generated via Android recording - MPEG4 (H.264 + AAC) - ffmpeg.ts - file generated via ffmpeg conversion - MPEG2TS (H.264 + AAC) - out.ts - file generated via my code - MPEGTS (H.264 + AAC)

in.mp4

filename=in.mp4
nb_streams=2
format_name=mov,mp4,m4a,3gp,3g2,mj2
format_long_name=QuickTime/MPEG-4/Motion JPEG 2000 format
start_time=0:00:00.000000
duration=0:00:09.961383
size=4.730 Mibyte
bit_rate=3.983 Mbit/s
TAG:major_brand=isom
TAG:minor_version=0
TAG:compatible_brands=isom3gp4
TAG:creation_time=2013-05-28 17:06:57

ffmpeg.ts

filename=ffmpeg.ts
nb_streams=2
format_name=mpegts
format_long_name=MPEG-2 transport stream format
start_time=0:00:01.400000
duration=0:00:09.741267
size=5.132 Mibyte
bit_rate=4.419 Mbit/s

out.ts

filename=out.ts
nb_streams=2
format_name=mpegts
format_long_name=MPEG-2 transport stream format
start_time=0:00:00.000000
duration=0:00:09.741267
size=5.166 Mibyte
bit_rate=4.449 Mbit/s

Firstly, I was unable to affect my output file's start_time. Next, upon examining the -show_packets output of probe, I saw the following:

ffmpeg.ts

[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=0
duration_time=0:00:00.000000
size=20.506 Kibyte
pos=564
flags=K
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=0
duration_time=0:00:00.000000
size=11.727 Kibyte
pos=22936
flags=_
[/PACKET]
...
[PACKET]
codec_type=audio
stream_index=1
pts=126000
pts_time=0:00:01.400000
dts=126000
dts_time=0:00:01.400000
duration=2089
duration_time=0:00:00.023211
size=285.000 byte
pos=109416
flags=K
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=128089
pts_time=0:00:01.423211
dts=128089
dts_time=0:00:01.423211
duration=2089
duration_time=0:00:00.023211
size=374.000 byte
pos=-1
flags=K
[/PACKET]
...
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=0
duration_time=0:00:00.000000
size=20.000 Kibyte
pos=87232
flags=_
[/PACKET]
[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=0
duration_time=0:00:00.000000
size=16.852 Kibyte
pos=112800
flags=_
[/PACKET]

out.ts

[PACKET]
codec_type=audio
stream_index=1
pts=0
pts_time=0:00:00.000000
dts=0
dts_time=0:00:00.000000
duration=2089
duration_time=0:00:00.023211
size=285.000 byte
pos=22936
flags=K
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=1024
pts_time=0:00:00.011378
dts=1024
dts_time=0:00:00.011378
duration=2089
duration_time=0:00:00.023211
size=374.000 byte
pos=23312
flags=K
[/PACKET]
...

[PACKET]
codec_type=video
stream_index=0
pts=N/A
pts_time=N/A
dts=N/A
dts_time=N/A
duration=0
duration_time=0:00:00.000000
size=11.727 Kibyte
pos=25004
flags=_
[/PACKET]
[PACKET]
codec_type=audio
stream_index=1
pts=7168
pts_time=0:00:00.079644
dts=7168
dts_time=0:00:00.079644
duration=2089
duration_time=0:00:00.023211
size=299.000 byte
pos=55460
flags=K
[/PACKET]

As you can see, ffmpeg.ts starts out with video packets that do not have pts/dts. The audio packets that follow contain pts/dts. This repeats until the end. All video packets do not have pts/dts according to ffprobe output.

However, out.ts starts with audio packets and alternate with video packets. Here, video packets also do not have pts/dts. The difference is that here there is one video packet between a series of audio packets. What happened to the rest of the video packets (ffmpeg.ts has ~5 audio followed by ~5 video).

Obviously, I'm still learning and don't know way too much yet... Does anything jump out as obvious a problem to anyone? I will greatly appreciate any info / suggestions but will continue to grind at it!!

Answer 1

Ok, as I suspected, it was obvious but not to the newbie. Basically, even if you do not have to re-encode packets, the streams in the original container come with their own time base per stream. This needs to be converted or "rescaled" to the new output stream's time base.

packet.pts = av_rescale_q(packet->pts, inStream->time_base, outStream->time_base);
packet.dts = av_rescale_q(packet->dts, inStream->time_base, outStream->time_base);

This will properly rescale packets for each stream. Of course, this is probably the most basic action that can be taken and additional alternations / rescalings may need to be performed (is a packet doesn't have a dts / pts). However, I am in control of the input file so this should suffice for me.