H264 with multiple PPS and SPS

Question

I have a card that produces a H264 stream with a SPS (Sequence Parameter Set) and a PPS (Picture Parameter Set), in that order, directly before each I-Frame. I see that most H264 streams contain a PPS and SPS at the first I-Frame.

Is this recommended? Do decoders/muxers typically support multiple PPS and SRS?

Answer 1

H.264 comes in a variety of stream formats. One variation is called "Annex B".

(AUD)(SPS)(PPS)(I-Slice)(PPS)(P-Slice)(PPS)(P-Slice) ... (AUD)(SPS)(PPS)(I-Slice).

Typically you see SPS/PPS before each I frame and PPS before other slices.

Most decoders/muxers are happy with "Annex B" and the repetition of SPS/PPS.

Most decoders/muxers won't do anything meaningful if you change the format and SPS/PPS midstream.

Most decoders/muxers parse the first SPS/PPS as part of a setup process and ignore subsequent SPSs.

Some decoders/muxers prefer H.264 without the (AUD), start codes and SPS/PPS. Then you have to feed SPS/PPS out of band as part of setting up the decoders/muxers.

Answer 2

An IDR frame, or an I-slice can not be decoded without a SPS and PPS. In the case of a container like mp4, the SPS and PPS is stored away from the video data in the file header. Upon playback the mp4 is parsed, the SPS/PPS is used to configure the AVC decoder once, then video can be played back starting at any IDR/I-slice.

There is a second scenario, Live video. With live video, there is no file header, because there is no file. So when a TV tunes into a channel, where does it get the SPS/PPS? Because television is broadcast, meaning the television has no way to request the SPS/PPS, it is repeated in the stream.

So when you start encoding video, your encoder does not know what you intend to do with the video. Now if the extra SPS/PPS show up in an mp4, the decoder just ignores them, but if you are streaming to a TV, without them the stream would never play. So most of the default to repeating SPS/PPS just in case.

Answer 3

I know about matroska(mkv) spec so here SPS and PPS are stored only once as codec private data section. So they are not repeating with every i frame or IDR frame.

If your h264 stream's each i frame/IDR frame has SPS/PPS then matroska muxer will store only 1 copy in codec private data.

So while storing usecase based container format suggest to use only one copy of SPS/PPS but broadcasting and streaming based container formats suggest send SPS/PPS before every iFrame/IDR frame or whenever any codec changes change in h264 stream at that time