Why do I get these artifacts when using Microsoft MPEG-TS Demultiplexer and H.264 decoder on a H.264 MPEG-TS stream?

Question

I have written a live push filter that gets an mpeg-ts stream containing h.264 video and aac audio. I set up a directshow pipeline and configure the output pins. I can render the h.264 stream but I get artifacts in the rendering as can be seen from this screenshot when streaming from gstreamer using a videotestsrc and the "ball" pattern.This screenshot should contain only one white dot on a black background. The two additional are "leftovers" that appear when the animation plays.

If I stream MPEG-2 and change the pipeline accordingly, the pattern renders without error. I have tried to use settings described on msdn to configure the pin, both using H264 and AVC1 explicitly providing the sequence header and so on. I still get the same kind of artifacts.

One interesting thing is that the artifacts mostly appear at the same frequency as the I-Frames arrive, and if we only send I-Frames (key-int-max=1), the artifacts disappear completely.

Also, the errors seem to appear at the top half of the image when the I-Frame interval is 60, i.e. every 2 seconds. When we change to one I-Frame every second frame (key-int-max=2), the artifacts appear only in a narrow strip at the top of the image.

The following gstreamer pipeline produced the video stream:

videotestsrc live-source=true pattern=ball ! video/x-raw-yuv,format=(fourcc)I420,width=1366,height=768,framerate=30/1 ! timeoverlay halign=left valign=bottom shaded-background=true ! x264enc bitrate=4096 tune=zerolatency ! h264parse ! queue ! mux. audiotestsrc wave=ticks volume=0.2 ! voaacenc ! mux. mpegtsmux name=mux ! udpsink host=<ip> port=<port>

This is what the pipeline looks like:

The configuration in this example is majortype = MEDIATYPE_Video, subtype = MEDIASUBTYPE_H264, formattype = FORMAT_MPEG2Video. No sequenceheader specifically provided, etc.

So the question is, are these kinds of artifacts symptoms of some common configuration problem?

Answer 1

It turns out that the MPEG-2 Demultiplexer is not designed to handle H.264 content. That is why these effects appear.

Answer 2

You are loosing data in your transmission possibly leading to errors. The issue is that I pictures have a higher bitrate than P pictures. If you have a cbr n/w then you can potentially see this issue.

Why the artifacts appear this way:

Now if you had a video with only 1 white ball and rest of background is black, when you loose frames the reference pictures are lost and it will appear garbled because a decoder may simply try to use frames from the last valid frame to display. Since your last valid frame is all black rest of the screen looks ok, the part where the while ball is still shows up with errors.

Replace this pattern with a different one and you will see more clearly what I mean.

With I pictures only there is no question of any reference pictures and hence you will get clean output.

One way to check if transmission is ok is dump your output to a file and read it on the other side with a file. If that works fine you know your transmission is loosing data.

Also since you have the clock at the bottom you should see the clock also jumping if there is loss in frames.