Gsteamer rtp video mixer, found a working pipeline, however need improvement

Question

I'm attempting to mix multiple rtp h264 payload video streams into a single video stream of 15FPS.

A working pipeline that mixes two video streams over a videotestsource pattern of 15FPS

VIDEO_CAPS="application/x-rtp,media=(string)video,clock-rate=(int)90000,encoding-name=(string)H264"

gst-launch -vvvve videomixer2 name=mix ! ffmpegcolorspace ! xvimagesink  

udpsrc caps=$VIDEO_CAPS port=3030 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox top=0 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix.  

udpsrc caps=$VIDEO_CAPS port=6666 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox top=0 left=-178 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix. 

videotestsrc ! video/x-raw-yuv, framerate=15/1, width=640, height=360 ! mix.

Above pipeline has a strange issue, When attempting to mix, the first video source goes blank, (i'm certain that there is still streaming going on for the first video source, however it doesn't turn up. Rather second video stream is shown. The verbose is given below.

Verbose for the above pipeline

/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_0: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, color-matrix=(string)sdtv
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession0.GstPad:recv_rtp_sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_sink_0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_sink_0.GstProxyPad:proxypad0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession0.GstPad:recv_rtp_src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpSsrcDemux:rtpssrcdemux0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpJitterBuffer:rtpjitterbuffer0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSession:rtpsession1.GstPad:recv_rtp_sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_sink_0: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_sink_0.GstProxyPad:proxypad1: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSession:rtpsession1.GstPad:recv_rtp_src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpSsrcDemux:rtpssrcdemux1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpJitterBuffer:rtpjitterbuffer1.GstPad:src: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpJitterBuffer:rtpjitterbuffer1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpBin:rtpbin0/GstRtpPtDemux:rtpptdemux0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/GstRtpBin:rtpbin0.GstGhostPad:recv_rtp_src_0_4186542290_99.GstProxyPad:proxypad2: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2640.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpBin:rtpbin1/GstRtpPtDemux:rtpptdemux1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay1.GstPad:src: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/GstRtpH264Depay:rtph264depay1.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/GstRtpBin:rtpbin1.GstGhostPad:recv_rtp_src_0_4186622237_99.GstProxyPad:proxypad3: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)99
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2641.GstPad:sink: caps = video/x-h264, stream-format=(string)byte-stream, alignment=(string)nal
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2640.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale0.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox0.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp1.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp1.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_1: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)176, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/ffdec_h264:ffdec_h2641.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale1.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoScale:videoscale1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:src: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter3.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox1.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoBox:videobox1.GstPad:sink: caps = video/x-raw-yuv, width=(int)176, height=(int)144, framerate=(fraction)15/1, format=(fourcc)I420, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter4.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstCapsFilter:capsfilter4.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp2.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp2.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstVideoMixer2:mix.GstVideoMixer2Pad:sink_2: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)354, height=(int)144, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:src: caps = video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstFFMpegCsp:ffmpegcsp0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)AYUV, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw-yuv, format=(fourcc)YUY2, width=(int)640, height=(int)360, framerate=(fraction)15/1, pixel-aspect-ratio=(fraction)1/1
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2875): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
There may be a timestamping problem, or this computer is too slow.

I highly suspect it has something to with videobox.

Answer 1

(Answered in a question edit. Converted to a community wiki answer. See What is the appropriate action when the answer to a question is added to the question itself? )

The OP wrote:

As I said this issue was with videobox, we have to add videobox border-alpha=0 or alpha channel.

Fully functional pipeline : Accepts 3 RTP h264 videos and mixes over a video test pattern.

gst-launch -vvvve videomixer2 name=mix ! ffmpegcolorspace ! xvimagesink sync=false  udpsrc caps=$VIDEO_CAPS port=3030 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=0 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix.  udpsrc caps=$VIDEO_CAPS port=6666 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=0 left=-176 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! ffmpegcolorspace !  mix. udpsrc caps=$VIDEO_CAPS port=2222 ! .recv_rtp_sink_0 gstrtpbin ! rtph264depay ! ffdec_h264  ! videoscale ! video/x-raw-yuv , width=176, height=144 ! videobox border-alpha=0 top=-144 left=0 ! video/x-raw-yuv,format=\(fourcc\)AYUV ! mix. videotestsrc ! video/x-raw-yuv, framerate=15/1, width=352, height=288 ! mix.