Transmitting additional data with h.264 frames using RTSP over TCP

Question

In one of my applications i am trying to implement sending and receiving of H.264 frames encapsulated in an object which has other data types in it as well. The requirement is to transmit these frames using RTSP over tcp. On the receiving side these frames need to be extracted along with the additional data and displyed. I have spent considerable time on trying to make this as confirming to the standards as possible but i dont seem to find any specific payload format which talks about passing custom objects in an RTSP session. Is using NAL Units my best bet or is there a better solution to this.

Answer 1

SEI NAL Unit is one of the placeholders for the data, where you embed data directly into H.264 stream. Other than that you can send your custom data as an alternate data stream of custom type where you define the payload format yourself.

Here is the response from one of the RTSP servers which is capable of streaming meta data apart from video and audio (two lines at the bottom). Note that rtpmap types 96-127 are "dynamic types" and are suitable for custom formats.

DESCRIBE rtsp://192.168.0.55/data RTSP/1.0
CSeq: 1
Accept: application/sdp
RTSP/1.0 200 OK
CSeq: 1
Date: Fri, Aug 03 2012 07:13:15 GMT
Content-Base: rtsp://192.168.0.55/data/
Content-Type: application/sdp
Content-Length: 629

v=0
o=- 1343377422075556 1 IN IP4 192.168.0.55
s=Session of first stream
i=First Codec Stream
t=0 0
a=tool:LIVE555 Streaming Media v2007.08.03
a=type:broadcast
a=control:*
a=range:npt=0-
a=x-qt-text-nam:Session of first stream
a=x-qt-text-inf:First Codec Stream
m=video 0 RTP/AVP 96
c=IN IP4 0.0.0.0
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=428028;sprop-parameter-sets=Z0KAKIyNQFoJIg==,aM48gA==
a=control:track1
m=audio 0 RTP/AVP 97
c=IN IP4 0.0.0.0
a=rtpmap:97 PCMU/16000
a=control:track2
m=metadata 0 RTP/AVP 98
c=IN IP4 0.0.0.0
a=rtpmap:98 METADATA/64000
a=control:track3