What you're trying to do is Start Codes Extraction.
If you have a raw Transport Stream with MPEG-2 video:
- Look for packets which have the PayloadUnitStartIndicator bit set in the TS header, this indicates the start of a PES packet.
You need to look at the 4th byte of such PES packet to determine the StreamId. The PES Start Code looks like this
0x00 0x00 0x01 0xXY
The PES Start Code for a packet with a Picture header is
0x00 0x00 0x01 0x00
For H.264 I believe you would be interested in packets which have the Sequence Parameter Set (SPS) and/or the Picture Parameter Set (PPS) NAL units instead of the picture header.