According to RFC 2250 M bit should indicate the end of frame in case of mpeg-ts. (3.3 RTP Fixed Header for MPEG ES encapsulation) but many decoder may not be putting it in header.
only other way to find the start of frame is to decode the header of 188 byte mpeg-ts packet.mpeg-ts contains "Payload Unit Start Indicator".
so your algo will be like
- RTP data contain integer number of mpeg-ts packets.
- each packet starts with 0x47
- check the "payload unit start indicator" fiels for each packet
- if "payload unit start indicator == 1" check the if PES or PSI
- ignore packet if PSI and continue with step-1, else go to next step
- for PES packet check "Stream id" if its video you hit a new frame.