I have been using extensively live555 this year , starting from scratch.
I truly think that one of the best place to read some working code for several types of stream (video , Audio ) and several encoders is in the test programs folder that you can find here
Then it is really easy when you start understanding how this library works.
Typically , if you want to insert a decoder you are on a client.
You must know that all the data received (actual data , not the RTSP commands) ends up in the "sink"
You can then subclass this sink (several example are presents in the link that i gave you) and implement your decoder here.
Take a look at the video decoders example in testRTSPClient for example , i know it is for video but it is some self explaining code , you should have a good starting point there.