On the sample grabber callback you receive data time stamped. If you keep the time stamps you get, and you make sure you have the data playback sync'ed using these time stamps in terms of playing data with equal time stamps on all streams at any time - then you will have your synchronization in good standing.
Sample grabbers receive all data, but it does not mean this data reaches grabbers in synchronized order. Matching time stamps is absolutely necessary. Because video might be temporally compressed and audio is typically not, your seeking might be getting you additional preroll data on the video leg, which you might be not processing correctly.