Consider driver's latency. There's hidden function AudioManager.getOutputLatency(int) to get this.
Call it like this:
AudioManager am = (AudioManager)getSystemService(Context.AUDIO_SERVICE);
try{
Method m = am.getClass().getMethod("getOutputLatency", int.class);
latency = (Integer)m.invoke(am, AudioManager.STREAM_MUSIC);
}catch(Exception e){
}
I get about 45 - 50 ms on different devices. Use the result in your calculations.