If you request a long buffer, then you have to wait for the OS to fill it at the current sample rate. If you request a sample rate other than what the hardware ADC is running, then you have to additionally wait for the resampler filter delay. Different Android devices and OS versions may support different minimum buffer sizes and native hardware sample rates.
One technique to hide the latency is to start recording earlier in the app's life cycle, and just keep throwing away audio samples until the app needs them. Then there is no startup overhead.
Added: On some devices/OS versions, the data may really be captured into longer OS driver buffers at some hardware sample rate (for instance 4096 at 44.1k or 48kHz) and only after a couple of those buffers are filled, converted to another sample rate, and the chopped into the shorter requested buffer lengths, with the audio command start sending data to an app. To bypass, if even possible, you might need to mod the OS and write your own ADC driver. But try using a higher sample rate (44.1k or 48k) and request shorter buffers first.