play audio in real-time using opensl es and the android ndk

Question

What I would like to do is record a voice, process it like adding some effect to the recorded audio, and then replay it as fast as possible using opensl. I'm having trouble finding information on the net about doing something like this, so I actually have a couple questions.

I've looked at the android native audio sample, and the book beginning android ndk, which got me started with opensl, but they didn't really help me understand recording very well.

1) So first, to play back recorded audio immediately, would i be able to read a buffer from a thread that plays the audio, while the same time the recording thread is writing to the buffer? I know someone will probably go crazy about this idea because the idea of accessing some object in memory by two different threads at the same time could lead to problems, but if the recording thread is made sure to always be writing to memory just ahead of the playing audio thread, then that would be possible right?

2) Or to play back in real time, will i use the buffer queue with two or three very small buffers, and a callback that gets called each time one is filled, then play that buffer while the next buffer is being filled by the recording audio thread? but i was reading and someone said the callbacks are not always called (the most helpful link i've found about this is: https://groups.google.com/forum/#!msg/android-ndk/hLSygQrmcPI/qtwB76JNa_EJ). also, that means that the time difference between the recorded audio and the played back audio would be the size of the buffer plus the time it takes for the callback to let the recording object know to start recording using the next buffer. I think there would be a gap between when the recorder stops recording and when it starts recording again.

3) The android recording buffer queue is also where i'm having problems understanding. Do i have to use a buffer queue in android to record audio? or is it possible to record directly to a buffer without using a buffer queue? I'm having problems with this because the clear method of SLAndroidSimpleBufferQueueItf doesn't actually seem to work. apparently its a bug. and I can't seem to record over a buffer after its already been filled with data. since the clear method doesn't seem to work, how can i tell the recorder which buffer in the queue to record to?

I know this is kind of a lot to ask, and i didn't say everything i've tried, but i was hoping someone has some experience with this and can shed some light for me and anybody else that has a problem playing back recorded audio in real-time using opensl.

Answer 1

I found pretty much exactly what i'm looking for. A way to play recorded audio back in real time is to use something called a circular buffer (or round buffer), where basically the recorded audio is read into it, while the played back audio reads everything up to where the recorded audio is writing.

http://audioprograming.wordpress.com/

OpenSL ES in Android still does not exactly support "Low-latency", as there is still a slight delay, but it still works really well, it just sounds more like a slight echo

Answer 2

I realize this has been answered and it's an old question. However, to clarify, on my playback queue, I don't use Clear(). I simply stop the playback and then wait for the queue to drain by using GetState(state) in a loop, like so:

SLresult res;
SLAndroidSimpleBufferQueueState state = {0};
res = (*playbackBufferQueue)->GetState(playbackBufferQueue, &state);

while (state.count && (res == SL_RESULT_SUCCESS))  
{  
    prevState = state;
    res = (*m_playbackBufferQueue)->GetState(m_playbackBufferQueue, &state); 
    // Otherwise this will peg the CPU
    sleep(1);
}

And to support low latency you have use the "optimal" rate and sample size, which you query from Android. Can you do this using native API calls? Of course not, don't be silly. Do it in Java and pass it back to native:

int getBestPlaybackRate() {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN_MR1) {
        AudioManager audioManager = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
        return Integer.parseInt(audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_SAMPLE_RATE));
    }
    return 0;
}
int getBestBufferSize() {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN_MR1) {
        AudioManager audioManager = (AudioManager) context.getSystemService(Context.AUDIO_SERVICE);
        return Integer.parseInt(audioManager.getProperty(AudioManager.PROPERTY_OUTPUT_FRAMES_PER_BUFFER));
    }
    return 0;
}