How to handle audio streams and mixing for VoIP

Question

We're working on a Java based Voice over IP program, and we need to find an effective way of keeping streams and server side stress to a reasonable amount. We're writing this for a multiplayer game, and we plan to have a few different modes for determining how to mix and send out audio. We plan on having it "Opt In" to keep useless streams out of the equation, if they choose not to participate in voice chat. Aside from that, we're thinking channels, private chat (2 people), and proximity. If we were using a channel with 16+ people in it, we want to find if there is a way to avoid the x^2 number of streams (256 in this case). We also need to try to keep client side work to a reasonable level, as it will be running a game. We're not too sure how much work a large server could handle, as it will be exponential as more people are in a channel. We may need to limit the amount of people per channel, or allow the server owner to do that, as well as limit the number of channels. This game can have around 40-500 people on a server at once, depending on the popularity, and we're unsure of how to handle that type of stress on a server's processing and bandwidth.

Essentially, we're asking if anyone has any knowledge of existing systems that have an efficient way of handling this. We're using JSpeex for our audio encoding if that's of any relevance. So, are there any methods out there for handling this, or possibly even some ideas from the community? We also plan on re-implementing this in a smaller scale Skype like program that we've been developing.

Answer 1

I don't see why you would need 256 streams. You shouldn't be going direct from client to client.

For each client you need a single two way connection to the server. All input/output audio is directed through the server. The server than multiplexes all the audio a player is to receive into a single audio stream and forwards it along.

When you are in a private chat, the server simply only forwards packets from the other player in the private chat with you.

When using proximity, the server multiplexes audio from any player in range into a single stream.

I would assume any decent audio encoding package would have support for multiplexing audio together from multiple channels, but I don't know anything about JSpeex.