This will be difficult, especially with a large number of users. Since you only need to send audio in one direction, that will at least be a bit easier.
First, you're going to want to put that AP in the middle of the crowd with an omnidirectional antenna. Perhaps, in a backpack or something. Each phone on that network needs to "hear" when other phones are transmitting, or it will be a mess. Even though your application is one-way, 802.11 isn't.
Now, when you write your application, use UDP packets sent to the broadcast address. No need for TCP packets, as they will clog up your network anyway.
Use a simple voice codec, such as AMR. The codecs available vary from platform to platform. See this document for a list on Android: http://developer.android.com/guide/appendix/media-formats.html
Honestly, the easiest solution would be to go buy a small FM transmitter, since many phones have receivers in them anyway.