The problem is that the phone spends a lot of time opening the file. Unfortunately the App Inventor is very simple and it doesn´t have a lot of other media tools. You can put a progress bar while the file is charging, but there aren´t many solutions...
Remind that the problem should be in the process of download the file instead of the phone. If you put the audio file in the app resources, it will take you less time.
If you could make the app with Eclipse, you would have more options.
Hope it´s useful