For audio analysis I just use a combination of the generalized data analysis tools. Here are the tools I use for audio:
reading and writing .wav files:
Python wave module, and scipy.io.wave
These work for most .wav files, but not the very unusual formats
FFTs:
numpy rfft and friends (rfftfreq is particularly useful with rfft)
spectrograms and more general graphs:
matplotlib and, in particular matplotlib's specgram