Need an audio analysis library to create real time feedback from audio file?

Question

Real-time is not necessarily required, however I am creating a game for my final year project and I wish to use the power of audio to create dynamic levels based solely on a music track that is playing. I aim to create this game for the PS Vita using playstation mobile and C#, but if i want i can switch to C++ and PSP.

I can use a WAV file, and hopefully extract the amplitude of the waveform, as well as calculating other characteristics like average frequency and approximate BPM from this data to create a level.

I have no qualms about trying to work with this raw data, I just want to know a way I can actually GET that information first. If i can extract the samples and assertain different characteristics of these samples, I can store them and work out changes in loudness, pitch and more to create notes etc.

I am using C#, but if at all possible i can either use p/invoke or switch my project to another device that uses C++ instead of C#.

I'm panicking a bit here, cos I really am a bit stumped.

Many thanks guys.

Answer 1

The problem you describe here is one of music/audio feature extraction and a substantial body of academic work exists that you can draw on. Another useful term of art with which to search is Music Information Retrieval (MIR).

The list of 'features' that researchers have attempted to retrieve from recordings is large and varied, from deterministic things such as pitch and key through emotional characteristics, such as 'energy'.

Most of these turn out to be more difficult than you might imagine, and typically only about 60-70% accurate - although for your requirements, this is probably adequate.

A good entry point might be download Sonic Visualiser, for which a large number of feature extraction plug-ins exist, and are open-source. You'll at least get a feel for what's possible.

Update: Another useful term of art is Onset detection - this is typically used to describe beat detection algorithms.

Answer 2

Unfortunately i don't think you'll be able to use C# to do this - AFAIK, there is no JIT compiler for it. I remember reading about something for Mono, which would make it available to use with C#, but i'm not sure right now.

That said - i would go with c++. If you go that way, you can make use of a vast amount of audio analysis libraries, like CLAM (http://clam-project.org/).

Don't panic (imagine big, friendly letters.) Envision the necessary parts for the project step by step, tackle one by one, and you'll be done in no time. =)

Answer 3

Aubio is a C/C++ library that does pitch tracking, onset detection and bpm tracking, among other things.

As for "extracting the amplitude of the waveform", the waveform is amplitude, i.e., you could just pick the audio sample with the greatest absolute value every n samples and use that value to do the "amplitude" part of the visualization.

Here's some code that might help you get started reading WAVE data in C#.

Here's some information about writing a C# wrapper for the FFTW library.