Render image from audio

Question

Is there a possibility to render an visualization of an audio file?

Maybe with SoundManager2 / Canvas / HTML5 Audio? Do you know some technics?

I want to create something like this:

Answer 1

This isn't possible yet except by fetching the audio as binary data and unpacking the MP3 (not JavaScript's forte), or maybe by using Java or Flash to extract the bits of information you need (it seems possible but it also seems like more headache than I personally would want to take on).

But you might be interested in Dave Humphrey's audio experiments, which include some cool visualization stuff. He's doing this by making modifications to the browser source code and recompiling it, so this is obviously not a realistic solution for you. But those experiments could lead to new features being added to the <audio> element in the future.

Answer 2

You have a tone of samples and tutorials here : http://www.html5rocks.com/en/tutorials/#webaudio

For the moment it work in the last Chrome and the last last Firefox (Opera ?).

Demos : http://www.chromeexperiments.com/tag/audio/

To do it now, for all visitors of a web site, you can check SoundManagerV2.js who pass through a flash "proxy" to access audio data http://www.schillmania.com/projects/soundmanager2/demo/api/ (They already work on the HTML5 audio engine, to release it as soon as majors browsers implement it)

Up to you for drawing in a canvas 3 differents audio data : WaveForm, Equalizer and Peak.

soundManager.defaultOptions.whileplaying = function() { // AUDIO analyzer !!!
    $document.trigger({ // DISPATCH ALL DATA RELATIVE TO AUDIO STREAM // AUDIO ANALYZER 
        type             : 'musicLoader:whileplaying',
        sound            : {
            position         : this.position,          // In milliseconds
            duration         : this.duration,
            waveformDataLeft : this.waveformData.left, // Array of 256 floating-point (three decimal place) values from -1 to 1
            waveformDataRight: this.waveformData.right,
            eqDataLeft       : this.eqData.left,       // Containing two arrays of 256 floating-point (three decimal place) values from 0 to 1
            eqDataRight      : this.eqData.right,      // ... , the result of an FFT on the waveform data. Can be used to draw a spectrum (frequency range)
            peakDataLeft     : this.peakData.left,     // Floating-point values ranging from 0 to 1, indicating "peak" (volume) level
            peakDataRight    : this.peakData.right
        }
    });
};

With HTML5 you can get :

var freqByteData = new Uint8Array(analyser.frequencyBinCount);
var timeByteData = new Uint8Array(analyser.frequencyBinCount);
function onaudioprocess() {
    analyser.getByteFrequencyData(freqByteData);
    analyser.getByteTimeDomainData(timeByteData);
    /* draw your canvas */
}

Time to work ! ;)

Answer 3

Run samples through an FFT, and then display the energy within a given range of frequencies as the height of the graph at a given point. You'll normally want the frequency ranges going from around 20 Hz at the left to roughly the sampling rate/2 at the right (or 20 KHz if the sampling rate exceeds 40 KHz).

I'm not so sure about doing this in JavaScript though. Don't get me wrong: JavaScript is perfectly capable of implementing an FFT -- but I'm not at all sure about doing it in real time. OTOH, for user viewing, you can get by with around 5-10 updates per second, which is likely to be a considerably easier target to reach. For example, 20 ms of samples updated every 200 ms might be halfway reasonable to hope for, though I certainly can't guarantee that you'll be able to keep up with that.

Answer 4

http://ajaxian.com/archives/amazing-audio-sampling-in-javascript-with-firefox

Check out the source code to see how they're visualizing the audio

Answer 5

For this you would need to do a Fourier transform (look for FFT) which will be slow in javascript, and not possible in realtime at present.

If you really want to do this in the browser, I would suggest doing it in java/silverlight, since they deliver the fastest number crunching speed in the browser.