Audio data preprocessing using <audio> tag and Javascript

Question

Hi I'm currently working on an audio waveform editor written in pure HTML5 w/ Javascript.

I've made some progress using the <audio>.MozAudioAvailable event from Mozilla, to get the data in each frame and draw them on a canvas. However with MozAudioAvailable I can only get the frame which it is NOW playing.

As a waveform editor, my program have to seek and preprocess the data about several seconds PRIOR TO the currently playing, i.e. when playing 00:05:00, my program should probably be showing the waveform from 00:04:50 to 00:05:10, so I have to preprocess the data between 00:05:00 and 00:05:10 before they got played.

I've searched the Internet to get a solution (not restricted to the Mozilla approaches, Chrome or Opera ones are also accepted), but got no answer. The preload property and onprogress event doesn't help. Now I'm trying to make another <audio> tag which plays the same music as the original one, but several seconds faster to get the data in advance. However the solution is quite dirty as you seen.

I'm wondering if HTML5 group is working on some more flexible ways to deal with the multimedia objects, or if some browser development team is working on this. If you have any idea or experience on this topic, plz give me some instructions. Thx.

UPDATE:

Maybe I have not described my question clearly. Below is a picture taken from Audacity which may demonstrate my target.

The vertical line at about 1:55.10 indicates the frame currently playing. For frames to the left of the line, I can use the historical frames saved by my program. But for frames to the right of the line, which have not yet been played, I'm not able to get them before they've been played.

An ugly solution may be to add another <audio> tag which plays faster than the original one (should be playing 1:55.90 in the screenshot) so that I can get the frames to the right of the vertical line. But that's ugly, and not easy to implement, isn't it?

Answer 1

From https://wiki.mozilla.org/Audio_Data_API

The data you're looking for is in the framebuffer property of the event object passed to your listener function

To access a particular section of historical data, simply cache previous framebuffer captures yourself

var channels,
    rate,
    frameBufferLength,
    samples;

function audioInfo() {
  var audio = document.getElementById('audio');

  // After loadedmetadata event, following media element attributes are known:
  channels          = audio.mozChannels;
  rate              = audio.mozSampleRate;
  frameBufferLength = audio.mozFrameBufferLength;
}

function audioAvailable(event) {
  var samples = event.frameBuffer;
  var time    = event.time;

  for (var i = 0; i < frameBufferLength; i++) {
    // Do something with the audio data as it is played.
    processSample(samples[i], channels, rate);
  }
}

ADDENDUM:

OK, so you need your program to look into the future. As far as I know, this isn't doable unless you can use some kind of clever preload solution (although I suspect this won't work either - perhaps a subscription to the moz mailing list may enable you to request this as a future feature).

I still don't know what exactly it is you're trying to do, but I have an HTML5 player that uses a drawn waveform background and draws an oscilloscope during playback using RAW audio data which I rip from the audio file myself on the server - I don't use the mozilla API because I want it to work in all ogg-supporting browsers, so I load a separate data blob altogether. This of course means a) it is not pure HTML5 and Javascript, and b) my player only plays files that originate from my server. In any case, the stages are:

1) Use a command-line utility (I recommend sox) to extract the RAW data - I downmix it to mono, 8-bit 1khz PCM which is relatively small yet high enough resolution to be useful (1kb/sec). If you want to keep it (to draw an oscilloscope during playback in non-mozilla browsers) cache it in a database as a blob.

2) Use the RAW PCM to draw a PNG of the wave using php_gd, and cache that (for your player background)

3) if using the RAW data then you can load it into javascript as a base64-encoded string

It is not impossible for you to do this even with external audio files if you use your server simply as a proxy to extract the RAW data, but then you will need to take care of the security issues that introduces.

If this applies to you at all then let me know and I'll post some source code for these. The program in question will be released as an open source project at http://jukenix.org as soon as I've tidied it up a bit.

P.S. screenshot of this in action is at bottom of page of link above