How to process streaming HTTP GET data?

Question

Right now, I have a node.js server that's able to stream data on GET request, using the stream API. The GET request is Transfer-encoded set to 'chunked'. The data can be on the order of 10 to 30 MBs. (They are sometimes 3D models)

On the browser side, I wish to be able to process the data as I'm downloading it--I wish to be able to display the data on Canvas as I'm downloading it. So you can see the 3D model appear, face by face, as the data is coming in. I don't need duplex communication, and I don't need a persistent connection. But I do need to process the data as soon as it's downloaded, rather than waiting for the entire file to finish downloading. Then after the browser downloads the data, I can close the connection.

How do I do this?

JQuery ajax only calls back when all the data has been received.

I also looked at portal.js (which was jquery-streaming) and socket.io, but they seem to assume persistent reconnection.

So far, I was able to hack a solution using raw XMLHttpRequest, and making a callback when readyStead >= 2 && status == 200, and keeping track of place last read. However, that keeps all the data downloaded in the raw XMLHttpRequest, which I don't want.

There seems to be a better way to do this, but I'm not sure what it is. Any one have suggestions?

Answer 1

So I found the answer, and it's Server-sent events. It basically enables one-way http-streams that the browser can handle a chunk at a time. It can be a little tricky because some existing stream libs are broken (they don't assume you have \n in your stream, and hence you get partial data), or have little documentation. But it's not hard to roll your own (once you figure it out).

You can define your sse_transform like this:

// file sse_stream.coffee
var Transform = require('stream').Transform;
var util = require('util');
util.inherits(SSEStream, Transform);

function SSEStream(option) {
  Transform.call(this, option);
  this.id = 0;
  this.retry = (option && option.retry) || 0;
}

SSEStream.prototype._transform = function(chunk, encoding, cb) {
  var data = chunk.toString();
  if (data) {
    this.push("id:" + this.id + "\n" +
      data.split("\n").map(function (e) {
        return "data:" + e
      }).join("\n") + "\n\n");
    //"retry: " + this.retry);
  }
  this.id++;
  cb();
};

SSEStream.prototype._flush = function(next) {
  this.push("event: end\n" + "data: end" + "\n\n");
  next();
}

module.exports = SSEStream;

Then on the server side (I was using express), you can do something like this:

sse_stream = require('sse_stream')
app.get '/blob', (req, res, next) ->
  sse = new sse_stream()

  # It may differ here for you, but this is just a stream source.
  blobStream = repo.git.streamcmd("cat-file", { p: true }, [blob.id])

  if (req.headers["accept"] is "text/event-stream")
    res.type('text/event-stream')
    blobStream.on("end", () -> res.removeAllListeners()).stdout
      .pipe(
        sse.on("end", () -> res.end())
      ).pipe(res)
  else
    blobStream.stdout.pipe(res)

Then on the browser side, you can do:

    source = new EventSource("/blob")
    source.addEventListener('open', (event) ->
      console.log "On open..."
    , false)
    source.addEventListener('message', (event) ->
      processData(event.data)
    , false)
    source.addEventListener('end', (event) ->
      console.log "On end"
      source.close()
    , false)
    source.addEventListener('error', (event) ->
      console.log "On Error"
      if event.currentTarget.readyState == EventSource.CLOSED
        console.log "Connection was closed"
      source.close()
    , false)

Notice that you need to listen for the event 'end', that is sent from the server in the transform stream's _flush() method. Otherwise, EventSource in the browser is just going to request the same file over and over again.

Note that you can use libraries on the server side to generate SSE. On the browser side, you can use portal.js to handle SSE. I just spelt things out, so you can see how things would work.

Answer 2

oboe.js is a library for streaming responses in the browser.

However, that keeps all the data downloaded in the raw XMLHttpRequest, which I don't want.

I suspect this may be the case with oboe.js as well and potentially a limitation of XMLHttpRequest itself. Not sure as I haven't directly worked on this type of use case. Curious to see what you find out with your efforts and other answers to this question.