d3js (+crossfilter/dc) boxplot performance

Question

I'm trying to use a combination of D3, Crossfilter, and DC generate interactive boxplots, mainly using this example: https://github.com/dc-js/dc.js/blob/master/web/examples/box-plot.html

My data looks more like this:

id      date        met1
6368    10/24/2013  0.84
6369    10/24/2013  0.67
6374    10/24/2013  0.96
6375    10/24/2013  0.97

with around half a million data points, which works fine for everything else except for the boxplots. The code works fine and the boxplots are fine, but when I change the filter elsewhere it takes forever for the boxplots to update:

var met1Dim = data.dimension(function(d) {return "metric 01";});
var met1Values = met1Dim.group().reduce(
            function(p, v) {
                p.push(v.met1);
                return p;
            },
            function(p,v) {
                p.splice(p.indexOf(v.met1), 1);
                return p;
            },
            function() {
                return [];
            }

Performance is drastically improved (but still not exactly ideal) when I pass integers as the data (just by replacing v.met1 with parseInt(v.met1 * 100)), but that's sort of half-assed and I'd like to display the data in their proper range, not by coercing everything into an integer. The most significant slowdown occurs when I'm removing datasets, and I think it's the slice(indexOf()) that's slowing everything down (when using floats). Is there anything I can do to make this operation faster? I was thinking maybe of an associative array using the id data is a key, but I'm not sure how to pass associative arrays into the reduce() function.

Thanks.

Answer 1

Using a map lookup instead of indexOf will definitely help but it's still unfortunate that you need to jump through such hoops when all you really want as far as I can tell is to just do a basic grouping with the met1 values showing up at the top level because that's what dc.js expects.

It would be very nice if dc.js allowed you to specify an accessor function that allowed you to define how you want the met1 value pulled from the bound objects at the point at which it's needed. d3 uses this pattern everywhere and it's very convenient (and also saves you from this kind of performance intensive juggling).

Barring this sort of change to dc.js, I think your idea for a map lookup is probably the best bet. You can use the closure scope of your reduce functions to store it. Since your records have unique ids you can just use a single map without needing to worry about collisions:

var indexMap = {};
var met1Values = met1Dim.group().reduce(
    function(p, v) {
        indexMap[v.id] = p.length;  // length before push is index of pushed element
        p.push(v.met1);
        return p;
    },
    function(p,v) {
        p.splice(indexMap[v.id], 1);
        return p;
    },
    function() {
        return [];
    }
);