multiple sources for data leads to `invalid array width!` error

Question

I am loading records into a crossfilter object by doing looped ajax calls :

    for (i=0;i<rangeToLoad.length;i++) {

    url = "http://myserver/"+rangeToLoad[i]+"/myrecords.js"

    $.ajax({
        url: url,
        dataType:"json",
        success: function (jsondata) {

            var records = jsondata.records;
            var parseDate = d3.time.format("%Y-%m-%d").parse;

            records.forEach(function(d) {
                d.date = parseDate(d.date);
            });

            if (typeof myCrossfilter === "undefined") { //first call
                myCrossfilter = crossfilter(records);
            } else {
                myCrossfilter.add(records);
            }

            initData();
            showData();
        }
    });
}

My initData() function is basically where the dimensions and groups of my crossfilter objects are defined.

dims.date = myCrossfilter.dimension(function(d) {return(+d.date);});
groups.date={};
dims.date2 = myCrossfilter.dimension(function(d) {return(+d.date);});
groups.date2={};
dims.instrumenttype = myCrossfilter.dimension(function(d) {return(d.product);});
groups.instrumenttype = {};
groups.date.pnlSum = dims.date.group().reduceSum(function(d) {return(d.pnl);});
groups.date2.cumPnlSum = dims.date2.group().reduceSum(function(d) {return(d.cumpnl);});
groups.instrumenttype.pnlSum = dims.instrumenttype.group().reduceSum(function(d) {return(d.pnl);});

My guess is that instead of updating the existing dimension for each ajax call, a new dimension is created each time.

How can I prevent that from happening ?

Answer 1

The trick is to create the dimensions only on the first pass of the ajax calls by checking if the crossfilter is defined. I have also cancelled the asynchronous property of the ajax call as it is not a big issue in my case (data is very quick to load). It prevents the headache-inducing task of figuring out which call was done first.

for (i=0;i<rangeToLoad.length;i++) {

url = "http://myserver/"+rangeToLoad[i]+"/myrecords.js"

$.ajax({
    url: url,
    dataType:"json",
    async:false,
    success: function (jsondata) {


        if (typeof myCrossfilter === "undefined") { //first call
            myCrossfilter = crossfilter(records);
            initData()
        } else {
            myCrossfilter.add(records);
        }

        showData();
    }
});

}

Answer 2

Crossfilter only supports 32 dimensions maximum. You can manage the number of dimensions by using the dimension.dispose function to get rid of the dimensions either when you're done with them (after showData) or before new dimensions come in (prior to initData).

As is illustrated by the accepted answer, Crossfilter also supports adding new rows. Internally, Crossfilter managed a set of listeners that notifies the internal dimension and grouping structures when data is added or removed. So the correct thing to do is to make sure the dimensions are created just once and then change the data as needed in subsequent passes.