Using data.tables, trying to aggregate data by column index

Question

I'm having some trouble using the data.table package. I'm using this package because it seems to be very fast and efficient with memory and it will be working on a very large data set (~ 6m x 300).

So, basically an example of the problem I'm having is:

AA <- matrix(runif(50,0,100), 10,5)
AA <- data.table(AA)
colnames(AA) <- c("one","two","three","four","five")
AA[,"key"] <- c(1:10)
setkey(AA,key)

BB <- matrix(c("A1","A1","B1","A1","C1","F1","T1","Y1","S1","S1","B2","C2","V2","G2","R2","U2","P2","Q2","A2","R2"),10,2)
BB <- data.table(BB)
BB[,"key"] <- c(1:10)
setkey(BB,key)

CC <- AA[BB]

This gives the following

> CC
  key       one       two     three     four     five V1 V2
 [1,]   1 70.528360  7.901987 66.827238 44.51487 26.22273 A1 B2
 [2,]   2 38.560889 31.808611  7.877950 34.51093 51.27989 A1 C2
 [3,]   3 70.164154 16.636281 59.127573 79.95673 19.07643 B1 V2
 [4,]   4 82.019267 86.958215  3.335632 44.19048 46.29047 A1 G2
 [5,]   5 24.980403 25.352212 78.240760 93.69818 46.64401 C1 R2
 [6,]   6  1.062644 30.214449 15.920193 35.15496 97.86995 F1 U2
 [7,]   7  5.242374 47.591899 56.879902 70.05319 82.48689 T1 P2
 [8,]   8 69.646271 69.576102 38.766948 38.62866 74.69404 Y1 Q2
 [9,]   9 25.335255 54.638416  5.777238 80.87692 34.11951 S1 A2
[10,]  10 54.844424 18.645826 59.370042 48.24352 84.02630 S1 R2

What I'm trying to do is aggregate the data by V1 and V2

> CC[,length(one), by=V1]
     V1 V1.1
[1,] A1    3
[2,] B1    1
[3,] C1    1
[4,] F1    1
[5,] T1    1
[6,] Y1    1
[7,] S1    2

> CC[,length(one), by=V2]
  V2 V1
[1,] B2  1
[2,] C2  1
[3,] V2  1
[4,] G2  1
[5,] R2  2
[6,] U2  1
[7,] P2  1
[8,] Q2  1
[9,] A2  1

The problem I'm having is that if I don't know explicitly the names of the columns I want to aggregate by, or if I want to loop through say 100 columns getting 100 different aggregates, how can I do this?

The data.table reference manual says this works the way it does since the variables are referenced in the scope of the data table, so CC[, V1] will give the one column, whereas CC[, "V1"] won't. It says you can use something like

x <- quote(V1)
CC[,length(one), by=eval(x)]

But this doesn't seem to work, I've tried a few things such as setting up the variable names in a vector and various combinations of quote(), noquote(), enquote() but I can't seem to figure out if it's possible.

How can I set this up to loop through a list of column names aggregating by each as it goes?

If not, are there any better ways to aggregate a large data set like this quickly?

Thanks.

Answer 1

I'm not sure exactly what you are going for -- I think you may have to come up with a better example of what you are trying to do.

You can, for instance, pass in a character vector in by, so this would work:

agg.by <- "V1"
CC[, length(one), by=agg.by]

If you want to summarize over "unknown" columns in your subsets, you can lapply over the .SD data.table that is in scope inside each of your aggregates, eg:

CC[, lapply(.SD, mean), by=agg.by]

If you are only summarizing a few columns from your original data.table, use the .SDcols argument, eg:

CC[, lapply(.SD, mean), by=agg.by, .SDcols=c('one', 'two')]

I think some combination of the above is going to address the question you are asking, but I'm having a hard time understanding exactly what you're after.

If you can give a better chunk of example data and expected results, I'll be happy to help further.