Using data.tables, trying to aggregate data by column index
-
03-12-2019 - |
Question
I'm having some trouble using the data.table package. I'm using this package because it seems to be very fast and efficient with memory and it will be working on a very large data set (~ 6m x 300).
So, basically an example of the problem I'm having is:
AA <- matrix(runif(50,0,100), 10,5)
AA <- data.table(AA)
colnames(AA) <- c("one","two","three","four","five")
AA[,"key"] <- c(1:10)
setkey(AA,key)
BB <- matrix(c("A1","A1","B1","A1","C1","F1","T1","Y1","S1","S1","B2","C2","V2","G2","R2","U2","P2","Q2","A2","R2"),10,2)
BB <- data.table(BB)
BB[,"key"] <- c(1:10)
setkey(BB,key)
CC <- AA[BB]
This gives the following
> CC
key one two three four five V1 V2
[1,] 1 70.528360 7.901987 66.827238 44.51487 26.22273 A1 B2
[2,] 2 38.560889 31.808611 7.877950 34.51093 51.27989 A1 C2
[3,] 3 70.164154 16.636281 59.127573 79.95673 19.07643 B1 V2
[4,] 4 82.019267 86.958215 3.335632 44.19048 46.29047 A1 G2
[5,] 5 24.980403 25.352212 78.240760 93.69818 46.64401 C1 R2
[6,] 6 1.062644 30.214449 15.920193 35.15496 97.86995 F1 U2
[7,] 7 5.242374 47.591899 56.879902 70.05319 82.48689 T1 P2
[8,] 8 69.646271 69.576102 38.766948 38.62866 74.69404 Y1 Q2
[9,] 9 25.335255 54.638416 5.777238 80.87692 34.11951 S1 A2
[10,] 10 54.844424 18.645826 59.370042 48.24352 84.02630 S1 R2
What I'm trying to do is aggregate the data by V1 and V2
> CC[,length(one), by=V1]
V1 V1.1
[1,] A1 3
[2,] B1 1
[3,] C1 1
[4,] F1 1
[5,] T1 1
[6,] Y1 1
[7,] S1 2
> CC[,length(one), by=V2]
V2 V1
[1,] B2 1
[2,] C2 1
[3,] V2 1
[4,] G2 1
[5,] R2 2
[6,] U2 1
[7,] P2 1
[8,] Q2 1
[9,] A2 1
The problem I'm having is that if I don't know explicitly the names of the columns I want to aggregate by, or if I want to loop through say 100 columns getting 100 different aggregates, how can I do this?
The data.table reference manual says this works the way it does since the variables are referenced in the scope of the data table, so CC[, V1] will give the one column, whereas CC[, "V1"] won't. It says you can use something like
x <- quote(V1)
CC[,length(one), by=eval(x)]
But this doesn't seem to work, I've tried a few things such as setting up the variable names in a vector and various combinations of quote(), noquote(), enquote() but I can't seem to figure out if it's possible.
How can I set this up to loop through a list of column names aggregating by each as it goes?
If not, are there any better ways to aggregate a large data set like this quickly?
Thanks.
La solution
I'm not sure exactly what you are going for -- I think you may have to come up with a better example of what you are trying to do.
You can, for instance, pass in a character vector in by
, so this would work:
agg.by <- "V1"
CC[, length(one), by=agg.by]
If you want to summarize over "unknown" columns in your subsets, you can lapply
over the .SD
data.table that is in scope inside each of your aggregates, eg:
CC[, lapply(.SD, mean), by=agg.by]
If you are only summarizing a few columns from your original data.table, use the .SDcols
argument, eg:
CC[, lapply(.SD, mean), by=agg.by, .SDcols=c('one', 'two')]
I think some combination of the above is going to address the question you are asking, but I'm having a hard time understanding exactly what you're after.
If you can give a better chunk of example data and expected results, I'll be happy to help further.