Column in the j-expression of a data.table (with/without a by statement)

Question

Here are two artificial but I hope pedagogical examples of my problem.

1) When running this code:

> dat0 <- data.frame(A=c("a","a","b"), B="")
> data.table(dat0)[, lapply(.SD, function(x) length(A)) , by = "A"]
   A B
1: a 1
2: b 1

I expected the output

   A B
1: a 2
2: b 1

(similarly to plyr::ddply(dat0, .(A), nrow)).

Update to question 1)

Let me give a less artificial example. Consider the following dataframe:

dat0 <- data.frame(A=c("a","a","b"), x=c(1,2,3), y=c(9,8,7))
> dat0
  A x y
1 a 1 9
2 a 2 8
3 b 3 7

Using plyr package, I get the means of x and y by each value of A as follows:

> ddply(dat0, .(A), summarise, x=mean(x), y=mean(y))
  A   x   y
1 a 1.5 8.5
2 b 3.0 7.0

Very nice. Now imagine another variable H and the following calculations:

dat0 <- data.frame(A=c("a","a","b"), H=c(0,1,-1), x=c(1,2,3), y=c(9,8,7))
> ddply(dat0, .(A), summarise, x=mean(x)^mean(H), y=mean(y)^mean(H))
  A         x         y
1 a 1.2247449 2.9154759
2 b 0.3333333 0.1428571

Very nice too. But now, imagine there's a huge number of variables x for which you want to calculate mean(x)^mean(H). Then I don't want to type:

ddply(dat0, .(A), summarise, a=mean(a)^mean(H), b=mean(b)^mean(H), c=mean(c)^mean(H), d=mean(d)^mean(H), ...........)

So my idea was to try:

flipcols <- my_selected_columns # c("a", "b", "c", "d", ....)
data.table(dat0)[, lapply(.SD, function(x) mean(x)^mean(H)), by = "A", .SDcols = flipcols]

But that doesn't work because the presence of H in function(x) mean(x)^mean(H) is not handled as I expected! I have not been able to make it work with plyr::colwise too.

2) When running this code:

> dat0 <- data.frame(A=c("a","a","b"), B=1:3, c=0)
> data.table(dat0)[, lapply(.SD, function(x) B), .SDcols="c"]
Error in ..FUN(c) : object 'B' not found

I expected it works and generates :

   c
1: 1 
2: 2 
3: 3 

So is there a way to use the columns of the original data.table in a transformation ?

Answer 1

1) Use .N. The length of the grouping variable A there is 1 because there is just one value of A for each group (this is by definition of what grouping means):

dt <- data.table(A=c("a","a","b"), B="")
dt[, .N, by = A]
#   A N
#1: a 2
#2: b 1

(updated 1) This is the same issue as 2). A workaround is to not use .SDcols:

dt = data.table(A=c("a","a","b"), H=c(0,1,-1), x=c(1,2,3), y=c(9,8,7))
dt[, lapply(.SD[, !"H"], function(x) mean(x) ^ mean(H)), by = A]
#   A         x         y
#1: a 1.2247449 2.9154759
#2: b 0.3333333 0.1428571

2) This is a bug that's been reported before here: https://r-forge.r-project.org/tracker/index.php?func=detail&aid=5222&group_id=240&atid=975

Answer 2

I don't know if I understand you correctly.

1)

library(data.table)
dat0 <- data.frame(A=c("a","a","b"), B="")
data.table(dat0)[, list(l= nrow(.SD)) , by = "A"]

result:

   A l
1: a 2
2: b 1

2)

dat0 <- data.frame(A=c("a","a","b"), B=1:3, c=0)
data.table(dat0)[, list(c=unlist(.SD)), .SDcols= "B"]

result:

   c
1: 1
2: 2
3: 3

1')

Edit: I changed -1 to mycols

dat0 <- data.frame(A=c("a","a","b"), H=c(0,1,-1), x=c(1,2,3), y=c(9,8,7))

data.table(dat0)[, lapply(.SD, function(x) mean(x)^mean(H)), by = "A", .SDcols = c("x", "y")]

result:

   A         u         v
1: a 1.2247449 2.9154759
2: b 0.3333333 0.1428571

Note that if the data is huge, mean(H) will be calculated many times wastefully. We could do {muH = mean(H); lapply(.SD, function(x) mean(x)^muH)} in this case to save computation; the above is a bit more readable though.