This isn't intended. Looks like a problem with optimization ...
> a[,list(mean(a), sum(a)),by=b]
b V1 V2
1: 1 0 NA
2: 2 0 0
> options(datatable.optimize=FALSE)
> a[,list(mean(a), sum(a)),by=b]
b V1 V2
1: 1 NA NA
2: 2 0 0
>
Investigated and fixed in v1.8.9, soon to be on CRAN. From NEWS :
mean() in j has been optimized since v1.8.2 but wasn't respecting na.rm=TRUE (the default). Many thanks to Colin Fang for reporting. Test added.
The new feature in v1.8.2 was :
mean() is now automatically optimized, #1231. This can speed up grouping by 20 times when there are a large number of groups. See wiki point 3, which is no longer needed to know. Turn off optimization by setting options(datatable.optimize=0).