You should reverse the sequence of how you define groupmean
. Compute it as the group average for all rows, and substitute the rows where condition == TRUE
afterwards.
DT[, groupmean:=mean(numbers), by=group]
DT[condition==TRUE, groupmean:=mean(numbers), by='group,condition']
I hope that helps