How to find NAs in groups and create new column for data frame

Question

I have a data frame consisting of an "ID" column and a "Diff" column. The ID column is responsible for marking groups of corresponding Diff values. An example looks like this:

structure(list(ID = c(566, 566, 789, 789, 789, 487, 487, 11, 
11, 189, 189), Diff = c(100, 277, 529, 43, NA, 860, 780, 445, 
NA, 578, 810)), .Names = c("ID", "Diff"), row.names = c(9L, 10L, 
20L, 21L, 22L, 25L, 26L, 51L, 52L, 62L, 63L), class = "data.frame")

My goal is to search each group for NAs in the Diff column and create a new column, that has either a "True" or "False" value for each row, depending if the corresponding group has an NA in Diff.

I tried

x <- aggregate(Diff ~ ID, data, is.na)

and

y <- aggregate(Diff ~ ID, data, function(x) any(is.na(x)))

The idea was to merge the result depending on ID. However, none of the above created a useful result. I know R can do it … and after searching for quite a while I ask you how :)

Answer 1

(You have two viable strategies already, but here is another which may be conceptually easier to follow if you are relatively new to R and aren't familiar with the way plyr works.)

I often need to know how many NAs I have in different variables, so here is a convenience function I use standard:

sna <- function(x){
  sum(is.na(x))
}

From there, I sometimes use aggregate(), but sometimes I find ?summaryBy in the doBy package to be more convenient. Here's an example:

library(doBy)
z <- summaryBy(Diff~ID, data=my.data, FUN=sna)
z
   ID Diff.sna
1  11        1
2 189        0
3 487        0
4 566        0
5 789        1

After this, you just need to use ?merge and convert the count of NAs to a logical to get your final data frame:

my.data          <- merge(my.data, z, by="ID")
my.data$Diff.sna <- my.data$Diff.sna>0
my.data
    ID Diff Diff.sna
1   11  445     TRUE
2   11   NA     TRUE
3  189  578    FALSE
4  189  810    FALSE
5  487  860    FALSE
6  487  780    FALSE
7  566  100    FALSE
8  566  277    FALSE
9  789  529     TRUE
10 789   43     TRUE
11 789   NA     TRUE

Answer 2

You can use the plyr and ddply

require(plyr)
ddply(data, .(ID), transform, na_diff = any(is.na(Diff)))
##     ID Diff na_diff
## 1   11  445    TRUE
## 2   11   NA    TRUE
## 3  189  578   FALSE
## 4  189  810   FALSE
## 5  487  860   FALSE
## 6  487  780   FALSE
## 7  566  100   FALSE
## 8  566  277   FALSE
## 9  789  529    TRUE
## 10 789   43    TRUE
## 11 789   NA    TRUE

Answer 3

A very similar solution to @dickoa except in base:

do.call(rbind,by(data,data$ID,function(x)transform(x,na_diff=any(is.na(Diff)))))
#         ID Diff na_diff
# 11.51   11  445    TRUE
# 11.52   11   NA    TRUE
# 189.62 189  578   FALSE
# 189.63 189  810   FALSE
# 487.25 487  860   FALSE
# 487.26 487  780   FALSE
# 566.9  566  100   FALSE
# 566.10 566  277   FALSE
# 789.20 789  529    TRUE
# 789.21 789   43    TRUE
# 789.22 789   NA    TRUE

Similarly, you could avoid transform with:

data$na_diff<-with(data,by(Diff,ID,function(x) any(is.na(x)))[as.character(ID)])