NAs produced when overlaying shapefile on SpatialPixelsDataFrame

Question

I am experiencing an odd issue when overlaying a series of catchment polygons (from a shapefile) on to a SpatialPixelsDataFrame object. There are 70 catchments but I am getting NAs for 60 of the catchments.

The data are available on Dropbox and the following script will download, unpack etc the files and create a reproducible example.

## load packages required
require(rgdal)
require(sp)

## proj4 string
BNG <- CRS("+init=epsg:27700")

## load in the grid
download.file("https://www.dropbox.com/s/7dvcf6ddo6r3043/depogrid.rds?dl=1",
              "./depogrid.rds", method = "wget")
myGrid <- readRDS("depogrid.rds")

## convert to spatial pixels
depo.sp <- SpatialPixelsDataFrame(data.matrix(myGrid[, 1:2]),
                                  data = myGrid[, 3, drop = FALSE],
                                  proj4string = BNG)

## load the shapefile
download.file("https://www.dropbox.com/s/bwzmeumnrja3c0o/dart.zip?dl=1",
              "./dart.zip", method = "wget")
unzip("./dart.zip")
shp <- readOGR(".", "DARTall")
## project shp to OSGB36
shp <- spTransform(shp, BNG)

## overlay shapefile on depo grid
depo.dart <- over(shp, depo.sp)

The problem is the NAs:

> head(depo.dart)
  Deposition
0         NA
1         NA
2         NA
3         NA
4         NA
5  0.2604089
> sum(is.na(depo.dart[,1]))
[1] 60
> nrow(depo.dart)
[1] 70

There shouldn't be any NAs as the grid covers the entire UK region and the catchments are small, so I am probably doing something wrong! If so what am I doing wrong and how do I fix this? I want to end up with an average value for Deposition (the SpatialPixelsDataFrame data) for each of the 70 catchments in shp.

[Note that the grid is dummy data; I can't distribute the actual deposition grid, but the data and object is have is essentially (exactly) the same as depo.sp.

Additional Info

> over(shp, depo.sp, returnList = TRUE)
[[1]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[2]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[3]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[4]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[5]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[6]]
     Deposition
3143  0.2604089

[[7]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[8]]
     Deposition
3289  0.1139663

[[9]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[10]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[11]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[12]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[13]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[14]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[15]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[16]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[17]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[18]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[19]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[20]]
     Deposition
3585  0.8610678

[[21]]
     Deposition
3288  0.2242067

[[22]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[23]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[24]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[25]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[26]]
     Deposition
3141  0.3296081

[[27]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[28]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[29]]
     Deposition
3438  0.3609788

[[30]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[31]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[32]]
     Deposition
3287  0.4812801

[[33]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[34]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[35]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[36]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[37]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[38]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[39]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[40]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[41]]
     Deposition
3436  0.3223757

[[42]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[43]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[44]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[45]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[46]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[47]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[48]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[49]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[50]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[51]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[52]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[53]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[54]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[55]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[56]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[57]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[58]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[59]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[60]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[61]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[62]]
     Deposition
2580  0.2820617

[[63]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[64]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[65]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[66]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[67]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[68]]
     Deposition
3141  0.3296081
3287  0.4812801
3288  0.2242067
3438  0.3609788

[[69]]
[1] Deposition
<0 rows> (or 0-length row.names)

[[70]]
[1] Deposition
<0 rows> (or 0-length row.names)
> str(over(shp, depo.sp, returnList = TRUE))
List of 70
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.26
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.114
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.861
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.224
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.33
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.361
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.481
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.322
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   1 obs. of  1 variable:
  ..$ Deposition: num 0.282
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   4 obs. of  1 variable:
  ..$ Deposition: num [1:4] 0.33 0.481 0.224 0.361
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0) 
 $ :'data.frame':   0 obs. of  1 variable:
  ..$ Deposition: num(0)

Answer 1

If you are open to experimentation and willing to use raster package, here's a possible solution. Let us know.

The trick is, when overlaying polygons over rasters, to specify small = TRUE, because centers of polygons don't always include cell center (see ?extract).

If y represents polygons, a value is also returned for relatively small polygons (e.g. those smaller than a single cell of the Raster* object), or polygons with an odd shape, for which otherwise no values are returned because they do not cover any raster cell centers.

library(raster)
shp.raster <- raster(shp)
depo.sp.raster <- raster(depo.sp)
plot(depo.sp.raster)
plot(shp, add = TRUE)
zoom(depo.sp.raster) # click in the corners around the area to zoom in
plot(shp, add = TRUE)

res.out <- extract(depo.sp.raster, shp, layer = 1, small = TRUE)
table(unlist(lapply(res.out, is.null)))
> head(res.out)
[[1]]

0.0334963 

[[2]]

0.2604089 

[[3]]
[1] 0.5466648 0.7929624 0.4214012

[[4]]
[1] 0.2604089 0.3149418 0.5343358 0.0334963

[[5]]

0.5561869 

[[6]]

0.2604089