Here is an example:
library(gtable)
grid.draw(cbind(rbind(p1, p2, size="last"), rbind(p3, p4, size="last"), size = "first"))
Updated
This is a bad hack so I don't recommend to use. Probably this will not work in future.
gt <- cbind(rbind(p1, p2, size="last"), rbind(p3, p4, size="last"), size = "first")
for (i in which(gt$layout$name == "guide-box")) {
gt$grobs[[i]] <- gt$grobs[[i]]$grobs[[1]]
}
grid.draw(gt)