make .combine function scaleable

Question

I am trying to use foreach and am having problems making the .combine function scalable. For example, here is a simple combine function

MyComb <- function(part1,part2){
          xs <- c(part1$x,part2$x)
          ys <- c(part1$y,part2$y)
          return(list(xs,ys))
          }

When I use this function to combine a foreach statement with an iterator other than 2 it returns it incorrectly. For example this works:

   x = foreach(i=1:2,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)

But not this:

 x = foreach(i=1:3,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)

Is there a way to generalize the combine function to make it scalable to n iterations?

Answer 1

Your .combine function must take either two pieces and return something that "looks" like a piece (could be passed back in as a part) or take many arguments and put all of them together at once (with the same restrictions). Thus at least your MyComb must return a list with components x and y (which is what each piece of your %dopar% do.

A couple of ways to do this:

MyComb1 <- function(part1, part2) {
    list(x=c(part1$x, part2$x), y=c(part1$y, part2$y))
}

x = foreach(i=1:3,.combine=MyComb1) %dopar% list("x"=i*2,"y"=i*3)

This version takes only two pieces at a time.

MyComb2 <- function(...) {
    dots = list(...)
    ret <- lapply(names(dots[[1]]), function(e) {
        unlist(sapply(dots, '[[', e))
    })
    names(ret) <- names(dots[[1]])
    ret
}

s = foreach(i=1:3,.combine=MyComb2) %dopar% list("x"=i*2,"y"=i*3)
x = foreach(i=1:3,.combine=MyComb2, .multicombine=TRUE) %dopar% list("x"=i*2,"y"=i*3)

This one can take multiple pieces at a time and combine them. It is more general (but more complex).