R: reference iteration number in call to sfLapply(1:N, function(x))

Question

Is it possible to reference the iteration number in a sfLapply call as follows -

wrapper <- function(a) {
    y.mat <- data.frame(get(foo[i,1]), get(foo[i,2]))
        ...
        ...
        do other things....

    }


results <- sfLapply(1:200000, wrapper)

Where i is the iteration number as sfLapply cycles through 1:200000.

The problem I am faced with is that I have over 200,000 cases to test, with each case requiring the construction of a data.frame to which various operations will be performed.

I have a 2 Ghz Intel Core 2 Duo processor (macbook laptop) and so I began to investigate the snowfall package to take advantage of parallel processing. This led me to sfLapply and so I started to investigate whether I could re-write my code to work with lapply(). However, I have yet to come across examples that reference the iteration number in lappy() calls.

Maybe I am heading in the wrong direction. If anyone has any suggestions I would be greatly appreciative.

Answer 1

You're not using parameter a in the code to wrapper. All the numbers from 1:200000 will be passed to wrapper, so it is this a that represents your iteration (instead of i).

Don't forget, though, that these will not appear in order (courtesy of sfLapply).

As far as I know, there is no way of knowing the how manyth iteration your going into, as the different processes don't know what the others are doing.