doParallel foreach scoping issue: Error in eval(expr, envir, enclos) : could not find function

Question

I want to execute a tclTaskSchedule timer (from the tcltk2 package) inside each doParallel thread. However, the following minimal code

library(doParallel)

n <- detectCores()
cl <- makeCluster(n, outfile="out.log")
registerDoParallel(cl)

testfn <- function() print(paste("hello from", i))
foreach(i=1:n, .packages = c("tcltk", "tcltk2"), .verbose = T) %dopar% {
    tclTaskSchedule(1000, testfn(), id = paste0("task", i), redo = 10)
}

stopCluster(cl)

results in the error (printed not in the console, but in out.log)

Error in eval(expr, envir, enclos) : could not find function "testfn"

However, from the .verbose = T argument, I can see from the console that testfn is getting exported:

automatically exporting the following variables from the local environment:
  testfn

Indeed, calling foreach with .export = "testfn" results in the same error.

So what's going wrong?

---

(Why do I want to do this? In the end, I want to poll a number of data sources at regular time intervals asynchronously, and where each data source has its own particular polling interval)

Answer 1

I agree with Roland that the problem is in the way that tclTaskSchedule evaluates its arguments. My solution isn't pretty, but I got it to work by exporting testfn using clusterExport and assigning i to the worker's global environment in the foreach loop:

testfn <- function() print(paste("hello from", i))
clusterExport(cl, "testfn")

foreach(i=1:n, .packages = c("tcltk", "tcltk2"), .verbose = F) %dopar% {
    i <<- i
    tclTaskSchedule(1000, testfn(), id = paste0("task", i), redo = 10)
}

I would probably also pass .noexport="testfn" to foreach, but it isn't really necessary.

Answer 2

You're also going to discover another problem, even without tclTaskSchedule: print statements within a parallelized foreach loop will not output to an interactive session:

From a blog post:

One problem with foreach is that it creates new RScript instances for each iteration of the loop, which prevents status messages from being logged to the console output.

A work around is to create an output log file:

cat("", file="log.txt")
testfn <- function() cat("hello from", i, "\n", file="log.txt", append=TRUE)
foreach(i=1:n, .packages = c("tcltk", "tcltk2"), .verbose = T) %dopar% {
    tclTaskSchedule(1000, testfn(), id = paste0("task", i), redo = 10)
}

This can then be monitored using tail -f log.txt from a bash terminal.