doRedis workers shut down immediately on windows

Question

Solved -- look below.

I'm trying to run a doRedis parallel backend with R on Windows 7. And I'm running into problems while trying the first example from the doRedis vignette:

library('doRedis')
registerDoRedis('jobs')
startLocalWorkers(n = 2, queue = 'jobs', host = 'localhost')
...

And this is where things break down: first, two R.exe processes are run and are visible in the task manager, then two Rterm.exe processes are run, they load some 250 MB of data (which I expect to be libraries from my local project's .Rprofile file), and then instantaneously shut down.

I did run a few checks:

The Redis server seems to be working as intended, which I can verify by running:

redisInfo()

and seeing that it indeed has received connections, and that a client is connected.

Moreover, running multiple independent R sessions and registering for the 'jobs' queue works, i.e. running the following:

require('doRedis')
redisWorker('jobs')

And then running the MC example in the main R session by entering

foreach(icount(10), .combine = sum, .multicombine = TRUE, .inorder = FALSE) %dopar% 4*sum((runif(1e5)^2 + runif(1e5)^2)<1)/1e6

works as advertised -- the terminal sessions report activity, and the main session obtains the result. \newline

Hence, I need help to...

... track down the reason for which the worker processes which are invoked by the main R session shut down instantly. I have no idea where to start, however. I know the possible workarounds for the problem, but it would be much more convenient to be able to start it all up in a hassle-free way.

Thanks!

Solution

The problem was that the R processes invoked by the startLocalWorkers command didn't know here to look for libraries. If I invoked redisWorker in an R session that had a properly modified library path list, everything worked fine. If, however, I asked this R session to start up new workers -- everything failed.

The solution was to create an environment variable R_USER_LIBS which holds the path to the library folders. Once that is done, everything works perfectly.

RStudio users should beware that every modification of the environment variables requires them to restart RStudio before the child R processes can read the new value of the modified R_LIBS_USER.

Answer 1

I suggest that you try your tests using Rterm.exe, rather than something like Rgui, otherwise error messages may be lost. When you execute startLocalWorkers, you should immediately see a bunch of messages that are written to stdout. If you're not seeing them, you might also be missing some very useful error messages.

Also, since you were able to executeredisWorker successfully in an interactive R session, it's possible that something in your .Rprofile causes the workers to die in the non-interactive R sessions created by startLocalWorkers. You could test that by creating an R script that executes redisWorker, and executing it with a command like:

R --slave -f worker.R