All workers finished their job, foreach %dopar% got stuck

[telenskyt]

I have a code approx. like this:

removeQueue('jobs')
registerDoRedis('jobs', redis_server)
r <- foreach (i = c(226, 229, 230, 246), .errorhandling = 'pass', .verbose = TRUE) %dopar% {
   ... really long code (running JAGS using runjags package)
}

All workers finished their job without error, but the foreach loop got stuck:

numValues: 4, numResults: 0, stopped: TRUE automatically exporting the following objects from the local environment: [... cut out... ]

After I stopped the foreach in Rgui, it wrote:

[1] "interrupt: \n" Error in tryCatchOne(expr, names, parentenv, handlers[[1L]]) : attempt to apply non-function

(I am not sure if it is because of the stopping or some other cause).

Here follows the state of the Redis server:

redisKeys("*") [1] "jobs:workers" "jobs:1.env" "jobs:6.results" "jobs:6.env" "jobs:counter"
[6] "jobs:1.start.2" "jobs:1.start.1" "jobs:1.start.4" redisGet("jobs:workers") [1] "4" redisGet("jobs:counter") [1] "6" summary(redisLRange("jobs:6.results", 0, -1)) Length Class Mode [1,] 1 -none- list [2,] 1 -none- list [3,] 1 -none- list

So the jobs:6.results contains the results but only 3 tasks instead of 4 - the result of task 3 is missing. Looking at the console of worker that processed task 3, it was completed OK without errors! Perhaps the results of the task were not submitted by the worker? But then again - why the master did not receive the results for the other tasks? E.g. tasks 1, 2 were already finished. Normally, the master (the R session who runs the foreach) writes on the console right away that it received the results. But not this time!

Strange enough. This doesn't happen every time - it's hard to reproduce.

I use R 3.1.0, doRedis 1.1.1, rredis 1.6.9 on Windows XP on the master machine, R 3.1.1 on Windows 8 on the workers (packages are the same version), and redis server 2.8.8 on remote linux machine.

[cloudcell]

I had the same issue. In my case, R versions were different across machines (3.1.3 and 3.2.2 in my case). Try using the same version of R on all the machines (and also make sure all the packages you use have the same version). That should fix your problem.

My conclusion: this is not caused by doRedis but rather by conflicts between versions of R (and different versions of packages) used.