memoise icon indicating copy to clipboard operation
memoise copied to clipboard

Memoised Function Along With Using foreach() In A Package

Open drag05 opened this issue 1 year ago • 2 comments

I am using R-4.3.2 with doParallel v1.0.17 and foreach v1.5.2

I have built a proprietary R package which includes a function which parallelizes some tasks and sends them along with chunks of data to a number of workers. The package works as expected and the execution time was reduced to 1/4 of sequential execution time.:

f = function(data, workers, ...) {
              < some computations involving splitting the data >
       res = foreach( seq(along=data)
              .combine = 'rbind'
            , . multicombine = TRUE etc.) %dopar% { 
            < long computation >
  }
return (res)
}

However, I have tried to send the memoised (cached to memory) copy of this function (e.g. memoised_f above) to each worker by adding the zzz.R script suggested in help(memoise) to the R directory of the package. As result the execution time - while still shorter than the sequential time - increased compared to parallel unmemoised version above.

Using the code in the zzz.R script outside the package works well but the gain in execution time is not substantially different from before (~ 1/4 of sequential time). However, the function does not seem to have been memoised this way.

My question is: should zzz.R script be included in the package or should the suggested code be used independently, outside the package? The documentation is not entirely clear in this respect.

Thank you!

drag05 avatar Jan 05 '24 21:01 drag05