memoise
memoise copied to clipboard
Memoised Function Along With Using foreach() In A Package
I am using R-4.3.2
with doParallel v1.0.17
and foreach v1.5.2
I have built a proprietary R package which includes a function which parallelizes some tasks and sends them along with chunks of data to a number of workers. The package works as expected and the execution time was reduced to 1/4 of sequential execution time.:
f = function(data, workers, ...) {
< some computations involving splitting the data >
res = foreach( seq(along=data)
.combine = 'rbind'
, . multicombine = TRUE etc.) %dopar% {
< long computation >
}
return (res)
}
However, I have tried to send the memoised (cached to memory) copy of this function (e.g. memoised_f above) to each worker by adding the zzz.R
script suggested in help(memoise)
to the R directory of the package. As result the execution time - while still shorter than the sequential time - increased compared to parallel unmemoised version above.
Using the code in the zzz.R
script outside the package works well but the gain in execution time is not substantially different from before (~ 1/4 of sequential time). However, the function does not seem to have been memoised this way.
My question is: should zzz.R
script be included in the package or should the suggested code be used independently, outside the package? The documentation is not entirely clear in this respect.
Thank you!