clustermq icon indicating copy to clipboard operation
clustermq copied to clipboard

support for tidymodels?

Open nick-youngblut opened this issue 3 years ago • 1 comments

It appears that tidymodels only supports parallelization via the do* packages. That includes doFuture so I was hoping that clustermq would be supported by future, but it appears that this package is all that exists, and it is just a rough developmental package.

I really prefer using clustermq versus batchtools (thanks for making a great package!), but it looks like I'm going to have to go back to batchtools in order to parallelize tidymodels on a cluster.

Are there any plans to support tidymodels via future or another method?

nick-youngblut avatar Feb 08 '21 19:02 nick-youngblut

At some point, there should be both a future backend (via the package that you link), but also support for using workers as a cluster object, which would both enable your use case.

However, I do unfortunately not have an ETA for this.

mschubert avatar Mar 25 '21 15:03 mschubert

Solution to this will be a consequence of #119

mschubert avatar Mar 26 '23 21:03 mschubert