Trying to get drake transient workers going via {future.clustermq}

[pat-s]

I was playing around today with {future.clustermq} and wanted to share my experience. Maybe @mschubert has some thoughts here to push this initiative by @HenrikBengtsson. This relates to https://github.com/HenrikBengtsson/future/issues/204.

I've started with @wlandau {drake} instead of {targets} because I haven't dived yet into the latter. But this is more about {future.clustermq} than about drake vs. targets and if this one works at some point, porting it to drake might not be that hard hopefully.

I got stuck at the step sending the future to the worker for evaluation. The following snippet should get you to https://github.com/pat-s/future.clustermq/blob/e0a25dc4cedd12663aca1a561cff9a72e00ec3b4/R/ClusterMQFuture-class.R#L240 where I am trying to evaluate the expression.

Note: It could be that the semantics for the workers object at the linked reference are totally wrong - I do not have deep knowledge of the {clustermq} internals nor the {future} internals. I was just playing around with what was already there. In essence this is just a start to push this initiative a bit and maybe we can all do this together.

# remotes::install_github("pat-s/future.clustermq@test)

library(drake)

# Create the template file. You may have to modify it.
# drake_hpc_template_file("slurm_clustermq.tmpl")

# Configure clustermq.
# options(clustermq.scheduler = "slurm", template = "slurm_clustermq_future.tmpl")

options(clustermq.scheduler = "multiprocess")
suppressPackageStartupMessages(library(future.clustermq))
# FIXME: is this even correct?
future::plan("clustermq")

load_mtcars_example()
make(my_plan, parallelism = "future", jobs = 1)