Using HTcondor with future.batchtools

[yonicd]

can one of the backends (batchtools_*) be used to send jobs to HTCondor? or would it need to be done with a batchtools_custom?

[HenrikBengtsson]

I don't think so, but I also know nothing about HTCondor - you probably have to roll your own custom config setup.

There's an open issue over at batchtools for adding built-in support for HTCondor (https://github.com/mllg/batchtools/issues/68). The best would be to get it implemented there first. Then it'll be straightforward to include a corresponding future.batchtools backend.

[yonicd]

is there an example of an implementation of batchtools_custom?

[HenrikBengtsson]

It's used to with "cluster functions" (a batchtools concept), e.g.

cf <- batchtools::makeClusterFunctionsInteractive(external = TRUE)
plan(batchtools_custom, cluster.workers = cf)

## Create explicit future
f <- future({
  cat("PID:", Sys.getpid(), "\n")
  42L
})
v <- value(f)
print(v)

(I've added this to example("batchtools_custom") for the next release)

So, to if you can create proper batchtools cluster functions for HTCondor, then you can use the above.

[scottkosty]

@yonicd were you able to figure out how to use batchtools with HTCondor (and then with the future package) in the end? I am working with someone who uses HTCondor and would love to be able to use future.

[yonicd]

I have not used Condor for a while now (un)fortunately, sorry

[scottkosty]

I have not used Condor for a while now (un)fortunately, sorry

No worries, thanks a lot for the reply!

[yonicd]

This isn’t future based, but worked well enough with the condor hpc cluster https://github.com/yuliasidi/condor

[scottkosty]

Thank you! I am set on using future, but if someday I have the time to jump down the rabbit hole that might be helpful for figuring out a patch to batchtools.