batchtools

batchtools copied to clipboard

With future.batchtools and other wrappers, it become a bit tricky to track down where errors are coming from when running in batch mode. For instance, I got some:

Error : BatchtoolsError in BatchtoolsFuture ('future_lapply-1'): 'Work dir does not exist'

I wasn't sure if that was from the scheduler or batchtools, but it turns out it's from here:

https://github.com/mllg/batchtools/blob/1001440ab14e697a28c9d901bc976c7d5b9e404b/R/doJobCollection.R#L84-L86

Would you mind updating to something like:

error("Working directory (%s) for the batchtools registry does not exist", jc$work.dir)

Forgot to say, including what the working directory is also helpful in cases where it is set to, say, a local temp folder that is not available on the compute node processing the job.

Actually, even more informative would be:

error("Working directory (%s) for the batchtools registry does not exist on host %s",
      sQuote(jc$work.dir), sQuote(Sys.info()[["nodename"]]))

Annotating other errors that occur when trying to launch a job in a similar fashion would help troubleshoot failed jobs; it's not always clear when, where, and on which R process these errors take place.