parallelly

parallelly copied to clipboard

HELP WANTED: availableWorkers()

1

# Background When submitting a job to the TORQUE / PBS using something like: ```sh qsub -l nodes=3:ppn=2 myjob.sh ``` the scheduler will allocate 3 nodes with 2 cores each...

HenrikBengtsson

HELP WANTED: Agility of availableCores()

15

`parallelly::availableCores()` returns the number of cores available for multicore processing. R itself provides `parallel::detectCores()` for this. There is also the `mc.cores` option (default to environment variable `MC_CORES`) set when `parallel`...

HenrikBengtsson

makeClusterPSOCK(..., rscript_envs = ...) - more clever

8

* `makeClusterPSOCK()` gained argument 'rscript_envs' for setting environment variables in workers on startup, e.g. `rscript_envs = c(FOO = "3.14", "BAR")`. Instead of doing this [via `-e "Sys.setenv(''='')"` options](https://github.com/HenrikBengtsson/future/commit/107ed0dfb5c9d4860442c59b57e2c48d0600e43c), can't we...

HenrikBengtsson

makeClusterPSOCK(): Add `rscript_options`

1

Add argument `rscript_options` to `makeClusterPSOCK()`, cf. argument `rscript_envs`. This can be used to see R options on the cluster nodes on startup. For example, ```r cl

HenrikBengtsson

availableCores(): Add support for HTCondor

3

HTCondor users, I need your help to add support for HTCondor to `availableCores()`: HPC schedulers such as Slurm, SGE, and Torque/PBS set _environment variables_ that can be queried to figure...

HenrikBengtsson

WISH: Shutting down clusters upon session exit

3

If the parent session of a PSOCK cluster terminates without properly shutting down the cluster first, the workers will (i) keep running, and (ii) eventually produce an error due to...

HenrikBengtsson

# Wish For example, for a machine with: ```sh $ lscpu | grep -iF "core" Thread(s) per core: 2 Core(s) per socket: 4 Model name: Intel(R) Core(TM) i7-8650U CPU @...

HenrikBengtsson

Add S3 methods `psKill()` for `cluster` to send a signal to the cluster node R process, e.g. ```r > cl isAlive(cl) [1] TRUE TRUE TRUE TRUE > psKill(cl[2:3], signal =...

HenrikBengtsson

ROBUSTNESS: Protect main-worker communication using suspendInterrupts()?

14

# Issue If the user hits Ctrl-C (signals a user interrupt) while the main R session and a worker communicates data, then the communication ends up in an unrecoverable corrupt....

HenrikBengtsson

From https://github.com/HenrikBengtsson/future/issues/435: > Should localhost PSOCK cluster node inherit .libPaths() from parent R session? This could be done by introducing a new default `rscript_libs = ""` and have that to...

HenrikBengtsson