[question] how to provide a conda environment for scheduling SLURM jobs with cluster_tools?

[neuronflow]

Detailed Description

• Affected library: cluster_tools

I am trying to use cluster_tools to schedule SLURM jobs: My code looks like this:

from cluster_tools import SlurmExecutor

with SlurmExecutor(job_resources=slurm_settings_dict) as exc:
        # Create a WebKnossos dataset
        ...

How can I provide a conda environment as context for the scheduled slurm jobs? currently, the jobs run on the base environment which has the wrong python version and dependencies.

[philippotto]

Hi @neuronflow,

this SO post sums it up quite aptly:

Slurm is designed to push the user's environment at submit time to the job, except for variables explicitly disabled by the user or the system administrator.

In other words, it should work out of the box as long as the following is true:

• your "root python script" should be started within your conda environment, too
• the filesystem between the head node and the slurm nodes should be shared so that the conda environment also exists on the slurm nodes
• the sharing of the environment variables should not be disabled or forbidden. By default, this should be enabled, but you can try to set export SBATCH_EXPORT=ALL (some other script might have changed that value).

I hope this helps :crossed_fingers: