Does pyslurm support job arrays?

[vrindger]

I can't get it to recognize $SLURM_ARRAY_TASK_ID within the batch script, so I'm wondering if I should keep trying or give up on making job arrays work through the submit_batch_job function in pyslurm.

[vrindger]

I'm using an sbatch file and submitting the job like this: slurmJob = { "script": slurm_script_path } slurmJobId = pyslurm.job().submit_batch_job(slurmJob)

[vrindger]

Here's the gist of the SBATCH script file: #!/bin/bash #SBATCH --time=1:00:00 #SBATCH --ntasks=1 #SBATCH --array=0-1 mkdir -p output_run_$SLURM_ARRAY_TASK_ID run_me --output-dir=output_run_$SLURM_ARRAY_TASK_ID --run=$SLURM_ARRAY_TASK_ID

[vrindger]

/var/spool/slurmd/job02604/slurm_script: line 24: SLURM_ARRAY_TASK_ID: command not found run_me: error: argument --run: invalid int value: ''

[vrindger]

never mind, please ignore. This was just because I had parentheses around $(SLURM_ARRAY_TASK_ID) and the runs are actually working

[vrindger]

Sorry, never mind, the parenthesis thing fixed it when directly using the sbatch command but not when using pyslurm. When i use pyslurm with the same exact sbatch file, it doesn't work, whereas it does with the sbatch command for a job array. It does not look like the $SLURM_ARRAY_TASK_ID thing works through the submit_batch_job script submission?

[vrindger]

bump @giovtorres :)

[giovtorres]

The current implementation of submit_batch_job does not parse the SBATCH directives.

[giovtorres]

This part of the code could use some contributions :wink:

[0xstarfox]

I believe this works but the option name does not match the sbatch command line option like the docs suggest it should. Use "array_inx" instead of "array".

import pyslurm
test_job = {
    "job_name": "myjob",
    "array_inx": "1-5",
    "wrap": "sleep 1",
}
test_job_id = pyslurm.job().submit_batch_job(test_job)

[tazend]

The Job submission API has been reworked and now comes in a seperate class. (At the moment just for slurm 23.02, but soon will be backported to other versions) Feel free to have a look here

For example, submitting an array-job:

import pyslurm
desc = pyslurm.JobSubmitDescription(name="myjob", time_limit=5, array="1-5", script="#!/bin/bash\n srun sleep 1000")
job_id = desc.submit()
...

Note: When you specify a script that contains #SBATCH directives, you can now also load the values of the parameters from them into the JobSubmitDescription instance you have created. For more infos, see here