firefox-translations-training
firefox-translations-training copied to clipboard
mozilla
Preemptable marian training on slurm.
These changes allow marian training jobs on slurm to be interrupted without losing training progress. The script requests an early warning from slurm a set amount of time (currently 300 sec.) ahead of end of time slots, runs marian in the background and shuts it down nicely with SIGTERM so that marian can save its state of training. It then examins marians exit status and cancels (or not) the current slurm job array to avoid wasting subsequent slurm scheduling slots.
Needs testing.
One open question here is what Snakemake does with non-0 exit codes of scripts. I am completely new to Snakemake ...
@eu9ene For testing on slurm, you could just replace the marian job by tail -f /dev/null&. That will also return 143 when killed with SIGTERM.
One open question here is what Snakemake does with non-0 exit codes of scripts. I am completely new to Snakemake ...
Snakemake deletes the job output that is specified as "output" in the rule. It is a safety mechanism to prevent corrupted data.
I believe this is meant to solve specific issues on HPC that we're not using at the moment. Closing