Feature Request: Overwrite sbatch options via CLI

[n-gao]

#79 follow-up. At the moment, one must specify sbatch options in the yaml file. However, as discussed in #71, it may be beneficial to override these settings via the CLI. The suggested syntax would be:

seml <collection> add <yaml> -sb mem=25GB partition=gpu_all

The new syntax should be adapted to the -sb option in seml jupyter as well. For parsing the key, values we can reuse the key value parser introduced in #79.

[gasteigerjo]

In #79 Daniel mentioned that this should allow overriding Slurm parameters when starting an experiment, which is not addressed by this design.

However, I don't think it's ideal to have start do side-effects like changing the database. I'd rather propose a new command (update?) for updating config values. This would combine the filtering syntax of start with the overriding syntax of this and #79:

seml <collection> update -b 7 -sb mem=25GB -o dataset=imagenet

I'd suggest to add the overriding options to both add and update then.

[n-gao]

Having an update command raises the potential issue that an overwrite parameter could cause two identical configurations. We would then have to check for duplicates again and potentially remove these. Still, having an override option at add time seems beneficial to me.

[n-gao]

I'd suggest breaking this up into two issues, as these are more or less orthogonal:

1. Adding sbatch options to add (+ new syntax for jupyter)
2. Adding an update command.

[gasteigerjo]

Yes, good point! Let's keep the PRs small and manageable.

update could also break our assumption that every batch has consistent Slurm parameters. Creating batch arrays and sharing GPUs both rely on this. As a first step we could only allow changing Slurm parameters of full batches (and document the reasoning behind these restrictions).