dask-jobqueue
dask-jobqueue copied to clipboard
SLURMCluster: header_skip should not skip user-defined extra header lines
Currently, with SLURMCluster
, the user can add arbitrary header lines by adding them to the job_extra
list. These extra lines end up in the job_header
attribute right upon initialisation of the cluster.
Before submitting a job, the header_skip
list is used to filter all header lines and removes lines matching the skip list even if the user explicitly set them via job_extra
.
(This came up in #469.)
Thanks @willirath for analysing this.
Maybe we could use header_skip during initialization? I don't think we really need to do it every time we generate the job_script... This was probably simpler to do as there is no generic method for the header generation currently, and the job_extra are added in every implementation.
So I propose to add such a method, and put the header_skip before adding job_extra.