dask-jobqueue icon indicating copy to clipboard operation
dask-jobqueue copied to clipboard

SLURMCluster: header_skip should not skip user-defined extra header lines

Open willirath opened this issue 4 years ago • 1 comments

Currently, with SLURMCluster, the user can add arbitrary header lines by adding them to the job_extra list. These extra lines end up in the job_header attribute right upon initialisation of the cluster.

Before submitting a job, the header_skip list is used to filter all header lines and removes lines matching the skip list even if the user explicitly set them via job_extra.

(This came up in #469.)

willirath avatar Aug 24 '20 06:08 willirath

Thanks @willirath for analysing this.

Maybe we could use header_skip during initialization? I don't think we really need to do it every time we generate the job_script... This was probably simpler to do as there is no generic method for the header generation currently, and the job_extra are added in every implementation.

So I propose to add such a method, and put the header_skip before adding job_extra.

guillaumeeb avatar Aug 25 '20 11:08 guillaumeeb