support running Dask-Gateway on HPC systems without the use of sudo

[jglaser]

What happened:

Currently , the jobqueue component of dask-gateway relies on sudo to do user authentication on its own, rather than integrating with authentication mechanisms provided by the respective systems it is deployed on, or just submitting with the user privileges it is given by the admin.

The current implementation is a big NO for any production HPC system, and I can't imagine any sysadmin in their right mind who would allow such practice.

What you expected to happen:

Run as a user, never try to sudo.

Minimal Complete Verifiable Example:

Submit a job through the SLURM or PBS backend.

Anything else we need to know?:

In a production environment, it is OK if the dask-gateway server runs as a special user, either per-project or globally. However, that implies that all jobs should be submitted as that specific user, rather than trying to impersonate a different user. To still allow tracing back cluster jobs to the users who submitted them, it should be possible to assign individual names to the GatewayClusters submitted to disambiguate them later, e.g. a userid or userid+sessionid.

Environment:

• Dask version: 2.25.0
• Python version: 3.7.0
• Operating System: linux ppc64le
• Install method (conda, pip, source): source

[jcrist]

The current implementation is a big NO for any production HPC system, and I can't imagine any sysadmin in their right mind who would allow such practice.

This is a very discouraging comment. Dask-Gateway's use of sudo is modeled after JupyterHub's batchspawner, which is deployed in several "production" HPC systems. Dask-Gateway also has several HPC users who have been fine using sudo to allow clusters to run with the requesting user's credentials.

A more nuanced issue report would be something like "support running Dask-Gateway on HPC systems without the use of sudo". I'm certainly not an HPC system expert, so if there are other ways of running a job under a different user account in a maintainable way, I'd be happy to support them. Adding a config option to run all jobs as the dask-gateway server user would also be acceptable. If you have other productive suggestions on useful extension points we could add to support this I'd be glad to hear them.

[jglaser]

@jcrist I am sorry if I was sounding dismissive, I did not mean to discourage. This is a much needed project with a well thought out design. The only issue is, as I stated, that the use of sudo should not be default, or it should be possible to disable it. This will facilitate adoption on tightly secured HPC resources with many users. Case in point, I am adapting the software to work on Summit and Marble/Slate (an automation cluster connected to Summit). Please see my two pull requests, it works great. Thank you for your efforts!

[rcthomas]

We override the sudo part in our deployment of BatchSpawner. We're looking at the same kind of override here.