Allow forwarding CLI options

[jcrist]

Some options on the distributed CLI should be settable via dask-yarn (e.g. --dashboard-address). There are a few ways we could accomplish this:

• Update distributed so that all parameters are also part of the configuration. Users could then provide them via env (or via configuration #42).
• Add an option to specify these parameters as part of the command

From https://github.com/jcrist/skein/issues/176.

[mrocklin]

I think that it would be useful to start thinking about a uniform way to specify things across all of the projects.

On Mon, Jun 17, 2019 at 4:56 PM Jim Crist [email protected] wrote:

Some options on the distributed CLI should be settable via dask-yarn (e.g. --dashboard-address). There are a few ways we could accomplish this:

• Update distributed so that all parameters are also part of the configuration. Users could then provide them via env (or via configuration #42 https://github.com/dask/dask-yarn/issues/42).
• Add an option to specify these parameters as part of the command

From jcrist/skein#176 https://github.com/jcrist/skein/issues/176.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/dask-yarn/issues/69?email_source=notifications&email_token=AACKZTGS7G4W6MPPTRNJ74TP26QYVA5CNFSM4HYXPJLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GZ5AHFQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AACKZTBRTTTHKC7BX2WRGQDP26QYVANCNFSM4HYXPJLA .

[jcrist]

I agree. I think a precursor to this would be making the CLI scripts in distributed more uniform in terms of argument parsing/config, and how processes are started/stopped.

Edit: misread your comment. I still think the above is important for yarn specific (since we need our own cli scripts), but common arguments for the cluster managers for specifying the worker/scheduler command would be good.