Support both TLS and TCP connections between schedulers and workers

[orf]

This attempts to support both TLS and TCP connections between schedulers and workers. Closes #449

[orf]

This seems very, very convoluted. There are assumptions baked in everywhere about the protocol, and some of these are implicit (i.e relying on the existence of a keypair).

It doesn't seem possible to have the scheduler listen on TCP, whilst the gateway accepts connections via TLS (which is required due to some SNI routing in the go proxy?). But this seems quite convoluted.

[martindurant]

Looking at this now.

It doesn't seem possible to have the scheduler listen on TCP, whilst the gateway accepts connections via TLS

I think the idea is that you could have TLS termination in the proxy level; but agree that there is no principled reason that the API server and the dask-scheduler should be using the same protocol.

One major comment: actually we support other protocols in dask, particularly UCX. @quasiben : is it possible to have the schduler and workers within the cluster talk UCX, but TCP or TLS for client-scheduler? Or does UCX just figure it all out, magically?