Synchronize version number with dask and distributed

[erl987]

trafficstars

I suggest to have the same version number as a compatible dask and distributed version.

It is extremely difficult to know which versions are compatible. It looks like just using the latest of all packages usually fails.

I ran into this now a second time. I could only resolve it by trial and error. Unfortunately the error messages are extremely unclear. Typically it is like this when creating a cluster: GatewayClusterError: Cluster 'adrastea.b4286778ea9b49f4b4264f982f5b278d' failed to start, see logs for more information

Similar behaviour has been reported in #348.

These logs are in a terminated pod and are hard to get, and don't tell much.

Another variant is that getting the client fails in an unclear way such as "no connection".

A related request suggesting better documentation of this is already given in #161.

But the best solution would be to synchronize the version numbers of all involved packages, another to show clear errors as it is today for Python version mismatches.

I think that the current behaviour is a showstopper for many less experienced users.

[TomAugspurger]

Just to confirm: Is you question about which versions of dask-gateway are compatible with a given Dask / distributed version? This is a bit hard, since there isn't necessarily a straightforward mapping between the two sets of versions. Ideally dask-gateway a given version of dask gateway would be compatible with multiple versions of Dask and distributed going forwards and backwards from the release of Dask Gateway.

As you note: https://github.com/dask/dask-gateway/issues/161 suggests clarifying this through documentation. A grid showing which versions of dask and dask and distributed a given version of dask-gateway is compatible with.

[droctothorpe]

The values yaml references this image.

Here it is on Dockerhub.

You can see it specifies:

ARG DASK_VERSION=2.30.0
ARG DISTRIBUTED_VERSION=2.30.1

That's a good starting point.

I don't see the associated Dockerfile anywhere in the org, however.

There's this example image, but it doesn't specify Dask or Distributed versions.

I defer to @jcrist , of course, but it might not be a bad idea to commit the Dockerfile that was used to build the daskgateway/dask-gateway image to source for reference or update the existing example image to include the Dask and Distributed (and essential auxiliary package) installations since the scheduler/workers are not very useful without them.

I would be happy to submit an admittedly trivial PR (I'll take what I can get 🙃 ) if this makes sense.

[erl987]

@TomAugspurger right.