Does dask-cloudprovider support private dockerhub images?

[asmith26]

Does dask-cloudprovider support private docker-hub images (or AWS Elastic Container Registry/ECS)? I tried with no joy:

import dask_cloudprovider
from dask_cloudprovider import FargateCluster
dask_cloudprovider.__version__  # 0.2.0

cluster = FargateCluster(cloudwatch_logs_group="my_log_group",
                         cluster_name_template='test',
                         scheduler_timeout="1 hour",
                         security_groups=["A CUSTOM SECURITY GROUP"],
                         n_workers=1,
                         image="asmith26/test-dask_cloudprovider",  # points to a private dockerhub (works when made public)
                        )

[UPDATED following @martindurant comment, good idea, many thanks :)]

[martindurant]

It may be useful to show what you've tried so far, in case a trivial change is apparent to someone.

[asmith26]

I tried using AWS Elastic Container Registry/ECR today and that seems to be working, Thanks for your help and sorry for any inconvenience (I hadn't really used ECR before today).

[jacobtomlinson]

I'm going to reopen this as we currently do not support private image pulls in FargateCluster and it would be good to add that.

ECR is the recommended approach generally as image pulls will be faster. But there is no reason why folks shouldn't be able to pull from other private repos.

It looks like the credentials need to be stored in secrets manager and then provided in the task definition. I would be tempted to say putting the credentials in secrets manager is out of scope for FargateCluster, but we should add a kwarg for configuring the secret name in the task definition.

https://aws.amazon.com/blogs/compute/introducing-private-registry-authentication-support-for-aws-fargate/

[mhnatiuk]

Hey, what the status on that? Trying to use private ECR, everything is set up, but my (im using EC2Cluster) scheduler dies after short time. Unfortunately, though running with debug, the only thing i get is my client waiting:

Created instance i-5********** as dask-5e19ba61-scheduler
Waiting for scheduler to run at 3.x.x.x:8786

When i change docker_image argument to default (dask provided image), everything works. My own image provides dask-scheduler and dask-worker commands, as it's based on dask image, just installs couple of our private repos and configures jobs

[myan-numagic]

Is it also the case for the AzureVMCluster that there appears to be no way to pass private container registry authentication to the VM so the docker pull would fail here?

Or maybe I just missed something obvious, in which case if anybody could point that out it would be greatly appreciated...

One possible workaround I guess is to grant the Azure VMs access AFTER creation, like mentioned here, but I'm not sure how practical that would be ...

[jacobtomlinson]

@myan-numagic could you raise a separate issue for this as you are talking about a different cloud.