[feature request] Docker registry auth via driver config

[Oloremo]

Registry auth in task config was recently added in https://github.com/Roblox/nomad-driver-containerd/pull/89

This is great! But adding auth info into every Nomad job hcl seems like a bit inconvenient.

Nomad docker drive has an option to set registry auth in drive config: https://www.nomadproject.io/docs/drivers/docker#authentication

And it's very useful from both operations and security perspectives.

Any plans to add it?

[shishir-a412ed]

Yes, this will be added to the containerd-driver soon.

[shishir-a412ed]

@Oloremo I did some more research on this.

Currently, in Nomad docker driver, auth is supported in the following ways.

Task Config

• You can specify username and password in the job task config. For security concerns, you can also store the username and password in vault (If you use vault for managing secrets in your environment) and interpolate the username/password using consul templates in the nomad job. Example here

This currently is supported in containerd-driver.

Plugin config

1. Set username and password in Nomad client config, so user don't need to set it on every job.

plugin "docker" {
  config {
    auth {
      username=$username
      password=$password
    }
  }
}

We can support the same thing in containerd-driver and this would allow you to specify auth (username/password) at the node level.

2. Config option.

plugin "docker" {
  config {
    auth {
       config = "/etc/docker-auth.json"
    }
  }
}

You can specify the location of your docker auth config. Usually it's stored under $HOME/.docker/config.json in your Linux environments. This config supports

• Basic auth (username/password)
• Credential store e.g. osxkeychain (If your username/password are stored in your macOSx keychain)
• Credential helper e.g. if you are using a EC2 container registry (ecr) or some third party registry.

Link to docker documentation

When you launch a job using config option, Nomad docker driver will use /etc/docker-auth.json (In this example) to auth against your private registry.

3. Helper option

plugin "docker" {
   config {
       auth {
           helper = "ecr-login"
       }
    }
}

When you launch a job using the helper option, Nomad docker driver will prefix the helper name with docker-credential-, and look for that script under your $PATH. E.g. in this case, it will look for docker-credential-ecr-login under your $PATH, and use that script to authenticate against your private registry.

I ll open a PR for (1) which should atleast get you going. (2) is a docker specific option so I need to think more if this can be supported in containerd or what would be the best way to address this. Similarly for (3) I don't have a good way to test this right now, so I ll leave it open, if someone in the community uses credentials-helpers, they can pick it up and implement the PR.

[Oloremo]

@shishir-a412ed

Thank you for looking into this!

Option 1) is indeed the most simple one and way better than defining it per task but it's not very secure. While I see how option 2) could be considered docker-specific, I'm not sure if it should be a good reason to not go with it.

I could be wrong, but my understanding is that even if Docker engine will go away, the Dockerfile format and everything related to the registry(like the protocol) - will not. If we look at this in that light, isn't it makes sense to support that approach? I mean, we can rename config to registry.json :-) It's not really about docker anymore.

[shishir-a412ed]

@Oloremo I agree with you, and you are right, most of the orchestration systems are moving away from Docker. k8s has already moved to containerd (as the default). So it's only a matter of When and not If!

Having said that, I think I didn't articulate my response properly. What I meant was, the changes to support config option is not trivial, and I need to research/assess on what would be the best way to support this.

You can call it registry.json but e.g. containerd might not support osxkeychain credential store, since containerd is not supported on Mac. This is just one example, and there could be other cases too. Since it's a docker config, docker has the underlying libraries to support all the modes {auth, credential stores and credential helpers}. Containerd might not support these out of the box.

Definitely need this, but not super high priority right now.

[Oloremo]

Thank you for your detailed response!

Makes sense. Now I wonder what is the plan for the whole container ecosystem moving on in terms of the registry...

[shishir-a412ed]

@Oloremo Option (1) is available to use. I ll keep the issue open, and see if we can make some progress on (2) at somepoint.

[devops-at-home]

I have a PR ready to go for options (2) and (3), are you able to give me access to create a branch and push my code?

[shishir-a412ed]

@devops-at-home You have to fork the repo, and open the PR from the fork. Typically open source repositories uses fork-and-PR model.

This might be helpful: https://sqldbawithabeard.com/2019/11/29/how-to-fork-a-github-repository-and-contribute-to-an-open-source-project/

[devops-at-home]

Apologies for the delay with this. I ran in to a few issues and have since been flat out with work. I see @shoenig has since updated the dependencies and this was one of the problems I experienced.

I skipped 2. Config option and instead just implemented 3. Helper option which I am now running in production from my own fork and my Raspberry Pi4 is authenticating against AWS ECR using credentials that are managed by AWS SSM.

A problem with the config spec at the top of this thread is that 1. Set username and password in Nomad client config requires two params and I am unsure how to configure HCL to accept (username AND password) OR auth_config OR auth_helper. If you look at Hashicorps's Docker plugin they do not support username and password in the client config, likely because of this reason.

https://github.com/hashicorp/nomad/blob/f2a37a0a03045d922b933373e71385b3c615ac63/drivers/docker/config.go#L189

Also for anyone who wants to use Amazon's ECR Helper on Ubuntu 20.04 this is how to install it without Docker which we don't need as we're going straight to containerd. I didn't install containerd or runc with apt as I installed the binaries directly with nerdctl. https://github.com/containerd/nerdctl/releases

sudo apt install amazon-ecr-credential-helper \
  --no-install-recommends \
  bridge-utils- \
  containerd- \
  docker.io- \
  pigz- \
  runc- \
  ubuntu-fan-

One final gotcha is that if you are using AWS SSM as root user you will need to set

Environment="HOME=/root"

in your systemd config for Nomad otherwise the ecr-helper will not have the correct credentials for authentication.

[Oloremo]

Any reasons to skip option 2?

[kholisrag]

@devops-at-home mind to PR to main repo? so many user can use your third method?