Watchtower is not using the credential helper with IAM Roles

[ldb]

Describe the bug

Even though the credential helper is configured as described here, it appears to not be used by watchtower. From inside an alpine container I am able to correctly receive credentials by executing the helper binary inside the helper volume. Watchtower, on the other hand, simply says it does not find any credentials.

To Reproduce

Expected behavior

It should just work Screenshots

Environment

Logs from running watchtower with the --debug option

level=debug msg="Trying to load authentication credentials." container=/[redacted ]image="[redacted].dkr.ecr.eu-central-1.amazonaws.com/[redacted]:latest"
level=debug msg="No credentials for [redacted].dkr.ecr.eu-central-1.amazonaws.com found" config_file=/config.json
...
level=debug msg="Reason: no credentials available" container=/[redacted] image="[redacted].dkr.ecr.eu-central-1.amazonaws.com/[redacted]:latest"
...
level=debug msg="Error pulling image [redacted ].dkr.ecr.eu-central-1.amazonaws.com/[redacted]:latest, Error response from daemon: Head https://[redacted].dkr.ecr.eu-central-1.amazonaws.com/v2/[redacted ]/manifests/latest: no basic auth credentials"

Additional context

This is not the same as #957 as we are trying to authenticate the credential helper via IAM Roles instead of access keys. I still tried that for completeness and it still did not work.

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[piksel]

Yeah, the step that doesn't work is probably the lookup in config.json, although if the credential-helper returned an error you would get the same message. Could you post the config json as well? It shouldn't contain any actual credentials (but you could just remove them in that case). Also, when doing those redactions, if you use a token label instead it's easier to follow ([aws-account] instead of [redacted] etc.)

[ldb]

The config looked similar to what was suggested in the documentation:

{
   "credsStore" : "ecr-login",
   "auths" : {
     "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com" : {}
   },
   "credHelpers": {
     "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com" : "ecr-login"
   }
}

We have since changed it to just using docker login together with ECR to store credentials in the file to workaround the helper not being used.

[piksel]

But your image is on eu-central-1? I guess you just copied what was in the documentation, that's not really helpful.

But if you solved it another way, that's great.

[ldb]

Yes, I did copy that part from the documentation.Our configuration looked exactly the same, though (of course with the correct regions and account IDs).

Well, the other solution is not really a solution to this problem, more a workaround, as it requires us to docker login periodically.

It is really weird, it seemed that the ECR login helper was just not being used. As mentioned, we verified that it works (also through the mounted volume) using an Alpine container. We managed to get credentials that way by calling the helper manually.

[ldb]

Closing due to inactivity :(

[piksel]

Yeah, sorry. It works for me, and you are giving me very little to work with.